Add buffer.transcode(source, from, to) method. Primarily uses ICU
to transcode a buffer's content from one of Node.js' supported
encodings to another.
Originally part of a proposal to add a new unicode module. Decided
to refactor the approach towrds individual PRs without a new module.
Refs: https://github.com/nodejs/node/pull/8075
PR-URL: https://github.com/nodejs/node/pull/9038
Reviewed-By: Anna Henningsen <anna@addaleax.net>
ICU should be compiled with -frtti (and it sets that flag in its gyp
file) but it was also inheriting the -fno-rtti flag from common.gypi,
breaking the build on some systems.
Fixes: https://github.com/nodejs/node/issues/8867
PR-URL: https://github.com/nodejs/node/pull/8886
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com>
ICU has a punycode implementation built in. Use it instead of the
javascript implementation because it's much faster.
PR-URL: https://github.com/nodejs/node/pull/7355
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
* Change configure default to "small-icu" (Intl on, English only)
* add "--without-intl" and "vcbuild without-intl" options, equivalent
to --with-intl=none
* update BUILDING.md with above changes
* Checks in tools that generate the deps/icu-small source directory
from ICU source
* Tools and process for updating ICU documented in tools/icu/README.md
Fixes: https://github.com/nodejs/node/issues/3476
PR-URL: https://github.com/nodejs/node/pull/6088
Reviewed-By: James M Snell <jasnell@gmail.com>
* Exclude `derb.cpp` as well as `derb.c` from Node builds
(file was renamed in ICU 56)
ICU 56 renamed derb.c to derb.cpp because of C++ yay.
This broke the exclusion of "derb.c" when building tools.
Solution is to add derb.c AND derb.cpp to exclusion.
We don't build the 'derb' tool, so it's fine to list the
excluded source twice.
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
PR-URL: https://github.com/nodejs/node/pull/3066
Fixes: https://github.com/nodejs/node/issues/3065
For ICU 55 we are currently stripping out bidi and
the callers. AIX is more pedantic and identified
an additional caller than needed to be stripped out.
This PR adds that caller to those stripped out.
PR-URL: https://github.com/nodejs/node/pull/3115
Reviewed-By: Steven Loomis <srloomis@us.ibm.com>
Fix a build error that occurs when icutrim is run to cut down
the ICU locale set on Big-Endian platforms when building with
--with-intl=small-icu (which is done by the "make binary" target).
This fixes the binary build on POWER platforms.
Fixes: https://github.com/nodejs/node/issues/2601
PR-URL: https://github.com/nodejs/node/pull/2602
Reviewed-By: Steven Loomis <srloomis@us.ibm.com>
Make "--with-intl=none" the default and add "intl-none" option to
vcbuild.bat.
If icu data is missing print a warning unless either --download=all or
--download=icu is set. If set then automatically download, verify (MD5)
and unpack the ICU data if not already available.
There's a "list" of URLs being used, but right now only the first is
picked up. The logic works something like this:
* If there is no directory deps/icu,
* If no zip file (currently icu4c-54_1-src.zip),
* Download zip file (icu-project.org -> sf.net)
* Verify the MD5 sum of the zipfile
* If bad, print error and exit
* Unpack the zipfile into deps/icu
* If deps/icu now exists, use it, else fail with help text
Add the configuration option "--with-icu-source=..."
Usage:
* --with-icu-source=/path/to/my/other/icu
* --with-icu-source=/path/to/icu54.zip
* --with-icu-source=/path/to/icu54.tgz
* --with-icu-source=http://example.com/icu54.tar.bz2
Add the configuration option "--with-icu-locals=...". Allows choosing
which locales are used in the "small-icu" case.
Example:
configure --with-intl=small-icu --with-icu-locales=tlh,grc,nl
(Also note that as of this writing, neither Klingon nor Ancient Greek
are in upstream CLDR data. Serving suggestion only.)
Don't use hard coded ../../out paths on windows. This was suggested by
@misterdjules as it causes test failures. With this fix, "out" is no
longer created on windows and the following can run properly:
python tools/test.py simple
Reduce space by about 1MB with ICU 54 (over without this patch). Also
trims a few other source files, but only conditional on the exact ICU
version used. This is to future-proof - a file that is unneeded now may
be needed in future ICUs.
Also:
* Update distclean to remove icu related files
* Refactor some code into tools/configure.d/nodedownload.py
* Update docs
* Add test
PR-URL: https://github.com/joyent/node/pull/8719
Fixes: https://github.com/joyent/node/issues/7676#issuecomment-64704230
[trev.norris@gmail.com small change to test's whitespace and logic]
Signed-off-by: Trevor Norris <trev.norris@gmail.com>
So what I did here is to make the icutools library actually contain the
entire ICU dependencies needed for host-side tools. Sadly, this will
build ICU twice, but avoids conflicts between host and target side.
This all seems like a gyp bug of some sort, but without docs for
toolsets, who’s to say?
I removed the icuio library as a separate target, because it was only
used by the host-side tools.
PR-URL: https://github.com/joyent/node/pull/8681
Reviewed-by: Trevor Norris <trev.norris@gmail.com>
The two main goals of this change are:
- To make it easier to build the Intl option using ICU (particularly,
using a newer ICU than v8/Chromium's version)
- To enable a much smaller ICU build with only English support The goal
here is to get node.js binaries built this way by default so that the
Intl API can be used. Additional data can be added at execution time
(see Readme and wiki)
More details are at https://github.com/joyent/node/pull/7719
In particular, this change adds the "--with-intl=" configure option to
provide more ways of building "Intl":
- "full-icu" picks up an ICU from deps/icu
- "small-icu" is similar, but builds only English
- "system-icu" uses pkg-config to find an installed ICU
- "none" does nothing (no Intl)
For Windows builds, the "full-icu" or "small-icu" options are added to
vcbuild.bat.
Note that the existing "--with-icu-path" option is not removed from
configure, but may not be used alongside the new option.
Wiki changes have already been made on
https://github.com/joyent/node/wiki/Installation
and a new page created at
https://github.com/joyent/node/wiki/Intl
(marked as provisional until this change lands.)
Summary of changes:
* README.md : doc updates
* .gitignore : added "deps/icu" as this is the location where ICU is
unpacked to.
* Makefile : added the tools/icu/* files to cpplint, but excluded a
problematic file.
* configure : added the "--with-intl" option mentioned above.
Calculate at config time the list of ICU source files to use and data
packaging options.
* node.gyp : add the new files src/node_i18n.cc/.h as well as ICU
linkage.
* src/node.cc : add call into
node::i18n::InitializeICUDirectory(icu_data_dir) as well as new
--icu-data-dir option and NODE_ICU_DATA env variable to configure ICU
data loading. This loading is only relevant in the "small"
configuration.
* src/node_i18n.cc : new source file for the above Initialize..
function, to setup ICU as needed.
* tools/icu : new directory with some tools needed for this build.
* tools/icu/icu-generic.gyp : new .gyp file that builds ICU in some new
ways, both on unix/mac and windows.
* tools/icu/icu-system.gyp : new .gyp file to build node against a
pkg-config detected ICU.
* tools/icu/icu_small.json : new config file for the "English-only" small
build.
* tools/icu/icutrim.py : new tool for trimming down ICU data. Reads the
above .json file.
* tools/icu/iculslocs.cc : new tool for repairing ICU data manifests
after trim operation.
* tools/icu/no-op.cc : dummy file to force .gyp into using a C++ linker.
* vcbuild.bat : added small-icu and full-icu options, to call into
configure.
* Fixed toolset dependencies, see
https://github.com/joyent/node/pull/7719#issuecomment-54641687
Note that because of a bug in gyp {CC,CXX}_host must also be set.
Otherwise gcc/g++ will be used by default for part of the build.
Reviewed-by: Trevor Norris <trev.norris@gmail.com>
Reviewed-by: Fedor Indutny <fedor@indutny.com>
