Provide an (initially experimental) implementation of the WHATWG Encoding
Standard API (`TextDecoder` and `TextEncoder`). The is the same API
implemented on the browser side.
By default, with small-icu, only the UTF-8, UTF-16le and UTF-16be decoders
are supported. With full-icu enabled, every encoding other than iso-8859-16
is supported.
This provides a basic test, but does not include the full web platform
tests. Note: many of the web platform tests for this would fail by default
because we ship with small-icu by default.
A process warning will be emitted on first use to indicate that the
API is still experimental. No runtime flag is required to use the
feature.
Refs: https://encoding.spec.whatwg.org/
PR-URL: https://github.com/nodejs/node/pull/13644
Reviewed-By: Timothy Gu <timothygu99@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
* Remove pinning of eslint-plugin-markdown
An issue affecting Node.js source has been fixed in
eslint-plugin-markdown so we don't need to pin it to beta-4 anymore.
Refs: https://github.com/eslint/eslint-plugin-markdown/issues/69
* Update eslint-plugin-markdown up to 1.0.0-beta.7
* Fix docs for eslint-plugin-markdown@1.0.0-beta.7
PR-URL: https://github.com/nodejs/node/pull/14047
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Yuta Hiroto <hello@about-hiroppy.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
The library is only used in a single build-time tool where it can be
easily substituted by regular libc I/O functions.
PR-URL: https://github.com/nodejs/node/pull/13656
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Refael Ackermann <refack@gmail.com>
ninja on Windows chokes if libs are named .Lib (Capital L)
PR-URL: https://github.com/nodejs/node/pull/12522
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Intention was to get to `PRODUCT_DIR` so no need to do path voodoo
Also added `'msvs_quote_cmd': 0` and more precise quoting
PR-URL: https://github.com/nodejs/node/pull/11217
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Fedor Indutny <fedor.indutny@gmail.com>
Reviewed-By: Steven R Loomis <srloomis@us.ibm.com>
* we use u_setDataDirectory() in "unicode/putil.h"
* at present, this header is indirectly included,
but this will change in ICU 59
* no impact on past ICUs.
* this is an exact analog to https://github.com/nodejs/node/issues/11753
PR-URL: https://github.com/nodejs/node/pull/12078
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Add buffer.transcode(source, from, to) method. Primarily uses ICU
to transcode a buffer's content from one of Node.js' supported
encodings to another.
Originally part of a proposal to add a new unicode module. Decided
to refactor the approach towrds individual PRs without a new module.
Refs: https://github.com/nodejs/node/pull/8075
PR-URL: https://github.com/nodejs/node/pull/9038
Reviewed-By: Anna Henningsen <anna@addaleax.net>
ICU should be compiled with -frtti (and it sets that flag in its gyp
file) but it was also inheriting the -fno-rtti flag from common.gypi,
breaking the build on some systems.
Fixes: https://github.com/nodejs/node/issues/8867
PR-URL: https://github.com/nodejs/node/pull/8886
Reviewed-By: Johan Bergström <bugs@bergstroem.nu>
Reviewed-By: Franziska Hinkelmann <franziska.hinkelmann@gmail.com>
1. Normalize headings.
2. Specify language in all code blocks.
PR-URL: https://github.com/nodejs/node/pull/8660
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
The format specifier is incomplete and without this the program will
fail at runtime, with "incomplete format" error.
PR-URL: https://github.com/nodejs/node/pull/7620
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
PR-URL: https://github.com/nodejs/node/pull/7483
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
ICU has a punycode implementation built in. Use it instead of the
javascript implementation because it's much faster.
PR-URL: https://github.com/nodejs/node/pull/7355
Reviewed-By: Trevor Norris <trev.norris@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
* Change configure default to "small-icu" (Intl on, English only)
* add "--without-intl" and "vcbuild without-intl" options, equivalent
to --with-intl=none
* update BUILDING.md with above changes
* Checks in tools that generate the deps/icu-small source directory
from ICU source
* Tools and process for updating ICU documented in tools/icu/README.md
Fixes: https://github.com/nodejs/node/issues/3476
PR-URL: https://github.com/nodejs/node/pull/6088
Reviewed-By: James M Snell <jasnell@gmail.com>
* Exclude `derb.cpp` as well as `derb.c` from Node builds
(file was renamed in ICU 56)
ICU 56 renamed derb.c to derb.cpp because of C++ yay.
This broke the exclusion of "derb.c" when building tools.
Solution is to add derb.c AND derb.cpp to exclusion.
We don't build the 'derb' tool, so it's fine to list the
excluded source twice.
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
PR-URL: https://github.com/nodejs/node/pull/3066
Fixes: https://github.com/nodejs/node/issues/3065
For ICU 55 we are currently stripping out bidi and
the callers. AIX is more pedantic and identified
an additional caller than needed to be stripped out.
This PR adds that caller to those stripped out.
PR-URL: https://github.com/nodejs/node/pull/3115
Reviewed-By: Steven Loomis <srloomis@us.ibm.com>
Fix a build error that occurs when icutrim is run to cut down
the ICU locale set on Big-Endian platforms when building with
--with-intl=small-icu (which is done by the "make binary" target).
This fixes the binary build on POWER platforms.
Fixes: https://github.com/nodejs/node/issues/2601
PR-URL: https://github.com/nodejs/node/pull/2602
Reviewed-By: Steven Loomis <srloomis@us.ibm.com>
Make "--with-intl=none" the default and add "intl-none" option to
vcbuild.bat.
If icu data is missing print a warning unless either --download=all or
--download=icu is set. If set then automatically download, verify (MD5)
and unpack the ICU data if not already available.
There's a "list" of URLs being used, but right now only the first is
picked up. The logic works something like this:
* If there is no directory deps/icu,
* If no zip file (currently icu4c-54_1-src.zip),
* Download zip file (icu-project.org -> sf.net)
* Verify the MD5 sum of the zipfile
* If bad, print error and exit
* Unpack the zipfile into deps/icu
* If deps/icu now exists, use it, else fail with help text
Add the configuration option "--with-icu-source=..."
Usage:
* --with-icu-source=/path/to/my/other/icu
* --with-icu-source=/path/to/icu54.zip
* --with-icu-source=/path/to/icu54.tgz
* --with-icu-source=http://example.com/icu54.tar.bz2
Add the configuration option "--with-icu-locals=...". Allows choosing
which locales are used in the "small-icu" case.
Example:
configure --with-intl=small-icu --with-icu-locales=tlh,grc,nl
(Also note that as of this writing, neither Klingon nor Ancient Greek
are in upstream CLDR data. Serving suggestion only.)
Don't use hard coded ../../out paths on windows. This was suggested by
@misterdjules as it causes test failures. With this fix, "out" is no
longer created on windows and the following can run properly:
python tools/test.py simple
Reduce space by about 1MB with ICU 54 (over without this patch). Also
trims a few other source files, but only conditional on the exact ICU
version used. This is to future-proof - a file that is unneeded now may
be needed in future ICUs.
Also:
* Update distclean to remove icu related files
* Refactor some code into tools/configure.d/nodedownload.py
* Update docs
* Add test
PR-URL: https://github.com/joyent/node/pull/8719
Fixes: https://github.com/joyent/node/issues/7676#issuecomment-64704230
[trev.norris@gmail.com small change to test's whitespace and logic]
Signed-off-by: Trevor Norris <trev.norris@gmail.com>
So what I did here is to make the icutools library actually contain the
entire ICU dependencies needed for host-side tools. Sadly, this will
build ICU twice, but avoids conflicts between host and target side.
This all seems like a gyp bug of some sort, but without docs for
toolsets, who’s to say?
I removed the icuio library as a separate target, because it was only
used by the host-side tools.
PR-URL: https://github.com/joyent/node/pull/8681
Reviewed-by: Trevor Norris <trev.norris@gmail.com>
The two main goals of this change are:
- To make it easier to build the Intl option using ICU (particularly,
using a newer ICU than v8/Chromium's version)
- To enable a much smaller ICU build with only English support The goal
here is to get node.js binaries built this way by default so that the
Intl API can be used. Additional data can be added at execution time
(see Readme and wiki)
More details are at https://github.com/joyent/node/pull/7719
In particular, this change adds the "--with-intl=" configure option to
provide more ways of building "Intl":
- "full-icu" picks up an ICU from deps/icu
- "small-icu" is similar, but builds only English
- "system-icu" uses pkg-config to find an installed ICU
- "none" does nothing (no Intl)
For Windows builds, the "full-icu" or "small-icu" options are added to
vcbuild.bat.
Note that the existing "--with-icu-path" option is not removed from
configure, but may not be used alongside the new option.
Wiki changes have already been made on
https://github.com/joyent/node/wiki/Installation
and a new page created at
https://github.com/joyent/node/wiki/Intl
(marked as provisional until this change lands.)
Summary of changes:
* README.md : doc updates
* .gitignore : added "deps/icu" as this is the location where ICU is
unpacked to.
* Makefile : added the tools/icu/* files to cpplint, but excluded a
problematic file.
* configure : added the "--with-intl" option mentioned above.
Calculate at config time the list of ICU source files to use and data
packaging options.
* node.gyp : add the new files src/node_i18n.cc/.h as well as ICU
linkage.
* src/node.cc : add call into
node::i18n::InitializeICUDirectory(icu_data_dir) as well as new
--icu-data-dir option and NODE_ICU_DATA env variable to configure ICU
data loading. This loading is only relevant in the "small"
configuration.
* src/node_i18n.cc : new source file for the above Initialize..
function, to setup ICU as needed.
* tools/icu : new directory with some tools needed for this build.
* tools/icu/icu-generic.gyp : new .gyp file that builds ICU in some new
ways, both on unix/mac and windows.
* tools/icu/icu-system.gyp : new .gyp file to build node against a
pkg-config detected ICU.
* tools/icu/icu_small.json : new config file for the "English-only" small
build.
* tools/icu/icutrim.py : new tool for trimming down ICU data. Reads the
above .json file.
* tools/icu/iculslocs.cc : new tool for repairing ICU data manifests
after trim operation.
* tools/icu/no-op.cc : dummy file to force .gyp into using a C++ linker.
* vcbuild.bat : added small-icu and full-icu options, to call into
configure.
* Fixed toolset dependencies, see
https://github.com/joyent/node/pull/7719#issuecomment-54641687
Note that because of a bug in gyp {CC,CXX}_host must also be set.
Otherwise gcc/g++ will be used by default for part of the build.
Reviewed-by: Trevor Norris <trev.norris@gmail.com>
Reviewed-by: Fedor Indutny <fedor@indutny.com>
