0
0
mirror of https://github.com/nodejs/node.git synced 2024-11-30 15:30:56 +01:00
nodejs/tools/icu
James M Snell ed21cb1774 util: implement WHATWG Encoding Standard API
Provide an (initially experimental) implementation of the WHATWG Encoding
Standard API (`TextDecoder` and `TextEncoder`). The is the same API
implemented on the browser side.

By default, with small-icu, only the UTF-8, UTF-16le and UTF-16be decoders
are supported. With full-icu enabled, every encoding other than iso-8859-16
is supported.

This provides a basic test, but does not include the full web platform
tests. Note: many of the web platform tests for this would fail by default
because we ship with small-icu by default.

A process warning will be emitted on first use to indicate that the
API is still experimental. No runtime flag is required to use the
feature.

Refs: https://encoding.spec.whatwg.org/
PR-URL: https://github.com/nodejs/node/pull/13644
Reviewed-By: Timothy Gu <timothygu99@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
2017-07-24 14:04:13 -07:00
..
patches deps: Intl: ICU 58 bump: configure/LICENSE/docs 2016-10-31 13:42:52 -07:00
icu_small.json tools: do not disable ICU's transliteration 2016-09-22 09:55:39 +02:00
icu-generic.gyp util: implement WHATWG Encoding Standard API 2017-07-24 14:04:13 -07:00
icu-system.gyp
iculslocs.cc build: remove dependency on icu io library 2017-06-26 11:36:24 +02:00
icutrim.py
no-op.cc
README.md tools: update: eslint-plugin-markdown@1.0.0-beta.7 2017-07-05 06:51:10 +03:00
shrink-icu-src.py build: remove dependency on icu io library 2017-06-26 11:36:24 +02:00

Notes about the icu directory.

How to upgrade ICU

  • Make sure your node workspace is clean (clean git status) should be sufficient.
  • Configure Node with the specific ICU version you want to upgrade to, for example:
./configure \
    --with-intl=small-icu \
    --with-icu-source=http://download.icu-project.org/files/icu4c/58.1/icu4c-58_1-src.tgz
make

(The equivalent vcbuild.bat commands should work also. Note that we use the .tgz and not the .zip here, that is because of line endings.)

  • (note- may need to make changes in icu-generic.gyp or tools/icu/patches for version specific stuff)

  • Verify the node build works

make test-ci

Also running

new Intl.DateTimeFormat('es', {month: 'long'}).format(new Date(9E8));

…Should return January not enero. (TODO here: improve testing)

  • Now, copy deps/icu over to deps/icu-small
python tools/icu/shrink-icu-src.py
  • Now, do a clean rebuild of node to test:

(TODO: fix this when these options become default)

./configure --with-intl=small-icu --with-icu-source=deps/icu-small
make
  • Test this newly default-generated Node.js
process.versions.icu;
new Intl.DateTimeFormat('es', {month: 'long'}).format(new Date(9E8));

(should return your updated ICU version number, and also January again.)

  • You are ready to check in the updated deps/small-icu. This is a big commit, so make this a separate commit from other changes.

  • Now, rebuild the Node license.

# clean up - remove deps/icu
make clean
tools/license-builder.sh
  • Now, fix the default URL for the full-icu build in /configure, in the configure_intl() function. It should match the ICU URL used in the first step. When this is done, the following should build with full ICU.
# clean up
rm -rf out deps/icu deps/icu4c*
./configure --with-intl=full-icu --download=all
make
make test-ci
  • commit the change to configure along with the updated LICENSE file.

Notes about these tools

The files in this directory were written for the node.js effort. It's the intent of their author (Steven R. Loomis / srl295) to merge them upstream into ICU, pending much discussion within the ICU-PMC.

icu_small.json is somewhat node-specific as it specifies a "small ICU" configuration file for the icutrim.py script. icutrim.py and iculslocs.cpp may themselves be superseded by components built into ICU in the future.

The following tickets were opened during this work, and their resolution may inform the reader as to the current state of icu-trim upstream:

  • #10919 (experimental branch - may copy any source patches here)
  • #10922 (data packaging improvements)
  • #10923 (rewrite data building in python)

When/if components (not including the .json file) are merged into ICU, this code and configure will be updated to detect and use those variants rather than the ones in this directory.