Problem:
Node implements fs.readFile as:
- a call to stat, then
- a C++ -> libuv request to read the entire file using the stat size
Why is this bad?
The effect is to place on the libuv threadpool a potentially-large
read request, occupying the libuv thread until it completes.
While readFile certainly requires buffering the entire file contents,
it can partition the read into smaller buffers
(as is done on other read paths)
along the way to avoid threadpool exhaustion.
If the file is relatively large or stored on a slow medium, reading
the entire file in one shot seems particularly harmful,
and presents a possible DoS vector.
Solution:
Partition the read into multiple smaller requests.
Considerations:
1. Correctness
I don't think partitioning the read like this raises
any additional risk of read-write races on the FS.
If the application is concurrently readFile'ing and modifying the file,
it will already see funny behavior. Though libuv uses preadv where
available, this doesn't guarantee read atomicity in the presence of
concurrent writes.
2. Performance
Downside: Partitioning means that a single large readFile will
require into many "out and back" requests to libuv,
introducing overhead.
Upside: In between each "out and back", other work pending on the
threadpool can take a turn.
In short, although partitioning will slow down a large request,
it will lead to better throughput if the threadpool is handling
more than one type of request.
Fixes: https://github.com/nodejs/node/issues/17047
PR-URL: https://github.com/nodejs/node/pull/17054
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Move functions that deal with `BufferList` to `BufferList.prototype`.
PR-URL: https://github.com/nodejs/node/pull/18353
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Instead of passing along the handle object, just set it as a
property on the stream handle object and let the read handler
grab it from there.
PR-URL: https://github.com/nodejs/node/pull/18334
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Instead of setting individual callbacks on streams and tracking
stream ownership through a boolean `consume_` flag, always have
one specific listener object in charge of a stream, and call
methods on that object rather than generic C-style callbacks.
PR-URL: https://github.com/nodejs/node/pull/18334
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This commit updates node_crypto to use the non-deprecated Set functions
that return a v8::Maybe<bool>.
PR-URL: https://github.com/nodejs/node/pull/17482
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Added tests to clarify the implicit behaviour of array header setting vs
object header setting
PR-URL: https://github.com/nodejs/node/pull/16568
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Under some conditions, the error received from getaddrinfo might
actually be EAGAIN, meaning the request should be retried. Allowing for
5 retries before erroring out.
Also replace one-off function with common.mustNotCall().
PR-URL: https://github.com/nodejs/node/pull/16534
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
I mistakenly introduced user defined variables using uppercase
characters, reading the gyp documentation they state:
"Predefined variables. By convention, these are named with
CAPITAL_LETTERS. Predefined variables are set automatically by GYP"
and also "By convention, user-defined variables are named with
lowercase_letters."
This commit renames the user defined variables to lowercase to follow
the above mentioned convention.
PR-URL: https://github.com/nodejs/node/pull/16238
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: James M Snell <jasnell@gmail.com>
Right now it is not documentated that WeakMap entries are not
compared. This might result in some confusion. This adds a note
about the behavior in `assert.deepStrictEqual`. This documentation
is also references in `util.isDeepStrictEqual`, so we do not have
to document it again for that function as the underlying algorithm
is the same.
PR-URL: https://github.com/nodejs/node/pull/18248
Fixes: https://github.com/nodejs/node/issues/18228
Refs: https://github.com/nodejs/node/pull/18228
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Returns the promises created by link so that they can be awaited to get
rid of race conditions while resolving and loading modules.
PR-URL: https://github.com/nodejs/node/pull/18394
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
In case no arguments are passed to `assert.ok` it should just
use a default message. Otherwise `assert.ok` can not be used as
a callback.
PR-URL: https://github.com/nodejs/node/pull/18319
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Weijia Wang <starkwang@126.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Also, alphabetize all types in type-parser.js
and fix some nits in type formats.
PR-URL: https://github.com/nodejs/node/pull/18444
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
currently if you want to use dynamic import you must use both the
`--experimental-modules` and the `--harmony-dynamic-imports` flags.
Chrome is currently shipping dynamic import unflagged, the flag
only remains in V8 to guard embedders who have not set the appropriate
callback from throwing an unhandled rejection when the feature is used.
As such it is reasonable to enable the flag by default for
`--experimental-modules`
PR-URL: https://github.com/nodejs/node/pull/18387
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Jan Krems <jan.krems@gmail.com>
Reviewed-By: Bradley Farias <bradley.meck@gmail.com>
Reviewed-By: Guy Bedford <guybedford@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Install URL and URLSearchParams on the global object, like they can be
found in browser environments.
PR-URL: https://github.com/nodejs/node/pull/18281
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Evan Lucas <evanlucas@me.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Use common.busyLoop() which happens to make the test robust.
PR-URL: https://github.com/nodejs/node/pull/18461
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Adds vm.Module, which wraps around ModuleWrap to provide an interface
for developers to work with modules in a more reflective manner.
Co-authored-by: Timothy Gu <timothygu99@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/17560
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Flattens ContextifyContext allows the context interface to be used in
other parts of the code base.
PR-URL: https://github.com/nodejs/node/pull/17560
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Some whitespace was lost when #17489 landed. While I restored the one
file causing the V8-CI to fail, I missed the remaining changes from
Myles. This changes restores all whitespace differences with upstream.
PR-URL: https://github.com/nodejs/node/pull/18366
Refs: https://github.com/nodejs/node/pull/18360
Reviewed-By: Myles Borins <myles.borins@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
It makes more sense to provide instructions on how to update the PR
branch before instructions on pushing the commit.
PR-URL: https://github.com/nodejs/node/pull/18355
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
This adds a mention of `--pending-deprecation` flag effect
in the Deprecated APIs document, and explicitly labels
deprecations that support it.
Refs: https://github.com/nodejs/node/issues/18417
PR-URL: https://github.com/nodejs/node/pull/18433
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com>
Reviewed-By: Rich Trott <rtrott@gmail.com>
