Problem:
Node implements fs.readFile as:
- a call to stat, then
- a C++ -> libuv request to read the entire file using the stat size
Why is this bad?
The effect is to place on the libuv threadpool a potentially-large
read request, occupying the libuv thread until it completes.
While readFile certainly requires buffering the entire file contents,
it can partition the read into smaller buffers
(as is done on other read paths)
along the way to avoid threadpool exhaustion.
If the file is relatively large or stored on a slow medium, reading
the entire file in one shot seems particularly harmful,
and presents a possible DoS vector.
Solution:
Partition the read into multiple smaller requests.
Considerations:
1. Correctness
I don't think partitioning the read like this raises
any additional risk of read-write races on the FS.
If the application is concurrently readFile'ing and modifying the file,
it will already see funny behavior. Though libuv uses preadv where
available, this doesn't guarantee read atomicity in the presence of
concurrent writes.
2. Performance
Downside: Partitioning means that a single large readFile will
require into many "out and back" requests to libuv,
introducing overhead.
Upside: In between each "out and back", other work pending on the
threadpool can take a turn.
In short, although partitioning will slow down a large request,
it will lead to better throughput if the threadpool is handling
more than one type of request.
Fixes: https://github.com/nodejs/node/issues/17047
PR-URL: https://github.com/nodejs/node/pull/17054
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Sakthipriyan Vairamani <thechargingvolcano@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Move functions that deal with `BufferList` to `BufferList.prototype`.
PR-URL: https://github.com/nodejs/node/pull/18353
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Instead of passing along the handle object, just set it as a
property on the stream handle object and let the read handler
grab it from there.
PR-URL: https://github.com/nodejs/node/pull/18334
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Instead of setting individual callbacks on streams and tracking
stream ownership through a boolean `consume_` flag, always have
one specific listener object in charge of a stream, and call
methods on that object rather than generic C-style callbacks.
PR-URL: https://github.com/nodejs/node/pull/18334
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Added tests to clarify the implicit behaviour of array header setting vs
object header setting
PR-URL: https://github.com/nodejs/node/pull/16568
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Returns the promises created by link so that they can be awaited to get
rid of race conditions while resolving and loading modules.
PR-URL: https://github.com/nodejs/node/pull/18394
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
In case no arguments are passed to `assert.ok` it should just
use a default message. Otherwise `assert.ok` can not be used as
a callback.
PR-URL: https://github.com/nodejs/node/pull/18319
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Weijia Wang <starkwang@126.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Install URL and URLSearchParams on the global object, like they can be
found in browser environments.
PR-URL: https://github.com/nodejs/node/pull/18281
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Daijiro Wachi <daijiro.wachi@gmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Evan Lucas <evanlucas@me.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Adds vm.Module, which wraps around ModuleWrap to provide an interface
for developers to work with modules in a more reflective manner.
Co-authored-by: Timothy Gu <timothygu99@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/17560
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Calling `writable.end()` will probably synchronously call
`writable.write()`, in such a situation the `state.ended`
is false and `writable.write()` doesn't trigger `writeAfterEnd()`.
PR-URL: https://github.com/nodejs/node/pull/18170
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Currently, both process.binding and internalBinding have to call into
C++ regardless of whether the module has been cached or not. This
creates significant overhead to all binding calls and unfortunately
the rest of the codebase doesn't really optimize the amount of times
that bindings are required (as an example: 12 files require the
async_wrap binding).
Changing all the usage of this function throughout the codebase would
introduce a lot of churn (and is kind of a hassle) so instead this PR
introduces a JS fast path for both functions for cases where the
binding has already been cached. While micro benchmarks are not super
meaningful here (it's not like we call binding that often...), this
does speed up the cached call by 400%.
In addition, move moduleLoadList creation and management entirely into
JS-land as it requires less code and is more efficient.
PR-URL: https://github.com/nodejs/node/pull/18365
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Move the majority of C++ domain-related code into JS land by introducing
a top level domain callback which handles entering & exiting the domain.
Move the rest of the domain necessities into their own file that creates
an internal binding, to avoid exposing domain-related code on the
process object.
Modify an existing test slightly to better test domain-related code.
PR-URL: https://github.com/nodejs/node/pull/18291
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: James M Snell <jasnell@gmail.com>
The (h|readableH|writableH)ighWaterMark options should only permit
positive numbers and zero.
PR-URL: https://github.com/nodejs/node/pull/18098
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Michaël Zasso <targos@protonmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Removes a few lines of C++ code while making `isIPv4()` about 3x faster.
`isIPv6()` and `isIP()` for the IPv6 case stay about the same.
I removed the homegrown `isIPv4()` in lib/dns.js that utilized a lookup
table. It is in fact a little faster than the new `isIPv4()` function
but:
1. The difference is only measurable at around 10M iterations, and
2. The function is a "probably IPv4" heuristic, not a proper validator.
PR-URL: https://github.com/nodejs/node/pull/18398
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Jan Krems <jan.krems@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
- Allow user to customize why the argument is invalid
- Display the argument with util.inspect so null bytes can be
displayed properly.
PR-URL: https://github.com/nodejs/node/pull/18358
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
Make the listOnTimeout function inline by attaching it to the TimeWrap
prototype.
It improves insertion and cancellation time of unpooled timers by 18%
and 28% respectively.
PR-URL: https://github.com/nodejs/node/pull/18388
Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Weijia Wang <starkwang@126.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Do not check/use unexistent property, use `OutgoingMessage` instead.
PR-URL: https://github.com/nodejs/node/pull/18325
Reviewed-By: Mithun Sasidharan <mithunsasidharan89@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Hidden via a symbol because I'm unsure exactly what the API should look
like in the end.
Removes the need to use _unrefActive for efficiently refreshing
timeouts.
It still uses it under the hood but that could be replaced with
insert() directly if it were in the same file.
PR-URL: https://github.com/nodejs/node/pull/18065
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
PR-URL: https://github.com/nodejs/node/pull/18330
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
PR-URL: https://github.com/nodejs/node/pull/18330
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
Long ago, V8 was much faster switching on string lengths than values.
That is no longer the case, so we can simplify a couple of methods.
PR-URL: https://github.com/nodejs/node/pull/18351
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Kyle Farnung <kfarnung@microsoft.com>
Reviewed-By: Anatoli Papirovski <apapirovski@mac.com>
Reviewed-By: Gireesh Punathil <gpunathi@in.ibm.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Jon Moss <me@jonathanmoss.me>
lchown and lchownSync were opening file descriptors without
closing them. Looks like it has been that way for 7 years.
Does anyone actually use these functions?
PR-URL: https://github.com/nodejs/node/pull/18329
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Evan Lucas <evanlucas@me.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
To avoid a function call `BufferList.prototype.concat()` is not called
when there is only a buffer in the list. That buffer is instead
accessed directly.
PR-URL: https://github.com/nodejs/node/pull/18239
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This makes `assert.ifError` stricter by only accepting `null` and
`undefined` from now on. Before any truthy value was accepted.
PR-URL: https://github.com/nodejs/node/pull/18247
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Destructure the necessary Error classes from internal/errors.
This improves the readability of the error creation.
PR-URL: https://github.com/nodejs/node/pull/18247
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
It is hard to know where ifError is actually triggered due to the
original error being thrown.
This changes it by wrapping the original error in a AssertionError.
This has the positive effect of also making clear that it is indeed
a assertion function that triggered that error.
The original stack can still be accessed by checking the `actual`
property.
PR-URL: https://github.com/nodejs/node/pull/18247
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
The offset was formerly coerced to a integer and this reimplements
that.
PR-URL: https://github.com/nodejs/node/pull/18215
Fixes: https://github.com/nodejs/node/issues/18208
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
