0
0
mirror of https://github.com/nodejs/node.git synced 2024-11-21 21:19:50 +01:00

test_runner: support running tests in process

This commit introduces a new --experimental-test-isolation flag
that, when set to 'none', causes the test runner to execute all
tests in the same process. By default, this is the main test
runner process, but if watch mode is enabled, it spawns a separate
process that runs all of the tests.

The default value of the new flag is 'process', which uses the
existing behavior of running each test file in its own child
process.

It is worth noting that when the isolation mode is 'none', globals
and all other top level logic (such as top level before() and after()
hooks) is shared among all files.

Co-authored-by: Moshe Atlow <moshe@atlow.co.il>
PR-URL: https://github.com/nodejs/node/pull/53927
Fixes: https://github.com/nodejs/node/issues/51548
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
This commit is contained in:
cjihrig 2024-07-13 11:10:59 -04:00 committed by Node.js GitHub Bot
parent 385ca623e4
commit cc26951180
22 changed files with 744 additions and 204 deletions

View File

@ -1091,6 +1091,20 @@ generated as part of the test runner output. If no tests are run, a coverage
report is not generated. See the documentation on
[collecting code coverage from tests][] for more details.
### `--experimental-test-isolation=mode`
<!-- YAML
added: REPLACEME
-->
> Stability: 1.0 - Early development
Configures the type of test isolation used in the test runner. When `mode` is
`'process'`, each test file is run in a separate child process. When `mode` is
`'none'`, all test files run in the same process as the test runner. The default
isolation mode is `'process'`. This flag is ignored if the `--test` flag is not
present. See the [test runner execution model][] section for more information.
### `--experimental-test-module-mocks`
<!-- YAML
@ -2196,7 +2210,9 @@ added:
-->
The maximum number of test files that the test runner CLI will execute
concurrently. The default value is `os.availableParallelism() - 1`.
concurrently. If `--experimental-test-isolation` is set to `'none'`, this flag
is ignored and concurrency is one. Otherwise, concurrency defaults to
`os.availableParallelism() - 1`.
### `--test-coverage-exclude`
@ -2361,7 +2377,7 @@ added: v22.3.0
> Stability: 1.0 - Early development
Regenerates the snapshot file used by the test runner for [snapshot testing][].
Regenerates the snapshot files used by the test runner for [snapshot testing][].
Node.js must be started with the `--experimental-test-snapshots` flag in order
to use this functionality.
@ -3534,6 +3550,7 @@ node --stack-trace-limit=12 -p -e "Error.stackTraceLimit" # prints 12
[snapshot testing]: test.md#snapshot-testing
[syntax detection]: packages.md#syntax-detection
[test reporters]: test.md#test-reporters
[test runner execution model]: test.md#test-runner-execution-model
[timezone IDs]: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
[tracking issue for user-land snapshots]: https://github.com/nodejs/node/issues/44014
[ways that `TZ` is handled in other environments]: https://www.gnu.org/software/libc/manual/html_node/TZ-Variable.html

View File

@ -445,18 +445,26 @@ in the [test runner execution model][] section.
### Test runner execution model
Each matching test file is executed in a separate child process. The maximum
number of child processes running at any time is controlled by the
[`--test-concurrency`][] flag. If the child process finishes with an exit code
of 0, the test is considered passing. Otherwise, the test is considered to be a
failure. Test files must be executable by Node.js, but are not required to use
the `node:test` module internally.
When process-level test isolation is enabled, each matching test file is
executed in a separate child process. The maximum number of child processes
running at any time is controlled by the [`--test-concurrency`][] flag. If the
child process finishes with an exit code of 0, the test is considered passing.
Otherwise, the test is considered to be a failure. Test files must be executable
by Node.js, but are not required to use the `node:test` module internally.
Each test file is executed as if it was a regular script. That is, if the test
file itself uses `node:test` to define tests, all of those tests will be
executed within a single application thread, regardless of the value of the
`concurrency` option of [`test()`][].
When process-level test isolation is disabled, each matching test file is
imported into the test runner process. Once all test files have been loaded, the
top level tests are executed with a concurrency of one. Because the test files
are all run within the same context, it is possible for tests to interact with
each other in ways that are not possible when isolation is enabled. For example,
if a test relies on global state, it is possible for that state to be modified
by a test originating from another file.
## Collecting code coverage
> Stability: 1 - Experimental
@ -933,7 +941,7 @@ the [`--experimental-test-snapshots`][] command-line flag.
Snapshot files are generated by starting Node.js with the
[`--test-update-snapshots`][] command-line flag. A separate snapshot file is
generated for each test file. By default, the snapshot file has the same name
as `process.argv[1]` with a `.snapshot` file extension. This behavior can be
as the test file with a `.snapshot` file extension. This behavior can be
configured using the `snapshot.setResolveSnapshotPath()` function. Each
snapshot assertion corresponds to an export in the snapshot file.
@ -1239,6 +1247,9 @@ added:
- v18.9.0
- v16.19.0
changes:
- version: REPLACEME
pr-url: https://github.com/nodejs/node/pull/53927
description: Added the `isolation` option.
- version: v22.6.0
pr-url: https://github.com/nodejs/node/pull/53866
description: Added the `globPatterns` option.
@ -1274,8 +1285,13 @@ changes:
* `inspectPort` {number|Function} Sets inspector port of test child process.
This can be a number, or a function that takes no arguments and returns a
number. If a nullish value is provided, each process gets its own port,
incremented from the primary's `process.debugPort`.
**Default:** `undefined`.
incremented from the primary's `process.debugPort`. This option is ignored
if the `isolation` option is set to `'none'` as no child processes are
spawned. **Default:** `undefined`.
* `isolation` {string} Configures the type of test isolation. If set to
`'process'`, each test file is run in a separate child process. If set to
`'none'`, all test files run in the current process. **Default:**
`'process'`.
* `only`: {boolean} If truthy, the test context will only run tests that
have the `only` option set
* `setup` {Function} A function that accepts the `TestsStream` instance
@ -1727,9 +1743,9 @@ added: v22.3.0
* `fn` {Function} A function used to compute the location of the snapshot file.
The function receives the path of the test file as its only argument. If the
`process.argv[1]` is not associated with a file (for example in the REPL),
the input is undefined. `fn()` must return a string specifying the location of
the snapshot file.
test is not associated with a file (for example in the REPL), the input is
undefined. `fn()` must return a string specifying the location of the snapshot
snapshot file.
This function is used to customize the location of the snapshot file used for
snapshot testing. By default, the snapshot filename is the same as the entry

View File

@ -185,6 +185,9 @@ Enable the experimental node:sqlite module.
.It Fl -experimental-test-coverage
Enable code coverage in the test runner.
.
.It Fl -experimental-test-isolation Ns = Ns Ar mode
Configures the type of test isolation used in the test runner.
.
.It Fl -experimental-test-module-mocks
Enable module mocking in the test runner.
.

View File

@ -21,7 +21,7 @@ markBootstrapComplete();
const options = parseCommandLine();
if (isUsingInspector()) {
if (isUsingInspector() && options.isolation === 'process') {
process.emitWarning('Using the inspector with --test forces running at a concurrency of 1. ' +
'Use the inspectPort option to run with concurrency');
options.concurrency = 1;

View File

@ -334,4 +334,5 @@ module.exports = {
after: hook('after'),
beforeEach: hook('beforeEach'),
afterEach: hook('afterEach'),
startSubtestAfterBootstrap,
};

View File

@ -11,6 +11,7 @@ const {
ArrayPrototypeMap,
ArrayPrototypePush,
ArrayPrototypeShift,
ArrayPrototypeSlice,
ArrayPrototypeSome,
ArrayPrototypeSort,
ObjectAssign,
@ -24,6 +25,7 @@ const {
StringPrototypeIndexOf,
StringPrototypeSlice,
StringPrototypeStartsWith,
Symbol,
TypedArrayPrototypeGetLength,
TypedArrayPrototypeSubarray,
} = primordials;
@ -44,18 +46,28 @@ const {
ERR_TEST_FAILURE,
},
} = require('internal/errors');
const esmLoader = require('internal/modules/esm/loader');
const {
validateArray,
validateBoolean,
validateFunction,
validateObject,
validateOneOf,
validateInteger,
} = require('internal/validators');
const { getInspectPort, isUsingInspector, isInspectorMessage } = require('internal/util/inspector');
const { isRegExp } = require('internal/util/types');
const { kEmptyObject } = require('internal/util');
const { pathToFileURL } = require('internal/url');
const {
createDeferredPromise,
getCWDURL,
kEmptyObject,
} = require('internal/util');
const { kEmitMessage } = require('internal/test_runner/tests_stream');
const { createTestTree } = require('internal/test_runner/harness');
const {
createTestTree,
startSubtestAfterBootstrap,
} = require('internal/test_runner/harness');
const {
kAborted,
kCancelledByParent,
@ -77,7 +89,11 @@ const {
triggerUncaughtException,
exitCodes: { kGenericUserError },
} = internalBinding('errors');
let debug = require('internal/util/debuglog').debuglog('test_runner', (fn) => {
debug = fn;
});
const kIsolatedProcessName = Symbol('kIsolatedProcessName');
const kFilterArgs = ['--test', '--experimental-test-coverage', '--watch'];
const kFilterArgValues = ['--test-reporter', '--test-reporter-destination'];
const kDiagnosticsFilterArgs = ['tests', 'suites', 'pass', 'fail', 'cancelled', 'skipped', 'todo', 'duration_ms'];
@ -130,7 +146,12 @@ function getRunArgs(path, { forceExit, inspectPort, testNamePatterns, testSkipPa
if (only === true) {
ArrayPrototypePush(argv, '--test-only');
}
ArrayPrototypePush(argv, path);
if (path === kIsolatedProcessName) {
ArrayPrototypePush(argv, '--test', ...ArrayPrototypeSlice(process.argv, 1));
} else {
ArrayPrototypePush(argv, path);
}
return argv;
}
@ -326,7 +347,9 @@ class FileTest extends Test {
function runTestFile(path, filesWatcher, opts) {
const watchMode = filesWatcher != null;
const subtest = opts.root.createSubtest(FileTest, path, { __proto__: null, signal: opts.signal }, async (t) => {
const testPath = path === kIsolatedProcessName ? '' : path;
const testOpts = { __proto__: null, signal: opts.signal };
const subtest = opts.root.createSubtest(FileTest, testPath, testOpts, async (t) => {
const args = getRunArgs(path, opts);
const stdio = ['pipe', 'pipe', 'pipe'];
const env = { __proto__: null, ...process.env, NODE_TEST_CONTEXT: 'child-v8' };
@ -418,10 +441,23 @@ function watchFiles(testFiles, opts) {
const filesWatcher = { __proto__: null, watcher, runningProcesses, runningSubtests };
opts.root.harness.watching = true;
async function restartTestFile(file) {
const runningProcess = runningProcesses.get(file);
if (runningProcess) {
runningProcess.kill();
await once(runningProcess, 'exit');
}
if (!runningSubtests.size) {
// Reset the topLevel counter
opts.root.harness.counters.topLevel = 0;
}
await runningSubtests.get(file);
runningSubtests.set(file, runTestFile(file, filesWatcher, opts));
}
watcher.on('changed', ({ owners, eventType }) => {
if (!opts.hasFiles && eventType === 'rename') {
const updatedTestFiles = createTestFileList(opts.globPatterns);
const newFileName = ArrayPrototypeFind(updatedTestFiles, (x) => !ArrayPrototypeIncludes(testFiles, x));
const previousFileName = ArrayPrototypeFind(testFiles, (x) => !ArrayPrototypeIncludes(updatedTestFiles, x));
@ -439,25 +475,22 @@ function watchFiles(testFiles, opts) {
}
watcher.unfilterFilesOwnedBy(owners);
PromisePrototypeThen(SafePromiseAllReturnVoid(testFiles, async (file) => {
if (!owners.has(file)) {
return;
}
const runningProcess = runningProcesses.get(file);
if (runningProcess) {
runningProcess.kill();
await once(runningProcess, 'exit');
}
if (!runningSubtests.size) {
// Reset the topLevel counter
opts.root.harness.counters.topLevel = 0;
}
await runningSubtests.get(file);
runningSubtests.set(file, runTestFile(file, filesWatcher, opts));
}, undefined, (error) => {
triggerUncaughtException(error, true /* fromPromise */);
}));
if (opts.isolation === 'none') {
PromisePrototypeThen(restartTestFile(kIsolatedProcessName), undefined, (error) => {
triggerUncaughtException(error, true /* fromPromise */);
});
} else {
watcher.unfilterFilesOwnedBy(owners);
PromisePrototypeThen(SafePromiseAllReturnVoid(testFiles, async (file) => {
if (!owners.has(file)) {
return;
}
await restartTestFile(file);
}, undefined, (error) => {
triggerUncaughtException(error, true /* fromPromise */);
}));
}
});
if (opts.signal) {
kResistStopPropagation ??= require('internal/event_target').kResistStopPropagation;
@ -485,6 +518,7 @@ function run(options = kEmptyObject) {
files,
forceExit,
inspectPort,
isolation = 'process',
watch,
setup,
only,
@ -566,6 +600,7 @@ function run(options = kEmptyObject) {
throw new ERR_INVALID_ARG_TYPE(name, ['string', 'RegExp'], value);
});
}
validateOneOf(isolation, 'options.isolation', ['process', 'none']);
const rootTestOptions = { __proto__: null, concurrency, timeout, signal };
const globalOptions = {
@ -576,21 +611,16 @@ function run(options = kEmptyObject) {
setup, // This line can be removed when parseCommandLine() is removed here.
};
const root = createTestTree(rootTestOptions, globalOptions);
if (process.env.NODE_TEST_CONTEXT !== undefined) {
process.emitWarning('node:test run() is being called recursively within a test file. skipping running files.');
root.postRun();
return root.reporter;
}
let testFiles = files ?? createTestFileList(globPatterns);
if (shard) {
testFiles = ArrayPrototypeFilter(testFiles, (_, index) => index % shard.total === shard.index - 1);
}
let postRun = () => root.postRun();
let teardown = () => root.harness.teardown();
let teardown;
let postRun;
let filesWatcher;
let runFiles;
const opts = {
__proto__: null,
root,
@ -602,21 +632,95 @@ function run(options = kEmptyObject) {
globPatterns,
only,
forceExit,
isolation,
};
if (watch) {
filesWatcher = watchFiles(testFiles, opts);
postRun = undefined;
teardown = undefined;
if (isolation === 'process') {
if (process.env.NODE_TEST_CONTEXT !== undefined) {
process.emitWarning('node:test run() is being called recursively within a test file. skipping running files.');
root.postRun();
return root.reporter;
}
if (watch) {
filesWatcher = watchFiles(testFiles, opts);
} else {
postRun = () => root.postRun();
teardown = () => root.harness.teardown();
}
runFiles = () => {
root.harness.bootstrapPromise = null;
root.harness.buildPromise = null;
return SafePromiseAllSettledReturnVoid(testFiles, (path) => {
const subtest = runTestFile(path, filesWatcher, opts);
filesWatcher?.runningSubtests.set(path, subtest);
return subtest;
});
};
} else if (isolation === 'none') {
if (watch) {
filesWatcher = watchFiles(testFiles, opts);
runFiles = async () => {
root.harness.bootstrapPromise = null;
root.harness.buildPromise = null;
const subtest = runTestFile(kIsolatedProcessName, filesWatcher, opts);
filesWatcher?.runningSubtests.set(kIsolatedProcessName, subtest);
return subtest;
};
} else {
runFiles = async () => {
const { promise, resolve: finishBootstrap } = createDeferredPromise();
await root.runInAsyncScope(async () => {
const parentURL = getCWDURL().href;
const cascadedLoader = esmLoader.getOrInitializeCascadedLoader();
let topLevelTestCount = 0;
root.harness.bootstrapPromise = promise;
for (let i = 0; i < testFiles.length; ++i) {
const testFile = testFiles[i];
const fileURL = pathToFileURL(testFile);
const parent = i === 0 ? undefined : parentURL;
let threw = false;
let importError;
root.entryFile = resolve(testFile);
debug('loading test file:', fileURL.href);
try {
await cascadedLoader.import(fileURL, parent, { __proto__: null });
} catch (err) {
threw = true;
importError = err;
}
debug(
'loaded "%s": top level test count before = %d and after = %d',
testFile,
topLevelTestCount,
root.subtests.length,
);
if (topLevelTestCount === root.subtests.length) {
// This file had no tests in it. Add the placeholder test.
const subtest = root.createSubtest(Test, testFile);
if (threw) {
subtest.fail(importError);
}
startSubtestAfterBootstrap(subtest);
}
topLevelTestCount = root.subtests.length;
}
});
debug('beginning test execution');
root.entryFile = null;
finishBootstrap();
root.processPendingSubtests();
};
}
}
const runFiles = () => {
root.harness.bootstrapPromise = null;
root.harness.buildPromise = null;
return SafePromiseAllSettledReturnVoid(testFiles, (path) => {
const subtest = runTestFile(path, filesWatcher, opts);
filesWatcher?.runningSubtests.set(path, subtest);
return subtest;
});
};
const setupPromise = PromiseResolve(setup?.(root.reporter));
PromisePrototypeThen(PromisePrototypeThen(PromisePrototypeThen(setupPromise, runFiles), postRun), teardown);

View File

@ -195,11 +195,12 @@ function parseCommandLine() {
let coverageExcludeGlobs;
let coverageIncludeGlobs;
let destinations;
let only;
let isolation;
let only = getOptionValue('--test-only');
let reporters;
let shard;
let testNamePatterns;
let testSkipPatterns;
let testNamePatterns = mapPatternFlagToRegExArray('--test-name-pattern');
let testSkipPatterns = mapPatternFlagToRegExArray('--test-skip-pattern');
let timeout;
if (isChildProcessV8) {
@ -230,10 +231,17 @@ function parseCommandLine() {
}
if (isTestRunner) {
isolation = getOptionValue('--experimental-test-isolation');
timeout = getOptionValue('--test-timeout') || Infinity;
concurrency = getOptionValue('--test-concurrency') || true;
only = false;
testNamePatterns = null;
if (isolation === 'none') {
concurrency = 1;
} else {
concurrency = getOptionValue('--test-concurrency') || true;
only = false;
testNamePatterns = null;
testSkipPatterns = null;
}
const shardOption = getOptionValue('--test-shard');
if (shardOption) {
@ -290,6 +298,7 @@ function parseCommandLine() {
coverageIncludeGlobs,
destinations,
forceExit,
isolation,
only,
reporters,
setup,
@ -305,6 +314,16 @@ function parseCommandLine() {
return globalTestOptions;
}
function mapPatternFlagToRegExArray(flagName) {
const patterns = getOptionValue(flagName);
if (patterns?.length > 0) {
return ArrayPrototypeMap(patterns, (re) => convertStringToRegExp(re, flagName));
}
return null;
}
function countCompletedTest(test, harness = test.root.harness) {
if (test.nesting === 0) {
harness.counters.topLevel++;

View File

@ -645,7 +645,8 @@ inline bool Environment::owns_inspector() const {
inline bool Environment::should_create_inspector() const {
return (flags_ & EnvironmentFlags::kNoCreateInspector) == 0 &&
!options_->test_runner && !options_->watch_mode;
!(options_->test_runner && options_->test_isolation == "process") &&
!options_->watch_mode;
}
inline bool Environment::should_wait_for_inspector_frontend() const {

View File

@ -143,6 +143,18 @@ void EnvironmentOptions::CheckOptions(std::vector<std::string>* errors,
}
if (test_runner) {
if (test_isolation == "none") {
debug_options_.allow_attaching_debugger = true;
} else {
if (test_isolation != "process") {
errors->push_back("invalid value for --experimental-test-isolation");
}
#ifndef ALLOW_ATTACHING_DEBUGGER_IN_TEST_RUNNER
debug_options_.allow_attaching_debugger = false;
#endif
}
if (syntax_check_only) {
errors->push_back("either --test or --check can be used, not both");
}
@ -159,10 +171,6 @@ void EnvironmentOptions::CheckOptions(std::vector<std::string>* errors,
errors->push_back(
"--watch-path cannot be used in combination with --test");
}
#ifndef ALLOW_ATTACHING_DEBUGGER_IN_TEST_RUNNER
debug_options_.allow_attaching_debugger = false;
#endif
}
if (watch_mode) {
@ -650,6 +658,9 @@ EnvironmentOptionsParser::EnvironmentOptionsParser() {
AddOption("--experimental-test-coverage",
"enable code coverage in the test runner",
&EnvironmentOptions::test_runner_coverage);
AddOption("--experimental-test-isolation",
"configures the type of test isolation used in the test runner",
&EnvironmentOptions::test_isolation);
AddOption("--experimental-test-module-mocks",
"enable module mocking in the test runner",
&EnvironmentOptions::test_runner_module_mocks);

View File

@ -191,6 +191,7 @@ class EnvironmentOptions : public Options {
std::vector<std::string> test_reporter_destination;
bool test_only = false;
bool test_udp_no_try_send = false;
std::string test_isolation = "process";
std::string test_shard;
std::vector<std::string> test_skip_pattern;
std::vector<std::string> coverage_include_pattern;

View File

@ -0,0 +1,32 @@
'use strict';
const { before, beforeEach, after, afterEach, test, suite } = require('node:test');
globalThis.GLOBAL_ORDER = [];
before(function() {
GLOBAL_ORDER.push(`before one: ${this.name}`);
});
beforeEach(function() {
GLOBAL_ORDER.push(`beforeEach one: ${this.name}`);
});
after(function() {
GLOBAL_ORDER.push(`after one: ${this.name}`);
});
afterEach(function() {
GLOBAL_ORDER.push(`afterEach one: ${this.name}`);
});
suite('suite one', function() {
GLOBAL_ORDER.push(this.name);
test('suite one - test', { only: true }, function() {
GLOBAL_ORDER.push(this.name);
});
});
test('test one', function() {
GLOBAL_ORDER.push(this.name);
});

View File

@ -0,0 +1,30 @@
'use strict';
const { before, beforeEach, after, afterEach, test, suite } = require('node:test');
before(function() {
GLOBAL_ORDER.push(`before two: ${this.name}`);
});
beforeEach(function() {
GLOBAL_ORDER.push(`beforeEach two: ${this.name}`);
});
after(function() {
GLOBAL_ORDER.push(`after two: ${this.name}`);
});
afterEach(function() {
GLOBAL_ORDER.push(`afterEach two: ${this.name}`);
});
suite('suite two', function() {
GLOBAL_ORDER.push(this.name);
before(function() {
GLOBAL_ORDER.push(`before suite two: ${this.name}`);
});
test('suite two - test', { only: true }, function() {
GLOBAL_ORDER.push(this.name);
});
});

View File

@ -0,0 +1,11 @@
'use strict';
const { snapshot, test } = require('node:test');
const { basename, join } = require('node:path');
snapshot.setResolveSnapshotPath((testFile) => {
return join(process.cwd(), `${basename(testFile)}.snapshot`);
});
test('has a snapshot', (t) => {
t.assert.snapshot('a snapshot from ' + __filename);
});

View File

@ -24,3 +24,17 @@ test('concurrency of two', async () => {
const cp = spawnSync(process.execPath, args, { cwd, env });
assert.match(cp.stderr.toString(), /concurrency: 2,/);
});
test('isolation=none uses a concurrency of one', async () => {
const args = ['--test', '--experimental-test-isolation=none'];
const cp = spawnSync(process.execPath, args, { cwd, env });
assert.match(cp.stderr.toString(), /concurrency: 1,/);
});
test('isolation=none overrides --test-concurrency', async () => {
const args = [
'--test', '--experimental-test-isolation=none', '--test-concurrency=2',
];
const cp = spawnSync(process.execPath, args, { cwd, env });
assert.match(cp.stderr.toString(), /concurrency: 1,/);
});

View File

@ -18,3 +18,11 @@ test('timeout of 10ms', async () => {
const cp = spawnSync(process.execPath, args, { cwd, env });
assert.match(cp.stderr.toString(), /timeout: 10,/);
});
test('isolation=none uses the --test-timeout flag', async () => {
const args = [
'--test', '--experimental-test-isolation=none', '--test-timeout=10',
];
const cp = spawnSync(process.execPath, args, { cwd, env });
assert.match(cp.stderr.toString(), /timeout: 10,/);
});

View File

@ -7,92 +7,162 @@ const { join } = require('path');
const fixtures = require('../common/fixtures');
const testFixtures = fixtures.path('test-runner');
{
// File not found.
const args = ['--test', 'a-random-file-that-does-not-exist.js'];
const child = spawnSync(process.execPath, args);
for (const isolation of ['none', 'process']) {
{
// File not found.
const args = [
'--test',
`--experimental-test-isolation=${isolation}`,
'a-random-file-that-does-not-exist.js',
];
const child = spawnSync(process.execPath, args);
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
assert.strictEqual(child.stdout.toString(), '');
assert.match(child.stderr.toString(), /^Could not find/);
}
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
assert.strictEqual(child.stdout.toString(), '');
assert.match(child.stderr.toString(), /^Could not find/);
}
{
// Default behavior. node_modules is ignored. Files that don't match the
// pattern are ignored except in test/ directories.
const args = ['--test'];
const child = spawnSync(process.execPath, args, { cwd: join(testFixtures, 'default-behavior') });
{
// Default behavior. node_modules is ignored. Files that don't match the
// pattern are ignored except in test/ directories.
const args = ['--test', `--experimental-test-isolation=${isolation}`];
const child = spawnSync(process.execPath, args, { cwd: join(testFixtures, 'default-behavior') });
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
assert.strictEqual(child.stderr.toString(), '');
const stdout = child.stdout.toString();
assert.match(stdout, /ok 1 - this should pass/);
assert.match(stdout, /not ok 2 - this should fail/);
assert.match(stdout, /ok 3 - subdir.+subdir_test\.js/);
assert.match(stdout, /ok 4 - this should pass/);
assert.match(stdout, /ok 5 - this should be skipped/);
assert.match(stdout, /ok 6 - this should be executed/);
}
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
assert.strictEqual(child.stderr.toString(), '');
const stdout = child.stdout.toString();
{
// Same but with a prototype mutation in require scripts.
const args = ['--require', join(testFixtures, 'protoMutation.js'), '--test'];
const child = spawnSync(process.execPath, args, { cwd: join(testFixtures, 'default-behavior') });
assert.match(stdout, /ok 1 - this should pass/);
assert.match(stdout, /not ok 2 - this should fail/);
assert.match(stdout, /ok 3 - subdir.+subdir_test\.js/);
assert.match(stdout, /ok 4 - this should pass/);
assert.match(stdout, /ok 5 - this should be skipped/);
assert.match(stdout, /ok 6 - this should be executed/);
}
const stdout = child.stdout.toString();
assert.match(stdout, /ok 1 - this should pass/);
assert.match(stdout, /not ok 2 - this should fail/);
assert.match(stdout, /ok 3 - subdir.+subdir_test\.js/);
assert.match(stdout, /ok 4 - this should pass/);
assert.match(stdout, /ok 5 - this should be skipped/);
assert.match(stdout, /ok 6 - this should be executed/);
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
assert.strictEqual(child.stderr.toString(), '');
}
{
// Same but with a prototype mutation in require scripts.
const args = [
'--require', join(testFixtures, 'protoMutation.js'),
'--test',
`--experimental-test-isolation=${isolation}`,
];
const child = spawnSync(process.execPath, args, { cwd: join(testFixtures, 'default-behavior') });
{
// User specified files that don't match the pattern are still run.
const args = ['--test', join(testFixtures, 'index.js')];
const child = spawnSync(process.execPath, args, { cwd: testFixtures });
const stdout = child.stdout.toString();
assert.match(stdout, /ok 1 - this should pass/);
assert.match(stdout, /not ok 2 - this should fail/);
assert.match(stdout, /ok 3 - subdir.+subdir_test\.js/);
assert.match(stdout, /ok 4 - this should pass/);
assert.match(stdout, /ok 5 - this should be skipped/);
assert.match(stdout, /ok 6 - this should be executed/);
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
assert.strictEqual(child.stderr.toString(), '');
}
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
assert.strictEqual(child.stderr.toString(), '');
const stdout = child.stdout.toString();
assert.match(stdout, /not ok 1 - .+index\.js/);
}
{
// User specified files that don't match the pattern are still run.
const args = [
'--test',
`--experimental-test-isolation=${isolation}`,
join(testFixtures, 'index.js'),
];
const child = spawnSync(process.execPath, args, { cwd: testFixtures });
{
// Searches node_modules if specified.
const args = ['--test', join(testFixtures, 'default-behavior/node_modules/*.js')];
const child = spawnSync(process.execPath, args);
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
assert.strictEqual(child.stderr.toString(), '');
const stdout = child.stdout.toString();
assert.match(stdout, /not ok 1 - .+index\.js/);
}
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
assert.strictEqual(child.stderr.toString(), '');
const stdout = child.stdout.toString();
assert.match(stdout, /not ok 1 - .+test-nm\.js/);
}
{
// Searches node_modules if specified.
const args = [
'--test',
`--experimental-test-isolation=${isolation}`,
join(testFixtures, 'default-behavior/node_modules/*.js'),
];
const child = spawnSync(process.execPath, args);
{
// The current directory is used by default.
const args = ['--test'];
const options = { cwd: join(testFixtures, 'default-behavior') };
const child = spawnSync(process.execPath, args, options);
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
assert.strictEqual(child.stderr.toString(), '');
const stdout = child.stdout.toString();
assert.match(stdout, /not ok 1 - .+test-nm\.js/);
}
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
assert.strictEqual(child.stderr.toString(), '');
const stdout = child.stdout.toString();
assert.match(stdout, /ok 1 - this should pass/);
assert.match(stdout, /not ok 2 - this should fail/);
assert.match(stdout, /ok 3 - subdir.+subdir_test\.js/);
assert.match(stdout, /ok 4 - this should pass/);
assert.match(stdout, /ok 5 - this should be skipped/);
assert.match(stdout, /ok 6 - this should be executed/);
{
// The current directory is used by default.
const args = ['--test', `--experimental-test-isolation=${isolation}`];
const options = { cwd: join(testFixtures, 'default-behavior') };
const child = spawnSync(process.execPath, args, options);
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
assert.strictEqual(child.stderr.toString(), '');
const stdout = child.stdout.toString();
assert.match(stdout, /ok 1 - this should pass/);
assert.match(stdout, /not ok 2 - this should fail/);
assert.match(stdout, /ok 3 - subdir.+subdir_test\.js/);
assert.match(stdout, /ok 4 - this should pass/);
assert.match(stdout, /ok 5 - this should be skipped/);
assert.match(stdout, /ok 6 - this should be executed/);
}
{
// Test combined stream outputs
const args = [
'--test',
`--experimental-test-isolation=${isolation}`,
'test/fixtures/test-runner/default-behavior/index.test.js',
'test/fixtures/test-runner/nested.js',
'test/fixtures/test-runner/invalid-tap.js',
];
const child = spawnSync(process.execPath, args);
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
assert.strictEqual(child.stderr.toString(), '');
const stdout = child.stdout.toString();
assert.match(stdout, /# Subtest: this should pass/);
assert.match(stdout, /ok 1 - this should pass/);
assert.match(stdout, / {2}---/);
assert.match(stdout, / {2}duration_ms: .*/);
assert.match(stdout, / {2}\.\.\./);
assert.match(stdout, /# Subtest: .+invalid-tap\.js/);
assert.match(stdout, /invalid tap output/);
assert.match(stdout, /ok 2 - .+invalid-tap\.js/);
assert.match(stdout, /# Subtest: level 0a/);
assert.match(stdout, / {4}# Subtest: level 1a/);
assert.match(stdout, / {4}ok 1 - level 1a/);
assert.match(stdout, / {4}# Subtest: level 1b/);
assert.match(stdout, / {4}not ok 2 - level 1b/);
assert.match(stdout, / {6}code: 'ERR_TEST_FAILURE'/);
assert.match(stdout, / {6}stack: |-'/);
assert.match(stdout, / {8}TestContext\.<anonymous> .*/);
assert.match(stdout, / {4}# Subtest: level 1c/);
assert.match(stdout, / {4}ok 3 - level 1c # SKIP aaa/);
assert.match(stdout, / {4}# Subtest: level 1d/);
assert.match(stdout, / {4}ok 4 - level 1d/);
assert.match(stdout, /not ok 3 - level 0a/);
assert.match(stdout, / {2}error: '1 subtest failed'/);
assert.match(stdout, /# Subtest: level 0b/);
assert.match(stdout, /not ok 4 - level 0b/);
assert.match(stdout, / {2}error: 'level 0b error'/);
assert.match(stdout, /# tests 8/);
assert.match(stdout, /# suites 0/);
assert.match(stdout, /# pass 4/);
assert.match(stdout, /# fail 3/);
assert.match(stdout, /# cancelled 0/);
assert.match(stdout, /# skipped 1/);
assert.match(stdout, /# todo 0/);
}
}
{
@ -115,57 +185,6 @@ const testFixtures = fixtures.path('test-runner');
}
}
{
// Test combined stream outputs
const args = [
'--test',
'test/fixtures/test-runner/default-behavior/index.test.js',
'test/fixtures/test-runner/nested.js',
'test/fixtures/test-runner/invalid-tap.js',
];
const child = spawnSync(process.execPath, args);
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
assert.strictEqual(child.stderr.toString(), '');
const stdout = child.stdout.toString();
assert.match(stdout, /# Subtest: this should pass/);
assert.match(stdout, /ok 1 - this should pass/);
assert.match(stdout, / {2}---/);
assert.match(stdout, / {2}duration_ms: .*/);
assert.match(stdout, / {2}\.\.\./);
assert.match(stdout, /# Subtest: .+invalid-tap\.js/);
assert.match(stdout, /# invalid tap output/);
assert.match(stdout, /ok 2 - .+invalid-tap\.js/);
assert.match(stdout, /# Subtest: level 0a/);
assert.match(stdout, / {4}# Subtest: level 1a/);
assert.match(stdout, / {4}ok 1 - level 1a/);
assert.match(stdout, / {4}# Subtest: level 1b/);
assert.match(stdout, / {4}not ok 2 - level 1b/);
assert.match(stdout, / {6}code: 'ERR_TEST_FAILURE'/);
assert.match(stdout, / {6}stack: |-'/);
assert.match(stdout, / {8}TestContext\.<anonymous> .*/);
assert.match(stdout, / {4}# Subtest: level 1c/);
assert.match(stdout, / {4}ok 3 - level 1c # SKIP aaa/);
assert.match(stdout, / {4}# Subtest: level 1d/);
assert.match(stdout, / {4}ok 4 - level 1d/);
assert.match(stdout, /not ok 3 - level 0a/);
assert.match(stdout, / {2}error: '1 subtest failed'/);
assert.match(stdout, /# Subtest: level 0b/);
assert.match(stdout, /not ok 4 - level 0b/);
assert.match(stdout, / {2}error: 'level 0b error'/);
assert.match(stdout, /# tests 8/);
assert.match(stdout, /# suites 0/);
assert.match(stdout, /# pass 4/);
assert.match(stdout, /# fail 3/);
assert.match(stdout, /# cancelled 0/);
assert.match(stdout, /# skipped 1/);
assert.match(stdout, /# todo 0/);
}
{
// Test user logging in tests.
const args = [

View File

@ -187,6 +187,44 @@ test('coverage is combined for multiple processes', skipIfNoInspector, () => {
assert.strictEqual(result.status, 0);
});
test('coverage works with isolation=none', skipIfNoInspector, () => {
// There is a bug in coverage calculation. The branch % in the common.js
// fixture is different depending on the test isolation mode. The 'none' mode
// is closer to what c8 reports here, so the bug is likely in the code that
// merges reports from different processes.
let report = [
'# start of coverage report',
'# -------------------------------------------------------------------',
'# file | line % | branch % | funcs % | uncovered lines',
'# -------------------------------------------------------------------',
'# common.js | 89.86 | 68.42 | 100.00 | 8 13-14 18 34-35 53',
'# first.test.js | 83.33 | 100.00 | 50.00 | 5-6',
'# second.test.js | 100.00 | 100.00 | 100.00 | ',
'# third.test.js | 100.00 | 100.00 | 100.00 | ',
'# -------------------------------------------------------------------',
'# all files | 92.11 | 76.00 | 88.89 |',
'# -------------------------------------------------------------------',
'# end of coverage report',
].join('\n');
if (common.isWindows) {
report = report.replaceAll('/', '\\');
}
const fixture = fixtures.path('v8-coverage', 'combined_coverage');
const args = [
'--test', '--experimental-test-coverage', '--test-reporter', 'tap', '--experimental-test-isolation=none',
];
const result = spawnSync(process.execPath, args, {
env: { ...process.env, NODE_TEST_TMPDIR: tmpdir.path },
cwd: fixture,
});
assert.strictEqual(result.stderr.toString(), '');
assert(result.stdout.toString().includes(report));
assert.strictEqual(result.status, 0);
});
test('coverage reports on lines, functions, and branches', skipIfNoInspector, async (t) => {
const fixture = fixtures.path('test-runner', 'coverage.js');
const child = spawnSync(process.execPath,

View File

@ -48,3 +48,21 @@ const { spawnSync } = require('child_process');
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
}
{
const child = spawnSync(process.execPath, [
'--test',
'--experimental-test-isolation=none',
fixtures.path('test-runner', 'async-error-in-test-hook.mjs'),
]);
const stdout = child.stdout.toString();
assert.match(stdout, /^# Error: Test hook "before" at .+async-error-in-test-hook\.mjs:3:1 generated asynchronous activity after the test ended/m);
assert.match(stdout, /^# Error: Test hook "beforeEach" at .+async-error-in-test-hook\.mjs:9:1 generated asynchronous activity after the test ended/m);
assert.match(stdout, /^# Error: Test hook "after" at .+async-error-in-test-hook\.mjs:15:1 generated asynchronous activity after the test ended/m);
assert.match(stdout, /^# Error: Test hook "afterEach" at .+async-error-in-test-hook\.mjs:21:1 generated asynchronous activity after the test ended/m);
assert.match(stdout, /^# pass 1$/m);
assert.match(stdout, /^# fail 0$/m);
assert.match(stdout, /^# cancelled 0$/m);
assert.strictEqual(child.status, 1);
assert.strictEqual(child.signal, null);
}

View File

@ -4,12 +4,21 @@ const { match, doesNotMatch, strictEqual } = require('node:assert');
const { spawnSync } = require('node:child_process');
const fixtures = require('../common/fixtures');
const fixture = fixtures.path('test-runner/throws_sync_and_async.js');
const r = spawnSync(process.execPath, ['--test', '--test-force-exit', fixture]);
strictEqual(r.status, 1);
strictEqual(r.signal, null);
strictEqual(r.stderr.toString(), '');
for (const isolation of ['none', 'process']) {
const args = [
'--test',
'--test-force-exit',
`--experimental-test-isolation=${isolation}`,
fixture,
];
const r = spawnSync(process.execPath, args);
const stdout = r.stdout.toString();
match(stdout, /error: 'fails'/);
doesNotMatch(stdout, /this should not have a chance to be thrown/);
strictEqual(r.status, 1);
strictEqual(r.signal, null);
strictEqual(r.stderr.toString(), '');
const stdout = r.stdout.toString();
match(stdout, /error: 'fails'/);
doesNotMatch(stdout, /this should not have a chance to be thrown/);
}

View File

@ -0,0 +1,69 @@
'use strict';
require('../common');
const fixtures = require('../common/fixtures');
const assert = require('node:assert');
const { spawnSync } = require('node:child_process');
const { test } = require('node:test');
const fixture1 = fixtures.path('test-runner', 'no-isolation', 'one.test.js');
const fixture2 = fixtures.path('test-runner', 'no-isolation', 'two.test.js');
test('works with --test-only', () => {
const args = [
'--test',
'--experimental-test-isolation=none',
'--test-only',
fixture1,
fixture2,
];
const child = spawnSync(process.execPath, args);
const stdout = child.stdout.toString();
assert.strictEqual(child.status, 0);
assert.strictEqual(child.signal, null);
assert.match(stdout, /# tests 2/);
assert.match(stdout, /# suites 2/);
assert.match(stdout, /# pass 2/);
assert.match(stdout, /ok 1 - suite one/);
assert.match(stdout, /ok 1 - suite one - test/);
assert.match(stdout, /ok 2 - suite two/);
assert.match(stdout, /ok 1 - suite two - test/);
});
test('works with --test-name-pattern', () => {
const args = [
'--test',
'--experimental-test-isolation=none',
'--test-name-pattern=/test one/',
fixture1,
fixture2,
];
const child = spawnSync(process.execPath, args);
const stdout = child.stdout.toString();
assert.strictEqual(child.status, 0);
assert.strictEqual(child.signal, null);
assert.match(stdout, /# tests 1/);
assert.match(stdout, /# suites 0/);
assert.match(stdout, /# pass 1/);
assert.match(stdout, /ok 1 - test one/);
});
test('works with --test-skip-pattern', () => {
const args = [
'--test',
'--experimental-test-isolation=none',
'--test-skip-pattern=/one/',
fixture1,
fixture2,
];
const child = spawnSync(process.execPath, args);
const stdout = child.stdout.toString();
assert.strictEqual(child.status, 0);
assert.strictEqual(child.signal, null);
assert.match(stdout, /# tests 1/);
assert.match(stdout, /# suites 1/);
assert.match(stdout, /# pass 1/);
assert.match(stdout, /ok 1 - suite two - test/);
});

View File

@ -0,0 +1,47 @@
import { allowGlobals, mustCall, mustNotCall } from '../common/index.mjs';
import * as fixtures from '../common/fixtures.mjs';
import { deepStrictEqual } from 'node:assert';
import { run } from 'node:test';
const stream = run({
files: [
fixtures.path('test-runner', 'no-isolation', 'one.test.js'),
fixtures.path('test-runner', 'no-isolation', 'two.test.js'),
],
isolation: 'none',
});
stream.on('test:fail', mustNotCall());
stream.on('test:pass', mustCall(5));
// eslint-disable-next-line no-unused-vars
for await (const _ of stream);
allowGlobals(globalThis.GLOBAL_ORDER);
deepStrictEqual(globalThis.GLOBAL_ORDER, [
'before one: <root>',
'suite one',
'before two: <root>',
'suite two',
'beforeEach one: suite one - test',
'beforeEach two: suite one - test',
'suite one - test',
'afterEach one: suite one - test',
'afterEach two: suite one - test',
'beforeEach one: test one',
'beforeEach two: test one',
'test one',
'afterEach one: test one',
'afterEach two: test one',
'before suite two: suite two',
'beforeEach one: suite two - test',
'beforeEach two: suite two - test',
'suite two - test',
'afterEach one: suite two - test',
'afterEach two: suite two - test',
'after one: <root>',
'after two: <root>',
]);

View File

@ -339,3 +339,75 @@ test('t.assert.snapshot()', async (t) => {
t.assert.match(child.stdout, /fail 0/);
});
});
test('snapshots from multiple files (isolation=none)', async (t) => {
tmpdir.refresh();
const fixture = fixtures.path('test-runner', 'snapshots', 'unit.js');
const fixture2 = fixtures.path('test-runner', 'snapshots', 'unit-2.js');
await t.test('fails prior to snapshot generation', async (t) => {
const args = [
'--test',
'--experimental-test-isolation=none',
'--experimental-test-snapshots',
fixture,
fixture2,
];
const child = await common.spawnPromisified(
process.execPath,
args,
{ cwd: tmpdir.path },
);
t.assert.strictEqual(child.code, 1);
t.assert.strictEqual(child.signal, null);
t.assert.match(child.stdout, /# tests 6/);
t.assert.match(child.stdout, /# pass 0/);
t.assert.match(child.stdout, /# fail 6/);
t.assert.match(child.stdout, /Missing snapshots/);
});
await t.test('passes when regenerating snapshots', async (t) => {
const args = [
'--test',
'--experimental-test-isolation=none',
'--experimental-test-snapshots',
'--test-update-snapshots',
fixture,
fixture2,
];
const child = await common.spawnPromisified(
process.execPath,
args,
{ cwd: tmpdir.path },
);
t.assert.strictEqual(child.code, 0);
t.assert.strictEqual(child.signal, null);
t.assert.match(child.stdout, /tests 6/);
t.assert.match(child.stdout, /pass 6/);
t.assert.match(child.stdout, /fail 0/);
});
await t.test('passes when snapshots exist', async (t) => {
const args = [
'--test',
'--experimental-test-isolation=none',
'--experimental-test-snapshots',
fixture,
fixture2,
];
const child = await common.spawnPromisified(
process.execPath,
args,
{ cwd: tmpdir.path },
);
t.assert.strictEqual(child.code, 0);
t.assert.strictEqual(child.signal, null);
t.assert.match(child.stdout, /tests 6/);
t.assert.match(child.stdout, /pass 6/);
t.assert.match(child.stdout, /fail 0/);
});
});