Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit d14029b

Browse filesBrowse files
pmarchiniaduh95
pmarchini
authored and
aduh95
committed
test_runner: support test order randomization
PR-URL: #61747 Reviewed-By: Jacob Smith <jacob@frende.me> Reviewed-By: Marco Ippolito <marcoippolito54@gmail.com> Reviewed-By: Ethan Arrowood <ethan@arrowood.dev> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Aviv Keller <me@aviv.sh> Reviewed-By: Chemi Atlow <chemi@atlow.co.il> Reviewed-By: Moshe Atlow <moshe@atlow.co.il>
1 parent ec8c6b9 commit d14029b
Copy full SHA for d14029b

28 files changed

+1,553-21Lines changed: 1553 additions & 21 deletions
Expand file treeCollapse file tree
Open diff view settings
Collapse file

‎doc/api/cli.md‎

Copy file name to clipboardExpand all lines: doc/api/cli.md
+35Lines changed: 35 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -2669,6 +2669,38 @@ changes:
26692669
Configures the test runner to only execute top level tests that have the `only`
26702670
option set. This flag is not necessary when test isolation is disabled.
26712671

2672+
### `--test-random-seed`
2673+
2674+
<!-- YAML
2675+
added: REPLACEME
2676+
-->
2677+
2678+
Set the seed used to randomize test execution order. This applies to both test
2679+
file execution order and queued tests within each file. Providing this flag
2680+
enables randomization implicitly, even without `--test-randomize`.
2681+
2682+
The value must be an integer between `0` and `4294967295`.
2683+
2684+
This flag cannot be used with `--watch` or `--test-rerun-failures`.
2685+
2686+
### `--test-randomize`
2687+
2688+
<!-- YAML
2689+
added: REPLACEME
2690+
-->
2691+
2692+
Randomize test execution order. This applies to both test file execution order
2693+
and queued tests within each file. This can help detect tests that rely on
2694+
shared state or execution order.
2695+
2696+
The seed used for randomization is printed in the test summary and can be
2697+
reused with `--test-random-seed`.
2698+
2699+
For detailed behavior and examples, see
2700+
[randomizing tests execution order][].
2701+
2702+
This flag cannot be used with `--watch` or `--test-rerun-failures`.
2703+
26722704
### `--test-reporter`
26732705

26742706
<!-- YAML
@@ -3582,6 +3614,8 @@ one is included in the list below.
35823614
* `--test-isolation`
35833615
* `--test-name-pattern`
35843616
* `--test-only`
3617+
* `--test-random-seed`
3618+
* `--test-randomize`
35853619
* `--test-reporter-destination`
35863620
* `--test-reporter`
35873621
* `--test-rerun-failures`
@@ -4162,6 +4196,7 @@ node --stack-trace-limit=12 -p -e "Error.stackTraceLimit" # prints 12
41624196
[libuv threadpool documentation]: https://docs.libuv.org/en/latest/threadpool.html
41634197
[module compile cache]: module.md#module-compile-cache
41644198
[preloading asynchronous module customization hooks]: module.md#registration-of-asynchronous-customization-hooks
4199+
[randomizing tests execution order]: test.md#randomizing-tests-execution-order
41654200
[remote code execution]: https://www.owasp.org/index.php/Code_Injection
41664201
[running tests from the command line]: test.md#running-tests-from-the-command-line
41674202
[scavenge garbage collector]: https://v8.dev/blog/orinoco-parallel-scavenger
Collapse file

‎doc/api/test.md‎

Copy file name to clipboardExpand all lines: doc/api/test.md
+100Lines changed: 100 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -626,6 +626,94 @@ prevent shell expansion, which can reduce portability across systems.
626626
node --test "**/*.test.js" "**/*.spec.js"
627627
```
628628

629+
### Randomizing tests execution order
630+
631+
<!-- YAML
632+
added: REPLACEME
633+
-->
634+
635+
> Stability: 1.0 - Early development
636+
637+
The test runner can randomize execution order to help detect
638+
order-dependent tests. When enabled, the runner randomizes both discovered
639+
test files and queued tests within each file. Use `--test-randomize` to
640+
enable this mode.
641+
642+
```bash
643+
node --test --test-randomize
644+
```
645+
646+
When randomization is enabled, the test runner prints the seed used for the run
647+
as a diagnostic message:
648+
649+
```text
650+
Randomized test order seed: 12345
651+
```
652+
653+
Use `--test-random-seed=<number>` to replay the same randomized order
654+
deterministically. Supplying `--test-random-seed` also enables randomization,
655+
so `--test-randomize` is optional when a seed is provided:
656+
657+
```bash
658+
node --test --test-random-seed=12345
659+
```
660+
661+
In most test files, randomization works automatically. One important exception
662+
is when subtests are awaited one by one. In that pattern, each subtest starts
663+
only after the previous one finishes, so the runner keeps declaration order
664+
instead of randomizing it.
665+
666+
Example: this runs sequentially and is **not** randomized.
667+
668+
```mjs
669+
import test from 'node:test';
670+
671+
test('math', async (t) => {
672+
for (const name of ['adds', 'subtracts', 'multiplies']) {
673+
// Sequentially awaiting each subtest preserves declaration order.
674+
await t.test(name, async () => {});
675+
}
676+
});
677+
```
678+
679+
```cjs
680+
const test = require('node:test');
681+
682+
test('math', async (t) => {
683+
for (const name of ['adds', 'subtracts', 'multiplies']) {
684+
// Sequentially awaiting each subtest preserves declaration order.
685+
await t.test(name, async () => {});
686+
}
687+
});
688+
```
689+
690+
Using suite-style APIs such as `describe()`/`it()` or `suite()`/`test()`
691+
still allows randomization, because sibling tests are enqueued together.
692+
693+
Example: this remains eligible for randomization.
694+
695+
```mjs
696+
import { describe, it } from 'node:test';
697+
698+
describe('math', () => {
699+
it('adds', () => {});
700+
it('subtracts', () => {});
701+
it('multiplies', () => {});
702+
});
703+
```
704+
705+
```cjs
706+
const { describe, it } = require('node:test');
707+
708+
describe('math', () => {
709+
it('adds', () => {});
710+
it('subtracts', () => {});
711+
it('multiplies', () => {});
712+
});
713+
```
714+
715+
`--test-randomize` and `--test-random-seed` are not supported with `--watch` mode.
716+
629717
Matching files are executed as test files.
630718
More information on the test file execution can be found
631719
in the [test runner execution model][] section.
@@ -666,6 +754,10 @@ test runner functionality:
666754
* `--test-reporter` - Reporting is managed by the parent process
667755
* `--test-reporter-destination` - Output destinations are controlled by the parent
668756
* `--experimental-config-file` - Config file paths are managed by the parent
757+
* `--test-randomize` - Randomization is managed by the parent process and
758+
propagated to child processes
759+
* `--test-random-seed` - Randomization seed is managed by the parent process and
760+
propagated to child processes
669761

670762
All other Node.js options from command line arguments, environment variables,
671763
and configuration files are inherited by the child processes.
@@ -1572,6 +1664,14 @@ changes:
15721664
that specifies the index of the shard to run. This option is _required_.
15731665
* `total` {number} is a positive integer that specifies the total number
15741666
of shards to split the test files to. This option is _required_.
1667+
* `randomize` {boolean} Randomize execution order for test files and queued tests.
1668+
This option is not supported with `watch: true`.
1669+
**Default:** `false`.
1670+
* `randomSeed` {number} Seed used when randomizing execution order. If this
1671+
option is set, runs can replay the same randomized order deterministically,
1672+
and setting this option also enables randomization. The value must be an
1673+
integer between `0` and `4294967295`.
1674+
**Default:** `undefined`.
15751675
* `rerunFailuresFilePath` {string} A file path where the test runner will
15761676
store the state of the tests to allow rerunning only the failed tests on a next run.
15771677
see \[Rerunning failed tests]\[] for more information.
Collapse file

‎doc/node-config-schema.json‎

Copy file name to clipboardExpand all lines: doc/node-config-schema.json
+16Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -524,6 +524,14 @@
524524
"type": "boolean",
525525
"description": "run tests with 'only' option set"
526526
},
527+
"test-random-seed": {
528+
"type": "number",
529+
"description": "seed used to randomize test execution order"
530+
},
531+
"test-randomize": {
532+
"type": "boolean",
533+
"description": "run tests in a random order"
534+
},
527535
"test-reporter": {
528536
"oneOf": [
529537
{
@@ -902,6 +910,14 @@
902910
"type": "boolean",
903911
"description": "run tests with 'only' option set"
904912
},
913+
"test-random-seed": {
914+
"type": "number",
915+
"description": "seed used to randomize test execution order"
916+
},
917+
"test-randomize": {
918+
"type": "boolean",
919+
"description": "run tests in a random order"
920+
},
905921
"test-reporter": {
906922
"oneOf": [
907923
{
Collapse file

‎doc/node.1‎

Copy file name to clipboardExpand all lines: doc/node.1
+16Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -498,6 +498,22 @@ Configures the type of test isolation used in the test runner.
498498
A regular expression that configures the test runner to only execute tests
499499
whose name matches the provided pattern.
500500
.
501+
.It Fl -test-random-seed
502+
Set the seed used to randomize test execution order.
503+
This applies to both test file execution order and queued tests within each file.
504+
Providing this flag enables randomization implicitly, even without
505+
\fB--test-randomize\fR.
506+
The value must be an integer between 0 and 4294967295.
507+
This flag cannot be used with \fB--watch\fR or \fB--test-rerun-failures\fR.
508+
.
509+
.It Fl -test-randomize
510+
Randomize test execution order.
511+
This applies to both test file execution order and queued tests within each file.
512+
This can help detect tests that rely on shared state or execution order.
513+
The seed used for randomization is printed in the test summary and can be
514+
reused with \fB--test-random-seed\fR.
515+
This flag cannot be used with \fB--watch\fR or \fB--test-rerun-failures\fR.
516+
.
501517
.It Fl -test-reporter
502518
A test reporter to use when running tests.
503519
.
Collapse file

‎lib/internal/test_runner/runner.js‎

Copy file name to clipboardExpand all lines: lib/internal/test_runner/runner.js
+73-4Lines changed: 73 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ const {
5959
validateObject,
6060
validateOneOf,
6161
validateInteger,
62+
validateUint32,
6263
validateString,
6364
validateStringArray,
6465
} = require('internal/validators');
@@ -84,6 +85,7 @@ const {
8485
const { FastBuffer } = require('internal/buffer');
8586

8687
const {
88+
createRandomSeed,
8789
convertStringToRegExp,
8890
countCompletedTest,
8991
kDefaultPattern,
@@ -105,12 +107,14 @@ const kIsolatedProcessName = Symbol('kIsolatedProcessName');
105107
const kFilterArgs = [
106108
'--test',
107109
'--experimental-test-coverage',
110+
'--test-randomize',
108111
'--watch',
109112
'--experimental-default-config-file',
110113
];
111114
const kFilterArgValues = [
112115
'--test-reporter',
113116
'--test-reporter-destination',
117+
'--test-random-seed',
114118
'--experimental-config-file',
115119
];
116120
const kDiagnosticsFilterArgs = ['tests', 'suites', 'pass', 'fail', 'cancelled', 'skipped', 'todo', 'duration_ms'];
@@ -168,6 +172,8 @@ function getRunArgs(path, { forceExit,
168172
argv: suppliedArgs,
169173
execArgv,
170174
rerunFailuresFilePath,
175+
randomize,
176+
randomSeed,
171177
root: { timeout },
172178
cwd }) {
173179
const processNodeOptions = getOptionsAsFlagsFromBinding();
@@ -209,6 +215,12 @@ function getRunArgs(path, { forceExit,
209215
if (rerunFailuresFilePath) {
210216
ArrayPrototypePush(runArgs, `--test-rerun-failures=${rerunFailuresFilePath}`);
211217
}
218+
if (randomize) {
219+
ArrayPrototypePush(runArgs, '--test-randomize');
220+
}
221+
if (randomSeed != null) {
222+
ArrayPrototypePush(runArgs, `--test-random-seed=${randomSeed}`);
223+
}
212224

213225
ArrayPrototypePushApply(runArgs, execArgv);
214226

@@ -646,6 +658,8 @@ function run(options = kEmptyObject) {
646658
lineCoverage = 0,
647659
branchCoverage = 0,
648660
functionCoverage = 0,
661+
randomize: suppliedRandomize,
662+
randomSeed: suppliedRandomSeed,
649663
execArgv = [],
650664
argv = [],
651665
cwd = process.cwd(),
@@ -674,6 +688,56 @@ function run(options = kEmptyObject) {
674688
if (globPatterns != null) {
675689
validateArray(globPatterns, 'options.globPatterns');
676690
}
691+
if (suppliedRandomize != null) {
692+
validateBoolean(suppliedRandomize, 'options.randomize');
693+
}
694+
if (suppliedRandomSeed != null) {
695+
validateUint32(suppliedRandomSeed, 'options.randomSeed');
696+
}
697+
let randomize = suppliedRandomize;
698+
let randomSeed = suppliedRandomSeed;
699+
700+
if (randomSeed != null) {
701+
randomize = true;
702+
}
703+
if (watch) {
704+
if (randomSeed != null) {
705+
throw new ERR_INVALID_ARG_VALUE(
706+
'options.randomSeed',
707+
randomSeed,
708+
'is not supported with watch mode',
709+
);
710+
}
711+
if (randomize) {
712+
throw new ERR_INVALID_ARG_VALUE(
713+
'options.randomize',
714+
randomize,
715+
'is not supported with watch mode',
716+
);
717+
}
718+
}
719+
if (rerunFailuresFilePath) {
720+
validatePath(rerunFailuresFilePath, 'options.rerunFailuresFilePath');
721+
// TODO(pmarchini): Support rerun-failures with randomization by
722+
// persisting the randomization seed in the rerun state file.
723+
if (randomSeed != null) {
724+
throw new ERR_INVALID_ARG_VALUE(
725+
'options.randomSeed',
726+
randomSeed,
727+
'is not supported with rerun failures mode',
728+
);
729+
}
730+
if (randomize) {
731+
throw new ERR_INVALID_ARG_VALUE(
732+
'options.randomize',
733+
randomize,
734+
'is not supported with rerun failures mode',
735+
);
736+
}
737+
}
738+
if (randomize) {
739+
randomSeed ??= createRandomSeed();
740+
}
677741

678742
validateString(cwd, 'options.cwd');
679743

@@ -683,10 +747,6 @@ function run(options = kEmptyObject) {
683747
);
684748
}
685749

686-
if (rerunFailuresFilePath) {
687-
validatePath(rerunFailuresFilePath, 'options.rerunFailuresFilePath');
688-
}
689-
690750
if (shard != null) {
691751
validateObject(shard, 'options.shard');
692752
// Avoid re-evaluating the shard object in case it's a getter
@@ -783,11 +843,18 @@ function run(options = kEmptyObject) {
783843
functionCoverage: functionCoverage,
784844
cwd,
785845
globalSetupPath,
846+
randomize,
847+
randomSeed,
786848
};
849+
787850
const root = createTestTree(rootTestOptions, globalOptions);
788851
let testFiles = files ?? createTestFileList(globPatterns, cwd);
789852
const { isTestRunner } = globalOptions;
790853

854+
if (randomize) {
855+
root.diagnostic(`Randomized test order seed: ${randomSeed}`);
856+
}
857+
791858
if (shard) {
792859
testFiles = ArrayPrototypeFilter(testFiles, (_, index) => index % shard.total === shard.index - 1);
793860
}
@@ -833,6 +900,8 @@ function run(options = kEmptyObject) {
833900
rerunFailuresFilePath,
834901
env,
835902
workerIdPool: isolation === 'process' ? workerIdPool : null,
903+
randomize,
904+
randomSeed,
836905
};
837906

838907
if (isolation === 'process') {

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.