test_runner: fix test runner concurrency

[MoLow]

Fixes: #47365
Fixes: #47696

SafePromiseAllSettledReturnVoid runs sequentially before this fix :(
this makes me wonder if we should add a benchmark for the test runner

[MoLow]

CC @nodejs/performance Who can help me with creating a benchmark? I never have before

[tniessen]

SafePromiseAllSettledReturnVoid runs sequentially before this fix

Is this the relevant invocation?

node/lib/internal/test_runner/runner.js

Lines 390 to 397 in 0c5f253

	const runFiles = () => {
	root.harness.bootstrapComplete = true;
	return SafePromiseAllSettledReturnVoid(testFiles, (path) => {
	const subtest = runTestFile(path, root, inspectPort, filesWatcher);
	runningSubtests.set(path, subtest);
	return subtest;
	});
	};

this makes me wonder if we should add a benchmark for the test runner

If it is expected that test files are executed in parallel, then that should be covered by a test before any benchmark. Benchmarks are good for performance comparisons, but this is about a difference between design and implementation, i.e., functional correctness.

[MoLow]

Is this the relevant invocation?

yes

If it is expected that test files are executed in parallel, then that should be covered by a test before any benchmark. Benchmarks are good for performance comparisons, but this is about a difference between design and implementation, i.e., functional correctness.

I will add a test

[tniessen]

@aduh95 In #45175, was the main motivation behind SafePromiseAllSettledReturnVoid not having to allocate an array for results, and so that the caller automatically obtains a Promise<void>? If so, this appears to be the only call site, so maybe we can just use SafePromiseAllSettled instead and return the Promise<void> from the test runner instead?

[tniessen]

Is the number of concurrent tests guaranteed to be at least two? I thought it depended on os.availableParallelism(), in which case it looks like this test might fail when only one or two CPU cores are assigned to the process group.

[MoLow]

this fix introduced a race condition in tests outputs being reported out of order, so I fixed that as well

[MoLow]

@cjihrig please confirm this skip makes sense

[cjihrig]

I think it makes sense. I think you could verify by replacing the use of os.availableParallelism() in the test runner code with 1 or 2 and seeing if the test hangs.

[mcollina]

lgtm

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/51437/

[MoLow]

@cjihrig can you take a look? I've had to fix multiple race conditions in the test runner.
this also depends on #47699 to accurately count tests

[MoLow]

@benjamingr @mcollina PTAL

[cjihrig]

The logic in this function (and even the function signature) seems like it should be part of #handleReportItem(). Is it possible to handle this there?

[MoLow]

handleReportItem happens only in order of reporting (using this.isClearToSend()), however, #skipReporting returns wrong values if it is called after the child process has completed but the test has not yet been reported

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/51450/

[MoLow]

🟢 CI: https://ci.nodejs.org/job/node-test-pull-request/51451/

[nodejs-github-bot]

Landed in 8becacb...dccd25e