test_runner: propagate only to test ancestors

[MoLow]

Fixes: #47945

this PR is broken into smaller commits that are intended to be logically independent and easier to review.

some limitations/todos:

• compare benchmarks (benchmark: add benchmarks for the test_runner #48931)
• This only works for a single file. meaning test.only will skip all tests in that file, but not in other files.
  solving this is much less trivial since it requires some kind of two-way communication between the test file and the parent process, in between the enqueue and the dequeue phases.
  we should consider if this solution is a big enough improvement compared to the current behavior for it to be released, or if we should solve multiple files as well
• need to update documentation?
• need to document deprecation of --run-only
• add a test for --test-only with multiple files

[nodejs-github-bot]

Review requested:

• @nodejs/test_runner

[cjihrig]

I haven't reviewed or tested this, but some initial thoughts:

This only works for a single file

This seems like it would be more confusing than what we currently have.

some kind of two-way communication between the test file and the parent process, in between the enqueue and the dequeue phases

I don't think this is going to end up in a very efficient design. Unless I'm mistaken, it also doesn't solve the problem of an only test that is not at the top level.

I think it would probably be better to keep the --test-only flag, if that flag is provided build the entire test tree, and then go back and run the only flags. I get that people don't want to specify --test-only, but it seems more powerful than the alternative here because you would not pay the extra cost in the common case and you could determine if there are nested only tests. Happy to be proven wrong on this though.

As a side note: I think solving #46728 would be more useful since it overlaps a lot with this feature and also makes IDE integration significantly better. I believe that issue can be solved in the current state of the test runner, but could also be solved by building the entire test tree.

[MoLow]

Unless I'm mistaken, it also doesn't solve the problem of an only test that is not at the top level.

you are mistaken (It might have a bug, but the intent is for a test at any level to cause all other tests to be skipped).
for example, this code results in this output:

describe('top level', () => {
  describe('nested', () => {
    describe('nested', () => {
      it.only('only', (t) => {
        t.diagnostic('only')
      });
      it('this is a test', () => {});
      it('this is a test', () => {});
    });
    it('this is a test', () => {});
  });
  it('this is a test', () => {});
});

it.only('this is a test', () => {});

it('this is a test', () => {
  console.log('this is a test')
});

▶ top level
  ▶ nested
    ▶ nested
      ✔ only (0.208458ms)
      ℹ only
      ﹣ this is a test (0.090208ms) # SKIP
      ﹣ this is a test (0.041084ms) # SKIP
    ▶ nested (1.021583ms)

    ﹣ this is a test (0.077125ms) # SKIP
  ▶ nested (1.2045ms)

  ﹣ this is a test (0.031ms) # SKIP
▶ top level (1.404667ms)

✔ this is a test (0.037875ms)
﹣ this is a test (0.041542ms) # SKIP
ℹ tests 7
ℹ suites 3
ℹ pass 2
ℹ fail 0
ℹ cancelled 0
ℹ skipped 5
ℹ todo 0
ℹ duration_ms 2.580958

I think it would probably be better to keep the --test-only flag, if that flag is provided build the entire test tree, and then go back and run the only flags. I get that people don't want to specify --test-only, but it seems more powerful than the alternative here because you would not pay the extra cost in the common case and you could determine if there are nested only tests. Happy to be proven wrong on this though.

doing this won't solve the multiple files issue.
the pain today is that you must specify {only:true} in each ancestor of a test that you really want to only run. once you eliminate that - even if you do that only behind the --test-only flag - you must handle tests in other files, otherwise all the tests in files that dont contain only tests will run

[MoLow]

@cjihrig Following our discussion, I have updated the pr

[MoLow]

seems like we might want to still require --test-only in single file execution, but bubble up the only tests when using --test-only

[cjihrig]

@MoLow are you still planning to work on this? I think it makes sense to implement this for suites now. If you don't want to, I'd be happy to take a shot at it.

[MoLow]

@MoLow are you still planning to work on this? I think it makes sense to implement this for suites now. If you don't want to, I'd be happy to take a shot at it.

Taking a look now