Better filtering story for MTP

[Youssef1313]

In MTP, filtering tests is a bit of an issue when talking about any "common" infrastructure that can be used by multiple repos/projects that might be using different test frameworks.

With VSTest, excluding by trait was simple. Just --filter "TestCategory!=Failing" and that works for all frameworks (hopefully?)

This is quite a problem with MTP.

• MSTest/NUnit: The VSTest filter continues to work.
• xunit 2 via YTest.MTP.XUnit2: The VSTest filter continues to work.
• xunit.v3: The VSTest filter no longer works as of today (Pending outcome of Experiment (PoC): Support VSTest-based --filter for MTP xunit/xunit#3466?)
• TUnit: The VSTest filter no longer works.

Generally speaking, there is unfortunately no standardized filter, which makes it harder for users.

[Evangelink]

We knew this was a problem of VSTest and we wanted to have a solution when designing MTP but because the frameworks are responsible for finding and running the tests so there is no simple way.

The best solution we had was to have filtering done via capabilities and to reject some frameworks in some contexts if the capability is not implemented.

As per VSTest docs, TestCategory is not available for all (e.g. not available for xUnit) https://learn.microsoft.com/en-us/dotnet/core/testing/selective-unit-tests?pivots=mstest

[Youssef1313]

@Evangelink If you have xunit test with [Trait("TestCategory", "Failing")] it will work with xUnit. In that case, it falls under "Traits" part in that documentation. Any arbitrary trait key can be filtered on.

[Youssef1313]

the frameworks are responsible for finding and running the tests so there is no simple way.

This is true for VSTest as well. But what VSTest did is that it kinda forces framework authors to implement --filter as the option is always available. However, this also led to issues when it's supposed to be a standardized option, but every framework could do it differently.

It's probably nice if we can force framework authors to implement the MTP treenode filter, which as of today is only implemented by TUnit. However, this is also challenging to implement for MSTest/NUnit because VSTestBridge is limiting what we can do, but we can try to look into options for how to get that possible in VSTestBridge.

Alternatively, we can push towards having the framework authors to implement the VSTest based filter syntax using --filter which makes the migration story from VSTest to MTP even easier.

[bradwilson]

xUnit.net v3 implements a new query filter syntax (along with with our older style held over from v2), and extends this filtering capability into MTP.

There are two equivalents today:

• --filter-not-trait Category=Failing (using a simple filter)
• --filter-query /[Category!=Failing] (using the query filter syntax)

---

The existing support for MSTest and NUnit presumably comes from the VSTest bridge, and would disappear if/when those projects decided to do native MTP implementations without using the bridge unless they re-implemented it. Since xUnit.net and TUnit are MTP native, they do not inherit this "free feature".

In my opinion, the migration from VSTest to MTP is involved enough that re-writing your filters is probably a relatively small part of the migration work. @Youssef1313 has offered a POC to support the VSTest --filter option for xUnit.net, but my current inclination is to reject this because it's a non-trivial implementation that I will be forced to support. I haven't yet seen any user requests for this, presumably because users are discovering that --filter isn't there and just moving to one of the existing filtering options that xUnit.net supports in MTP. I believe the migration from VSTest to MTP is a great time for users to move on from older, outdated features to newer options.

[Youssef1313]

There are two equivalents today:

The problem with that is there is no standardization across all frameworks, something that used to partially be true with VSTest. This is relevant for people owning test infrastructure that is not aware of what test framework is being used, or for solutions that mix different test frameworks. In addition to that, there has been complaints I received from people because this is causing AI agents to get confused and always requires custom instructions as of today.

The existing support for MSTest and NUnit presumably comes from the VSTest bridge, and would disappear if/when those projects decided to do native MTP implementations without using the bridge unless they re-implemented it.

For MSTest, whenever we move away from VSTestBridge, I would definitely like to re-implement the VSTest filter to keep compatibility and ease migration.

[bradwilson]

The problem with that is there is no standardization across all frameworks, something that used to partially be true with VSTest. This is relevant for people owning test infrastructure that is not aware of what test framework is being used, or for solutions that mix different test frameworks.

We are in agreement that the problem belongs primarily to MTP, and it should not be on the test frameworks alone to provide individual (drifting/incompatible/etc.) implementations of --filter, if the migration scenario requires supporting --filter.

Also I'm surprised that this hasn't come up before now, presumably because the original MTP design assumed test frameworks would just use the bridge (so it was a blind spot to whatever features are encoded into the bridge rather than the framework itself).

[Evangelink]

MTP assumed we would all use the tree/query filter but we were forced to back up most of our initial plan because of support for Test Explorer, and some management concerns.

As previously explained the idea was to not force things in the basic scenario so that frameworks have enough flexibility and platform can really run anywhere. On top of that, we wanted to rely on capabilities both on the client and server side so that there could be requirements in some scenarios (e.g. client can gracefully fallback or reject some server if it misses a feature and vice versa, allowing to build more complex architecture when needed).

[thomhurst]

In addition to that, there has been complaints I received from people because this is causing AI agents to get confused and always requires custom instructions as of today.

Yeah because TUnit doesn't have the old filter syntax (and can't afaik because the bridge package connects to the old VSTest implementations?) AI agents get in a right muddle. They're so trained on that syntax that it's all they primarily try to use unless you explicitly tell them not to.

[Youssef1313]

@thomhurst You can bring the old filter syntax to TUnit by copying the relevant parts from vstest (e.g, similar to xunit/xunit#3466). One thing I'll discuss with @nohwnd is the possibility of providing a source-only package from vstest that contains the filter-related logic so that xunit and VSTestBridge can use it without duplicating/copying the logic, and also you can use that for TUnit if you wanted to.