Add support for ordered tests

[sbaid]

Description

Is it possible to decide this based on telemetry data of ordered tests usage?

Steps to reproduce

What steps can reproduce the defect?
Please share the setup, sample project, target platform (desktop, core, uap)

Expected behavior

Share the expected output

Actual behavior

What is the behavior observed?

Environment

Please share additional details about the test environment.
Operating system, Build version of vstest.console, Package version of MSTest
framework and adapter

[sbaid]

@harshjain2 Do we have telemetry data of usage of ordered tests, generic tests?

[sbaid]

@pvlakshm to keep u in loop

[AbhitejJohn]

I believe @pvlakshm has also filed a user voice item for this.

[codito]

https://visualstudio.uservoice.com/forums/121579-visual-studio-ide/suggestions/16417609-enable-ordered-test-support-for-mstest-v2-framewor

[pvlakshm]

We should support ordered tests in a generic manner and not restrict it to only MSTest framework based tests - perhaps as yet another extension point offered by the test platform.

[divijkumar]

We have an app (on multiple platforms) where a lot of the code is in C++. We use a very thin cross-platform wrapper for writing unit tests. The wrapper internally uses CPPUnitTestFramework for unit testing on Windows.

Sometimes, when running the unit tests one of the tests will fail, and that will result in some of the tests that are executed after the failed test, to fail as well. Most of the times the fix is a trivial one, but time is spent in segregating the test which actually failed from those that failed as a result of some remnant of the earlier failed test.

We would like to know if there is a way to determine the order in which the unit tests are run inside Visual Studio. This will help us in determining the failing test quickly, and help resolve issues faster.

Moreover, it would be good to have a mechanism to control the order of the execution of the unit tests inside Visual Studio.

[AbhitejJohn]

@divijkumar : Ideally tests should be isolated units and should not adversely affect other tests. They do so by ensuring that the right cleanup is done even if a test fails. When this is not possible for some reason, you could turn on diagnostic logs(link for the default test platform, link for the new cross plat test platform) to figure out what order they actually ran.
Yes, this issue should probably cover providing ordered test support for C++ based tests as well.

[rhundhausen]

I'm not a huge fan of ordered tests, but they do serve a purpose (e.g. integration testing by sequencing many unit tests). That said, one would expect the default behavior of VS 2017 to just work.