[Proposal]: Flaky test management

[Youssef1313]

Summary

This is a concrete proposal for a flaky test management system that can be provided by test platform with a cooperation with test frameworks.

Goals:

• Have a native experience for managing flaky tests.
• We don't want to lose coverage from those tests by completely ignoring them forever.
• We want to have a way to track whether or not a test became stable.

Background and Motivation

Flaky tests are part of every big project that people need to deal with it. Today, there is no good way to manage and track flaky tests, and different efforts and approaches are getting invented by individual teams. None of the approaches that exist today is very ideal as well.

Proposed Feature

The test platform should have the knowledge of a test being known to be flaky. Such tests should be run, but shouldn't fail the test run. For that, such tests will have "inconclusive" result. But there we have two different kind of "inconclusive" results. First is "passed inconclusive", second is "failed inconclusive".

How this is going to work

Individual test frameworks who want to support this feature will:

• Ship their own attribute, let's say it's named FlakyTestAttribute or InconclusiveResultAttribute (that's a choice for the individual test frameworks to make)
• Have a way to let MTP know that the test is marked as such.
• MTP reporting provides a way to get the results of such tests.
  • Because of backcompat concerns for TRX, we might need a different way.
  • This could be a simple json file that doesn't have the full test reporting capabilities. The json will just have two arrays, one for "inconclusive passed" and the other for "inconclusive failed" and the new json file format won't be concerned about all other kind of tests, and won't be concerned by attachments, display names, duration, traits, ....
  • The info needed might be just the TestUid and FQN.

This allows users to track how their flaky tests performs without fully skipping them, and also by keeping them running in the same process as the other tests.

People can then build their system that deals with those json files.

Taking this a step further

There are two additional ways to take this a step further.

1. Manage a flaky info storage on Azure DevOps.

   • We could use Azure DevOps to store information about those flaky tests and have an easy way to do queries like "tests that haven't failed in the past X runs" or "tests that haven't failed in the past X days" or "tests with failure rate >= X%" or "tests with failure rate <= X%", ...

2. Specify a specific flakiness reason.

   • This is about having specific failure(s) to be considered "inconclusive failed".

   • Let's say:

     [FlakyTest(nameof(IsTimeout))]
     public void MyTest() { ... }
     
     public static void IsTimeout(Exception testFailureException)
          => testFailureException.Message.Contains("Timed out");

   • In the above scenario, we could have 3 outcomes:

     • Test is passing -> Consider as "inconclusive passed".
     • Test is failing matching the known flakiness reason (e.g, timeout check in example) -> Consider as "inconclusive failed".
     • Test is failing with an unknown reason -> Consider as "failed" and fail the test run.

   • Why is this nice?

     • This allows users to still run their flaky tests without failing CI much for "known reasons" that users acknowledged. But still fail in case it's a different failure. That way, the test coverage isn't fully lost, and regressions can still be captured by those tests. So we get the best of everything, CI stability and capturing new potential regressions.

   • The way how the "is known flakiness reason" is implemented will be up to the test frameworks. The test platform should provide some way for the test framework to hook the implementation. The right API shape for this might require some experimentation with implementation.

Additional notes

As of today, repos could have their own custom "quarantine" system. The way this works is that tests are by default "unquarantined", and with the existence of a specific test property (via attribute), the test is marked as "quarantined".

Repos could then setup the necessary infrastructure to have two CI pipelines, one that's required and runs only quarantined tests, and another that's not required and runs only unquarantined tests.

This has down sides:

• The detection for whether or not a test is stable cannot be reliable when quarantined tests are isolated from other tests.
• It's extra infrastructure that people have to maintain and potentially duplicate across repos.

Note that the above system is implemented by aspnetcore repo already, but isn't reliable IMO.

[Youssef1313]

Jakub Jareš (@nohwnd) Up to you if you want to consider implementing this for VSTest as well.

[Evangelink]

For context on what is already in the repo today: #8298 (merged) provides a partial, AzDO-only slice of this proposal. It lives entirely in Microsoft.Testing.Extensions.AzureDevOpsReport and does not introduce any platform- or framework-level concept of a "flaky" test.

Already supported (via Microsoft.Testing.Extensions.AzureDevOpsReport):

• --report-azdo-flaky-history <N> — queries AzDO REST history for the last N days and decorates failures in the AzDO log with [flaky: failed K/N in last Md] or [REGRESSION] (the latter only fires once there are ≥5 prior samples).
• --report-azdo-quarantine-file <path> — a static text file of FQNs / globs (# comments allowed). Listed tests have their failures demoted to warning in the AzDO log, tagged [quarantined], and emit a ##vso[build.addbuildtag]has-quarantined-test-failure build tag once per run.
• --report-azdo-demote-known-flaky — combined with --report-azdo-flaky-history, automatically demotes failures with a ≥25% historical flake rate to warning (default OFF, annotate-only).
• Resilient HTTP client (retries, Retry-After, AOT-safe JsonSerializerContext), 30s bounded history load, all errors swallowed so they can never fail the test run.

What this does not do (and what is therefore still in scope for this proposal):

• No FlakyTestAttribute (or equivalent) on the MSTest side.
• No new MTP outcome (InconclusivePassed / InconclusiveFailed); the current TestNodeStateProperty set is unchanged. A demoted-to-warning failure in AzDO is still a regular FailedTestNodeStateProperty everywhere else (terminal, TRX, exit code, --maximum-failed-tests accounting…).
• No API for a test framework to tell the platform "this test is flaky".
• No JSON "flaky results" side file (the inconclusive-passed / inconclusive-failed arrays of TestUid + FQN suggested above).
• No "specific flakiness reason" predicate — [FlakyTest(nameof(IsTimeout))] and the three-outcome model (passed → inconclusive-passed, known-reason failure → inconclusive-failed, unknown failure → fail the run) are not implemented anywhere.
• The quarantine list is file-only and AzDO-only — it isn't surfaced through any other reporter and there's no attribute-driven equivalent.

So #8298 mostly addresses the optional "Manage a flaky info storage on Azure DevOps" sub-bullet, but only as AzDO log decoration. The core platform/framework integration described in this proposal (and especially the per-reason predicate) is still entirely unimplemented.

[jjonescz]

It might be good to have a way to mark flaky tests without the need to actually change the code (adding FlakyTestAttribute). E.g., make it possible to store the list of flaky tests somewhere else. Perhaps the test platform would not care where, it would just provide an extensibility for this. Then a system like Build Analysis Known Issues could be connected to this.

Also, should there be an option to automatically rerun flaky tests? That might be simpler than needing the whole quarantined/unquarantined pipeline split - just have one pipeline as usual, require all tests to pass, but re-run flaky tests.

[Evangelink]

Also, should there be an option to automatically rerun flaky tests?

MTP has a retry extension (see https://learn.microsoft.com/dotnet/core/testing/microsoft-testing-platform-retry).

It might be good to have a way to mark flaky tests without the need to actually change the code

This is still not released but as specified above, I have added some new functionalities to the azdo extension, including:

--report-azdo-flaky-history <N> — queries AzDO REST history for the last N days and decorates failures in the AzDO log with [flaky: failed K/N in last Md] or [REGRESSION] (the latter only fires once there are ≥5 prior samples).

So we can know which test was retried (correlated with retry extension)

[Youssef1313]

MTP has a retry extension (see https://learn.microsoft.com/dotnet/core/testing/microsoft-testing-platform-retry).

Retry extensions re-tries all failures. I think the request here is to retry tests known to be flaky, and the end user will have an extensibility point to define what tests are flaky. As far as the test platform is concerned, it just needs the info from the test framework if a test is flaky or not. Then it's up to test framework authors to provide the extensibility points for defining flaky tests (this could be attribute as proposed in my original description, or it could be some other arbitrary source which is what Jan proposed, which I think is a great extensibility point to have and allows the flaky test management to be more flexible).

Also, should there be an option to automatically rerun flaky tests? That might be simpler than needing the whole quarantined/unquarantined pipeline split - just have one pipeline as usual, require all tests to pass, but re-run flaky tests.

With the proposal above, failing flaky tests don't fail the test run, so retrying could be pointless.

However, there could be a strict mode to say "retry flaky test until it passes", which is very possible today without any changes on the test platform as it's purely a test framework feature. For example, MSTest already has RetryAttribute (and xUnit could provide one if they don't already). Note that we also provide RetryBaseAttribute which you could inherit from to define when a test should be retried (so that you only retry when a "known" failure happens - and don't retry on any other failures which could mask real issues)

[Youssef1313]

Another feedback I got here is to have easier ways to help with flaky test investigations, so the tests get fixed instead of getting quarantined. I personally see value of both.

[Youssef1313]

One big challenge that we will need to think about is how to handle tests that hang or crash the process.

This is a case where running in the same process and ignoring the test result might not be sufficient. We can decide that these are out of scope for this feature and these tests need to be permanently ignored/skipped.

Anyways, these tests can be problematic today with the alternative quarantine approach that some pipelines do.

Viktor Hofer (@ViktorHofer) William Godbe (@wtgodbe) I wonder how these cases are handled by the recent work done in msbuild and also by the quarantine system in aspnetcore? If the quarantine job hangs or even crashes before executing all the tests. How do you determine whether some tests became stable or not. If those systems rely on parsing TRX, you might get a partial TRX today in these cases. Do you treat a test that didn't report result in TRX as "failing" (which means some tests could have become stable but won't be detected) or do you treat them as "passing" (which means some tests could have been still flaky but will be seen as stable)

[jaredpar]

This proposal doesn't seem to address the case where the environment around a test is flaky, not the test itself. For example:

1. Integration tests often suffer from the inherent complexity of the system. While the test and product code can be correct, the harness around the product can have issues. This leads to flakiness across the entire set of tests, not specific to individual ones.
2. The larger the test bed the more chance that the inherent flakiness of the system will lead to random test failures. In the past dotnet/runtime used to run 20+ million tests in CI (not a typo). At that scale even though the product and test code was sound there were almost always a handful of failures that popped up.
3. Networking tests are notoriously hard to get reliable as a system.

I definitely agree that the ability to mark specific tests as flaky and have flexibility around them is important. But to truly handle the inherent flakiness of these complex systems we need a way to monitor the entire test suite over a period of time. Ship or no ship decisions need to be able to look at the history of the tests, set metrics for build success (fail build if more than X% of tests fail), and be accessible to agents to track down persistent sources of flakiness.

[wtgodbe]

If the quarantine job hangs or even crashes before executing all the tests. How do you determine whether some tests became stable or not.

For aspnetcore, we just exclude those results. The threshold is that the test must have ran (and returned results) in at least 66% of all quarantine jobs over the past 30 days, and it must have passed in all of those runs