Report MSTest [Retry] attempts to the platform instead of discarding them

[Evangelink]

Implements #10292.

Problem

UnitTestRunner.RunSingleTestAsync kept only the last retry attempt:

result = retryResult.TryGetLast() ?? throw ApplicationStateGuard.Unreachable();

so an in-process [Retry] was invisible everywhere — the terminal showed a plain passing test, and the CTRF report claimed flaky: 0 with no retryAttempts[] even though CtrfReportEngine.CollapseAttempts / IsFlaky already implement the correct rule.

Approach

Publish every attempt as its own TestNodeUpdateMessage under the same TestNode.Uid, tagged with a new platform property:

public sealed class RetryAttemptProperty : IProperty
{
    public RetryAttemptProperty(int attemptNumber, bool isSuperseded);
    public int AttemptNumber { get; }   // 1-based, within a single test-host run
    public bool IsSuperseded { get; }   // a later attempt for the same uid follows
}

IsSuperseded lets each consumer opt in or out without buffering the stream, via the shared TestNode.IsSupersededRetryAttempt() helper.

Composes with #10329

#10329 ("Report retried and flaky tests by name") landed in main mid-review and reworked the same terminal subsystem. The two are complementary and now integrated — in-process [Retry] attempts feed the shared flaky reporting:

failed (try 1) FlakyTest (10s 000ms)
  Tests failed
passed (try 2) FlakyTest (10s 000ms)

Test run summary: Passed! - assembly.dll (net8.0|x64)
  total: 1
  failed: 0
  succeeded: 1
  skipped: 0
  flaky: 1 (passed after retry)
  retried: 1 test(s), 1 extra run(s)

Flaky tests:
  FlakyTest failed -> passed (2 attempts)

TestProgressState orders attempts by the pair (host attempt, in-process attempt), and tracks the execution count per uid so the two mechanisms compose rather than collide. CTRF additionally emits "flaky": 1, "retries": 1 and a populated retryAttempts[].

Design note: sibling property vs. part of the state property

Raised in review — should the attempt data hang off Passed/Failed/…? Two real points in favour: PropertyBag has a dedicated O(1) field for the state property, and enforces exactly one, which would make illegal states unrepresentable.

Kept as a sibling because the state types are sealed with public constructors and repo policy bans init on new public API, so carrying two values there means ~12–15 new public constructors plus an equality change on shipped types. Attempt number is also the same category as TimingProperty (it describes the execution occurrence, not the outcome) and applies to InProgress too. The O(n) lookup is not a new class of cost — TestApplicationResult already does an O(n) Any<TestNodeExecutionCompletedProperty>() walk per node.

Consumer audit

After several review rounds each surfaced another affected consumer, I swept every file referencing terminal state properties and classified each one.

Filter superseded attempts (want one row per test): TestApplicationResult (exit code), AbortForMaxFailedTestsExtension, RetryDataConsumer, MSBuildConsumer, GitHubActionsAnnotationReporter, AzureDevOpsReporter, AzureDevOpsArtifactUploader, VideoRecorderSessionHandler, TrxDataConsumer, JUnitReportGenerator.

Keep the test tracked until the final attempt (state machines, not tallies — a retry sequence has one in-progress update but a terminal update per attempt, so completing on a superseded one drops the test while it is still running): HangDumpActivityIndicator, SimplifiedConsoleOutputDeviceBase, SlowTestReporterBase.

Deliberately unchanged: CtrfReportEngine and HtmlReportEngine (keep the history by design — this is what fixes flaky), CrashDumpSequenceLogger (a diagnostic trace; the attempts leading to a crash are exactly what you want to see, so a guard would have been actively harmful), PerRequestServerDataConsumerService (already dedupes), DotnetTestDataConsumer (forwards by design).

VSTest is unaffected: its recorder drops superseded attempts, so Test Explorer and the VSTest TRX are byte-identical to before.

Other correctness fixes found in review

• Crash: class/assembly cleanup results are appended after the retry sequence and kept attempt 1, so the terminal's attempt ordering saw an out-of-order arrival and threw Unreachable(). Cleanup results now inherit the attempt the test finished on.
• Empty final attempt: the combined array was non-empty (superseded only), so the empty-result/execution-error path was skipped, every entry was filtered out, and AllPassed vacuously returned true — turning a missing result into a passing test that unblocked dependents.
• Flaky attempt count: Math.Max(host, retry) undercounts when both mechanisms advance; now tracked per uid.
• Server mode: both JSON-RPC serializers silently drop unregistered properties, so attempt data vanished under --server. Registered in both (System.Text.Json + Jsonite), documented as retry.attempt / retry.is-superseded.
• Serialization: TestResult is [Serializable] on .NET Framework; the two new members needed [field: OptionalField] to match the existing convention for version-tolerant AppDomain deserialization.
• BOM: restored the UTF-8 BOM a scripted edit stripped, and added it to the new files per .editorconfig.

Remaining follow-ups (not blockers)

• dotnet test orchestrator path does not carry the in-process attempt through the IPC wire model, so under dotnet test a [Retry] result is keyed (host attempt, 1) for every attempt. Needs a versioned wire-model field.

• Progress renderer advances once per attempt, so a retry-heavy run can exceed the discovered count. Verified pre-existing on main for host retries; the fix should cover both mechanisms at once.

• Per-attempt timestamps are coarse. TestExecutionManager measures startTime/endTime once around the whole RunSingleTestAsync call, so every flattened attempt carries the same window while keeping its own (correct) Duration. CTRF's retryAttempts[] therefore shows each attempt spanning the whole sequence — attempt 1's stop lands after attempt 2 began. Fixing it properly means capturing per-attempt boundaries, which needs new serializable TestResult members; simply omitting the timing would also drop the correct per-attempt duration.

• Cleanup results inflate the retried-execution count. A class/assembly cleanup failure is reported under the test's uid and is stamped with the final attempt number (so the attempt ordering does not see it as out-of-order). It then lands on TestProgressState's same-attempt branch, which treats an extra result for an already-retried uid as another execution — so one retry plus a cleanup failure reports 2 extra run(s) instead of 1. That layer cannot distinguish a cleanup result from a data-driven row, so a fix needs a distinguishing signal through the three tracked-internal-API Report*Test overloads. Cosmetic and edge-case.

• outcome-kind is documented in the protocol ("passed-with-retries") but implemented nowhere; now that attempts flow it could be populated for IDE filtering.

Validation

• Microsoft.Testing.Platform.UnitTests 1983 · Microsoft.Testing.Extensions.UnitTests 879 · MSTestAdapter.PlatformServices.UnitTests 1071 · TestFramework.UnitTests 1502 (net9.0) and 1376 (net48, the [Serializable] path) — all passing
• Acceptance after -pack: MSTest retry/CTRF/JUnit/TRX and platform retry + max-failed-tests + MSBuild — all passing
• New behavioral tests were mutation-checked: disabling each guard fails its test

Fixes #10292

[Copilot]

Pull request overview

Surfaces every MSTest [Retry] attempt to Microsoft.Testing.Platform, enabling terminal retry reporting and CTRF flaky-test detection.

Changes:

• Adds RetryAttemptProperty and propagates retry metadata.
• Reconciles attempts in terminal output and filters superseded results from selected consumers.
• Adds unit, acceptance, API, RFC, and changelog updates.

Show a summary per file

File	Description
test/UnitTests/Microsoft.Testing.Platform.UnitTests/Services/TestApplicationResultTests.cs	Tests exit-code filtering.
test/UnitTests/Microsoft.Testing.Platform.UnitTests/OutputDevice/Terminal/TerminalTestReporterTests.cs	Tests retry rendering and totals.
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/RetryTests.cs	Validates retry terminal output.
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/JUnitReportTests.cs	Verifies final-only JUnit results.
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/CtrfReportTests.cs	Verifies CTRF flaky reporting.
test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/Helpers/AcceptanceAssert.cs	Supports retried summary assertions.
src/TestFramework/TestFramework/InternalAPI/InternalAPI.Unshipped.txt	Tracks new internal result properties.
src/TestFramework/TestFramework/Attributes/TestMethod/TestResult.cs	Stores retry attribution.
src/TestFramework/TestFramework/Attributes/TestMethod/RetryResult.cs	Documents attempt publication.
src/TestFramework/TestFramework/Attributes/TestMethod/RetryBaseAttribute.cs	Updates retry contract documentation.
src/TestFramework/TestFramework/Attributes/TestMethod/RetryAttribute.cs	Updates retry behavior documentation.
src/Platform/Microsoft.Testing.Platform/Services/TestApplicationResult.cs	Ignores superseded outcomes.
src/Platform/Microsoft.Testing.Platform/PublicAPI/PublicAPI.Unshipped.txt	Declares the new public property.
src/Platform/Microsoft.Testing.Platform/OutputDevice/TerminalOutputDevice.DataConsumption.cs	Passes retry metadata to terminal reporting.
src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TestProgressState.cs	Reconciles host and in-process attempts.
src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.TestCompletion.cs	Annotates and counts attempts.
src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.Summary.cs	Documents retried totals.
src/Platform/Microsoft.Testing.Platform/Messages/RetryAttemptProperty.cs	Defines retry attempt metadata.
src/Platform/Microsoft.Testing.Extensions.TrxReport/TrxDataConsumer.cs	Filters superseded TRX results.
src/Platform/Microsoft.Testing.Extensions.JUnitReport/JUnitReportGenerator.cs	Filters superseded JUnit results.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsTestResultsPublisher.cs	Filters superseded published results.
src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsReporter.cs	Suppresses superseded annotations.
src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs	Flattens and tags retry attempts.
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestExecutionManager.Runner.cs	Uses final outcomes for verdicts.
src/Adapter/MSTest.TestAdapter/TestingPlatformAdapter/MSTestTestNodeConverter.cs	Adds retry properties to test nodes.
src/Adapter/MSTest.TestAdapter/Services/TestResultRecorderExtensions.cs	Preserves VSTest final-only behavior.
docs/RFCs/016-JUnit-Report.md	Documents JUnit retry handling.
docs/Changelog.md	Records the MSTest behavior change.
docs/Changelog-Platform.md	Records the platform API change.

Review details

Comments suppressed due to low confidence (1)

src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs:357

• A custom RetryBaseAttribute may legally add an empty final result array. Previously TryGetLast() returned that empty array, allowing SendTestResultsAsync to call RecordEmptyResultAsync; this combination instead retains only earlier superseded results, so no final outcome is recorded and AllPassed can return true after skipping them. Preserve the existing empty-result behavior when the final retry attempt is empty.

        IReadOnlyList<TestResult[]> retryAttempts = retryResult.AllResults;
        if (retryAttempts.Count == 0)
        {
            // A RetryBaseAttribute implementation is expected to add at least one attempt.
            throw ApplicationStateGuard.Unreachable();

• Files reviewed: 29/29 changed files
• Comments generated: 7
• Review effort level: Medium

[Copilot]

Review details

Comments suppressed due to low confidence (4)

src/Adapter/MSTest.TestAdapter/TestingPlatformAdapter/MSTestTestNodeConverter.cs:95

• Publishing these nodes exposes superseded failures to several unfiltered consumers. GitHubActionsAnnotationReporter.cs:88-117 emits a durable ::error, MSBuildConsumer.cs:118-193 forwards/counts it as a real failure, SimplifiedConsoleOutputDeviceBase.TestEvents.cs:168-197 produces a failed Browser/WASI summary, and AzureDevOpsArtifactUploader.cs:138-140 enables failure-only uploads. A fail-then-pass [Retry] test therefore still appears failed or triggers failure side effects on those surfaces. Filter superseded attempts in each final-outcome consumer and add regression tests.

        // Surface an in-process retry (MSTest's [Retry]) so the platform can tell the attempts of one test apart
        // instead of seeing repeated results for the same uid. Only added when a retry actually happened, so a
        // regular test node is byte-identical to before.
        if (result.RetryAttemptNumber > 1 || result.IsSupersededRetryAttempt)
        {
            testNode.Properties.Add(new RetryAttemptProperty(result.RetryAttemptNumber, result.IsSupersededRetryAttempt));

src/Platform/Microsoft.Testing.Platform/OutputDevice/TerminalOutputDevice.DataConsumption.cs:1

• This edit removed the UTF-8 BOM, but .editorconfig:65-67 requires utf-8-bom for every C# file. Restore the BOM to avoid an encoding-policy violation.

// Copyright (c) Microsoft Corporation. All rights reserved.

src/Platform/Microsoft.Testing.Platform/Messages/RetryAttemptProperty.cs:1

• This new C# file is missing the UTF-8 BOM required by .editorconfig:65-67. Save it as UTF-8 with BOM.

// Copyright (c) Microsoft Corporation. All rights reserved.

src/Platform/Microsoft.Testing.Platform/Messages/RetryAttemptPropertyExtensions.cs:1

• This new C# file is missing the UTF-8 BOM required by .editorconfig:65-67. Save it as UTF-8 with BOM.

// Copyright (c) Microsoft Corporation. All rights reserved.

• Files reviewed: 35/35 changed files
• Comments generated: 3
• Review effort level: Medium

[Copilot]

Review details

Suppressed comments (5)

src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TestProgressState.cs:526

• When both retry mechanisms are present, the number of executions is not the maximum of the two final coordinates. For example, two in-process attempts in host attempt 1 followed by three in host attempt 2 means five executions, but this reports three. Track the number of distinct (hostAttempt, retryAttempt) pairs per UID and use that count here; otherwise the flaky-test detail contradicts the retry totals.

                    // The attempt count is the pair (host attempt, in-process retry attempt): a test retried
                    // in-process reports 1 host attempt but several retry attempts, so the larger of the two is
                    // the number of times the test actually ran.
                    TestNodeInfoEntry resultEntry = _testUidToResults[entry.Key];
                    flaky.Add((entry.Value, Math.Max(resultEntry.LastAttemptNumber, resultEntry.LastRetryAttemptNumber)));

src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs:412

• The cleanup-attempt alignment that prevents the reported out-of-order terminal crash has no automated coverage in this change. Add a UnitTestRunnerTests case where a retry finishes on attempt 2+ and class or assembly cleanup produces a result, then assert that the cleanup result inherits the final attempt number and is not superseded.

    private static TestResult AlignAttemptNumber(TestResult result, TestResult[] precedingResults)
    {
        for (int i = precedingResults.Length - 1; i >= 0; i--)
        {
            if (!precedingResults[i].IsSupersededRetryAttempt)
            {
                result.RetryAttemptNumber = precedingResults[i].RetryAttemptNumber;

docs/RFCs/016-JUnit-Report.md:220

• These additions leave two descriptions of each retry mechanism, and the old MSTest bullet directly contradicts the new one by saying only the final attempt is surfaced. Keep the updated MSTest bullet and the existing detailed orchestrator bullet below, removing the duplicate abbreviated orchestrator entry and stale MSTest entry.

- **MSTest `[Retry]` attribute** — The MSTest adapter retries in-process and publishes **every** attempt as a `TestNodeUpdateMessage` under the same test node uid, tagged with `RetryAttemptProperty`. The JUnit report generator drops the attempts marked `IsSuperseded` (JUnit has no notion of attempts, so keeping them would inflate the suite totals), so the report still contains a single `<testcase>` row per logical test with its eventual outcome. No per-attempt disambiguation is applied.
- **`Microsoft.Testing.Extensions.Retry` (MTP-level orchestrator, `--retry-failed-tests`)** — The orchestrator re-runs the entire test-host child process on failure. Each attempt is a separate process and therefore produces **its own JUnit XML file** under the run's results directory. No special handling is required inside the report engine.
- **MSTest `[Retry]` attribute** — The MSTest adapter retries internally and surfaces only the **final** attempt as a `TestNodeUpdateMessage`. The JUnit report therefore contains a single `<testcase>` row per logical test, with its eventual outcome. No per-attempt disambiguation is applied.

docs/Changelog.md:22

• The PR explicitly dropped the (+N retried) total suffix after #10329, and the new terminal test expects a dedicated retried: line. This changelog example therefore documents output that this change does not produce.

* Report every `[Retry]` attempt to the test host instead of discarding all but the last one. A retried test now shows its attempts in the terminal (`failed (try 1) MyTest`), is reconciled in the summary (`total: 1 (+1 retried)`), and is finally detected as `flaky` (with `retries` / `retryAttempts[]`) by the CTRF report. Superseded attempts do not affect the process exit code and are filtered out of the TRX and JUnit reports, so those keep one row per test in [#10292](https://github.com/microsoft/testfx/issues/10292)

src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs:367

• This special empty-final-attempt path is not exercised by the added tests: the acceptance assets use the built-in retry implementation and always produce a final result, and UnitTestRunnerTests currently has no retry cases. Add a unit test with a custom RetryBaseAttribute returning an empty final array and verify that the empty-result/error path is preserved rather than treating the test and its dependents as passed.

This issue also appears on line 406 of the same file.

        if (retryAttempts[retryAttempts.Count - 1].Length == 0)
        {
            return [];

• Files reviewed: 38/38 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (11)

src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.TestCompletion.cs:135

• DotnetTestDataConsumer emits every newly surfaced attempt, but its success/failure IPC messages have no retry fields. Hard-coding (1, false) here means a fail-then-pass [Retry] reaches the parent under the same (host attempt, retry attempt) pair; ReportGenericTestResult then counts both outcomes instead of replacing the failure. This can make dotnet test report one ultimately passing test as two results with one failure. Carry the retry attribution through the IPC model/serializers, or filter superseded attempts before transport until that protocol exists.

            // The orchestrator attributes retries per host instance; it does not surface a test framework's
            // in-process retry attempt, so results arriving through this path are always attempt 1 of their host
            // attempt.
            retryAttemptNumber: 1,
            isRetryAttempt: false);

src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TestProgressState.cs:526

• Math.Max is not the number of executions when both retry mechanisms compose. For example, three in-process executions in host attempt 1 followed by one execution in host attempt 2 ran four times, but this reports two. Track the count of distinct (hostAttempt, retryAttempt) pairs per UID and use that count here.

                    // The attempt count is the pair (host attempt, in-process retry attempt): a test retried
                    // in-process reports 1 host attempt but several retry attempts, so the larger of the two is
                    // the number of times the test actually ran.
                    TestNodeInfoEntry resultEntry = _testUidToResults[entry.Key];
                    flaky.Add((entry.Value, Math.Max(resultEntry.LastAttemptNumber, resultEntry.LastRetryAttemptNumber)));

src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsTestResultsPublisher.cs:267

• The same filter is missing from AzureDevOpsArtifactUploader.ConsumeAsync: that consumer sets _hasTestFailures for any failed state and later emits the has-test-failures build tag. Consequently, a fail-then-pass [Retry] still tags the Azure build as having test failures even though this publisher, the reporter, and the process exit code treat it as passed. Apply IsSupersededRetryAttempt() there and add a tag regression test.

                    // A test framework that retries a test in-process reports every attempt under the same test
                    // node uid. Azure DevOps keys results by automated test name, so publishing the superseded
                    // attempts would create duplicate rows for one test; only the final attempt is published.
                    if (testNodeUpdateMessage.TestNode.IsSupersededRetryAttempt())
                    {
                        return;
                    }

docs/RFCs/016-JUnit-Report.md:219

• These bullets were inserted without replacing the existing retry bullets immediately below. The RFC now both says MSTest publishes every attempt and says it surfaces only the final attempt, while the orchestrator description is duplicated. Remove the stale MSTest bullet and one of the two orchestrator bullets.

- **MSTest `[Retry]` attribute** — The MSTest adapter retries in-process and publishes **every** attempt as a `TestNodeUpdateMessage` under the same test node uid, tagged with `RetryAttemptProperty`. The JUnit report generator drops the attempts marked `IsSuperseded` (JUnit has no notion of attempts, so keeping them would inflate the suite totals), so the report still contains a single `<testcase>` row per logical test with its eventual outcome. No per-attempt disambiguation is applied.
- **`Microsoft.Testing.Extensions.Retry` (MTP-level orchestrator, `--retry-failed-tests`)** — The orchestrator re-runs the entire test-host child process on failure. Each attempt is a separate process and therefore produces **its own JUnit XML file** under the run's results directory. No special handling is required inside the report engine.

docs/Changelog.md:22

• The (+N retried) total suffix was removed by #10329; the implementation and new terminal test now emit a dedicated retried: N test(s), M extra run(s) line. Update this changelog example so it describes the shipped output.

* Report every `[Retry]` attempt to the test host instead of discarding all but the last one. A retried test now shows its attempts in the terminal (`failed (try 1) MyTest`), is reconciled in the summary (`total: 1 (+1 retried)`), and is finally detected as `flaky` (with `retries` / `retryAttempts[]`) by the CTRF report. Superseded attempts do not affect the process exit code and are filtered out of the TRX and JUnit reports, so those keep one row per test in [#10292](https://github.com/microsoft/testfx/issues/10292)

src/TestFramework/TestFramework/Attributes/TestMethod/RetryResult.cs:24

• RetryResult is shared by the MTP and VSTest paths, but the VSTest recorder intentionally drops superseded results. Therefore “All attempts are reported to the test host” is not true for VSTest. Scope this statement to Microsoft.Testing.Platform or describe the results as retained for host integrations.

    /// <remarks>
    /// All attempts are reported to the test host: the last one as the test's outcome, the earlier ones tagged as
    /// superseded so tooling can surface the retry (see the platform's <c>RetryAttemptProperty</c>).

src/TestFramework/TestFramework/Attributes/TestMethod/RetryBaseAttribute.cs:34

• This return documentation applies to both host integrations, but VSTest intentionally filters the superseded results. Scope the reporting claim to Microsoft.Testing.Platform so consumers are not promised per-attempt VSTest results.

    /// Returns a <see cref="RetryResult"/> object that contains the results of all attempts. The last added
    /// element determines the test outcome; the earlier attempts are reported to the test host as superseded
    /// attempts so tooling can surface the retry (attempt count, flaky detection, per-attempt error messages).

src/TestFramework/TestFramework/Attributes/TestMethod/RetryAttribute.cs:88

• This return documentation applies to both host integrations, but VSTest intentionally filters the superseded results. Scope the reporting claim to Microsoft.Testing.Platform so consumers are not promised per-attempt VSTest results.

    /// Returns a <see cref="RetryResult"/> object that contains the results of all attempts. The last added
    /// element determines the test outcome; the earlier attempts are reported to the test host as superseded
    /// attempts so tooling can surface the retry.

src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs:367

• This branch preserves a subtle empty-result contract, but there is no UnitTestRunner test exercising a custom RetryBaseAttribute whose final attempt returns an empty array. Add a regression test proving the runner returns an empty result and does not treat the test as passing; otherwise this false-pass fix can regress unnoticed.

        if (retryAttempts[retryAttempts.Count - 1].Length == 0)
        {
            return [];

src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs:413

• The cleanup-attempt alignment fixes an out-of-order terminal crash, but no unit or acceptance test covers a retry followed by a class/assembly cleanup result. Add a regression test that verifies the cleanup result inherits the final retry number and is not marked superseded.

            if (!precedingResults[i].IsSupersededRetryAttempt)
            {
                result.RetryAttemptNumber = precedingResults[i].RetryAttemptNumber;
                break;

test/IntegrationTests/MSTest.Acceptance.IntegrationTests/CtrfReportTests.cs:185

• This generated test asset's private static field does not follow the repository's s_ camel-case rule (.editorconfig:601-603). Rename it to s_attempts and update its uses.

• Files reviewed: 38/38 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (3)

src/Adapter/MSTest.TestAdapter/TestingPlatformAdapter/MSTestTestNodeConverter.cs:104

• Publishing the superseded failures now also reaches VideoRecorderSessionHandler, which does not inspect RetryAttemptProperty: it marks _anyTestFailed, appends one TestRecord per attempt, and in default on-failure mode retains video for a test whose final outcome passed. Per-test/always mode also emits duplicate clips and session chapters for the same logical test. Filter superseded updates in the video consumer (or explicitly model attempt history there) so its persistence and per-test artifact semantics follow the final outcome.

        if (result.RetryAttemptNumber > 1 || result.IsSupersededRetryAttempt)
        {
            testNode.Properties.Add(new RetryAttemptProperty(result.RetryAttemptNumber, result.IsSupersededRetryAttempt));

docs/RFCs/016-JUnit-Report.md:220

• This adds replacement retry bullets without removing the old ones below. The section now says both that MSTest publishes every attempt and that it surfaces only the final attempt, and describes the orchestrator twice. Keep the new MSTest bullet and the existing detailed orchestrator bullet at line 221, removing the duplicate/stale entries.

- **MSTest `[Retry]` attribute** — The MSTest adapter retries in-process and publishes **every** attempt as a `TestNodeUpdateMessage` under the same test node uid, tagged with `RetryAttemptProperty`. The JUnit report generator drops the attempts marked `IsSuperseded` (JUnit has no notion of attempts, so keeping them would inflate the suite totals), so the report still contains a single `<testcase>` row per logical test with its eventual outcome. No per-attempt disambiguation is applied.
- **`Microsoft.Testing.Extensions.Retry` (MTP-level orchestrator, `--retry-failed-tests`)** — The orchestrator re-runs the entire test-host child process on failure. Each attempt is a separate process and therefore produces **its own JUnit XML file** under the run's results directory. No special handling is required inside the report engine.

docs/Changelog.md:22

• The changelog still advertises the removed total: 1 (+1 retried) suffix. The integrated #10329 output uses a dedicated retried: 1 test(s), 1 extra run(s) line, as asserted by the new terminal test, so update this example to avoid documenting output users will never see.

* Report every `[Retry]` attempt to the test host instead of discarding all but the last one. A retried test now shows its attempts in the terminal (`failed (try 1) MyTest`), is reconciled in the summary (`total: 1 (+1 retried)`), and is finally detected as `flaky` (with `retries` / `retryAttempts[]`) by the CTRF report. Superseded attempts do not affect the process exit code and are filtered out of the TRX and JUnit reports, so those keep one row per test in [#10292](https://github.com/microsoft/testfx/issues/10292)

• Files reviewed: 38/38 changed files
• Comments generated: 1
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (4)

src/Platform/Microsoft.Testing.Extensions.AzureDevOpsReport/AzureDevOpsReporter.cs:120

• The Azure DevOps failure handling is still incomplete: AzureDevOpsArtifactUploader.ConsumeAsync at lines 138-140 marks every failed update as _hasTestFailures, and lines 182-185 consequently emit the TestFailuresTag. A fail-then-pass [Retry] run will therefore still receive a test-failures build tag from its superseded attempt. Apply the same superseded-attempt guard in that consumer.

        if (nodeUpdateMessage.TestNode.IsSupersededRetryAttempt())

docs/RFCs/016-JUnit-Report.md:221

• This adds a second pair of retry-mechanism bullets without removing the existing pair below. The stale MSTest bullet still says only the final attempt is surfaced, directly contradicting the new behavior, while the orchestrator is documented twice. Keep the new MSTest explanation and the existing detailed orchestrator explanation only.

- **MSTest `[Retry]` attribute** — The MSTest adapter retries in-process and publishes **every** attempt as a `TestNodeUpdateMessage` under the same test node uid, tagged with `RetryAttemptProperty`. The JUnit report generator drops the attempts marked `IsSuperseded` (JUnit has no notion of attempts, so keeping them would inflate the suite totals), so the report still contains a single `<testcase>` row per logical test with its eventual outcome. No per-attempt disambiguation is applied.
- **`Microsoft.Testing.Extensions.Retry` (MTP-level orchestrator, `--retry-failed-tests`)** — The orchestrator re-runs the entire test-host child process on failure. Each attempt is a separate process and therefore produces **its own JUnit XML file** under the run's results directory. No special handling is required inside the report engine.

src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TestProgressState.cs:526

• Math.Max undercounts when both retry mechanisms advance. For example, (host 1, retry 1), (host 1, retry 2), then (host 2, retry 1) represents three executions, but the final tuple makes this report 2 attempts. Track the number of distinct (host attempt, in-process attempt) pairs per UID as transitions arrive instead of deriving it from only the final pair, and cover a combined-retry sequence.

                    flaky.Add((entry.Value, Math.Max(resultEntry.LastAttemptNumber, resultEntry.LastRetryAttemptNumber)));

docs/Changelog.md:22

• The terminal no longer uses the total: N (+M retried) suffix after #10329; it emits a dedicated retried: line, as the updated terminal test asserts. Update this release note so it does not promise output that this PR cannot produce.

* Report every `[Retry]` attempt to the test host instead of discarding all but the last one. A retried test now shows its attempts in the terminal (`failed (try 1) MyTest`), is reconciled in the summary (`total: 1 (+1 retried)`), and is finally detected as `flaky` (with `retries` / `retryAttempts[]`) by the CTRF report. Superseded attempts do not affect the process exit code and are filtered out of the TRX and JUnit reports, so those keep one row per test in [#10292](https://github.com/microsoft/testfx/issues/10292)

• Files reviewed: 38/38 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (4)

src/Platform/Microsoft.Testing.Platform/OutputDevice/SimplifiedConsoleOutputDeviceBase.TestEvents.cs:146

• isSupersededAttempt suppresses the outcome switch, but the active-test tracker is completed before that switch. MSTest emits one InProgress update for the whole retry sequence, so a superseded terminal update removes the UID from slow-test tracking before the final outcome; HangDumpActivityIndicator correctly preserves the same UID until the final attempt. Gate _activeTestTracker.Complete on !isSupersededAttempt.

                bool isSupersededAttempt = testNodeStateChanged.TestNode.IsSupersededRetryAttempt();

src/Platform/Microsoft.Testing.Platform/Messages/RetryAttemptProperty.cs:16

• The new stream contract also affects SlowTestReporterBase: its terminal-state branch at SharedExtensionHelpers/SlowTestReporterBase.cs:141-145 removes the UID for every result, including a superseded one. Azure DevOps and GitHub slow-test reporters inherit that behavior, so a framework that uses one InProgress update for the retry sequence stops receiving slow-test diagnostics after the first attempt. Apply the same superseded-attempt guard used by the hang-dump and simplified-console consumers.

/// A test framework that retries a test in-process (for example MSTest's <c>[Retry]</c> attribute) publishes one
/// <see cref="TestNodeUpdateMessage"/> per attempt, each carrying this property. Consumers that want a single row
/// per test - TRX, JUnit, the process exit code - ignore updates whose <see cref="IsSuperseded"/> is
/// <see langword="true"/>. Consumers that want the whole history - CTRF's <c>retryAttempts[]</c> / <c>flaky</c>,
/// the HTML report, the terminal's retried/flaky counters - keep them all.

docs/RFCs/016-JUnit-Report.md:220

• The next bullet still describes exposing every MSTest attempt as “future MSTest behavior,” which now directly contradicts this updated bullet. Reword that generic case to cover frameworks whose repeated updates are not tagged with RetryAttemptProperty.

- **MSTest `[Retry]` attribute** — The MSTest adapter retries in-process and publishes **every** attempt as a `TestNodeUpdateMessage` under the same test node uid, tagged with `RetryAttemptProperty`. The JUnit report generator drops the attempts marked `IsSuperseded` (JUnit has no notion of attempts, so keeping them would inflate the suite totals), so the report still contains a single `<testcase>` row per logical test with its eventual outcome. No per-attempt disambiguation is applied.

src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs:410

• Stamping a class/assembly cleanup result with the final retry number prevents the ordering crash, but it also makes the terminal classify that ancillary cleanup result as another retry execution. For a fail-then-pass test whose cleanup fails, ReportGenericTestResult sees the cleanup in the same retry pair and increments RetriedExecutions again, so the summary reports two extra runs although the test was retried once. Preserve the ordering metadata without counting cleanup as a test-method retry (for example by carrying an ancillary-result marker through the terminal path).

    private static TestResult AlignAttemptNumber(TestResult result, TestResult[] precedingResults)
    {
        for (int i = precedingResults.Length - 1; i >= 0; i--)
        {
            if (!precedingResults[i].IsSupersededRetryAttempt)

• Files reviewed: 42/42 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (1)

src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs:412

• Stamping an appended cleanup result with the final retry number avoids the out-of-order exception, but it also makes the terminal count that cleanup result as another retry execution. After attempt 1 fails and attempt 2 passes, _retriedUids already contains the UID; the cleanup result then enters TestProgressState with the same (host, retry) pair, so the same-attempt branch increments _retriedExecutions again. A single retry followed by a class/assembly cleanup failure therefore reports 2 extra run(s) instead of 1. Cleanup results need to retain ordering without participating in retry-execution accounting (and this scenario should be covered for both cleanup levels).

                result.RetryAttemptNumber = precedingResults[i].RetryAttemptNumber;

• Files reviewed: 42/42 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (3)

src/Adapter/MSTest.TestAdapter/TestingPlatformAdapter/MSTestTestNodeConverter.cs:104

• These newly published superseded terminal updates also reach SlowTestReporterBase, whose ConsumeAsync removes a UID on any non-in-progress state (SharedExtensionHelpers/SlowTestReporterBase.cs:141-145). Both the GitHub Actions and Azure DevOps slow-test reporters therefore stop tracking at the first superseded attempt instead of the final outcome. Apply the same superseded-attempt guard there and cover both derived reporters.

        if (result.RetryAttemptNumber > 1 || result.IsSupersededRetryAttempt)
        {
            testNode.Properties.Add(new RetryAttemptProperty(result.RetryAttemptNumber, result.IsSupersededRetryAttempt));

src/Platform/Microsoft.Testing.Platform/OutputDevice/SimplifiedConsoleOutputDeviceBase.TestEvents.cs:146

• The active-test bookkeeping still treats this superseded result as terminal. MSTest emits one RecordStartAsync before RunSingleTestAsync and then publishes the flattened results, so line 158 removes the UID from ActiveTestTracker and line 209 clears its progress display on attempt 1. Defer both actions until !isSupersededAttempt; otherwise active/slow-test tracking ends before the final outcome is published.

                // A test framework that retries in-process reports every attempt under the same test node uid. A
                // superseded attempt is not the test's outcome, so it must not be counted or printed as a failure -
                // a fail-then-pass [Retry] test would otherwise show a failed summary here. The active-test
                // bookkeeping below still runs: each attempt has its own in-progress/terminal pair.
                bool isSupersededAttempt = testNodeStateChanged.TestNode.IsSupersededRetryAttempt();

src/Platform/Microsoft.Testing.Platform/OutputDevice/TerminalOutputDevice.DataConsumption.cs:218

• This reduces the retry property to its number/presence and drops IsSuperseded. TerminalTestReporter.TestCompleted consequently removes the UID from TestNodeResultsState for every attempt, so --show-active-tests drops a test on the first superseded result even though the sequence has only one in-progress update. Carry the superseded flag through and remove the running node only for the final attempt.

                // A test framework that retries in-process reports every attempt under the same uid; the attempt
                // number lets the reporter replace (rather than double-count) the earlier attempt and annotate the
                // per-test line with "(try N)".
                int retryAttemptNumber = retryAttempt?.AttemptNumber ?? 1;
                bool isRetryAttempt = retryAttempt is not null;

• Files reviewed: 42/42 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (2)

src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs:306

• The same retry-accounting inflation occurs for an assembly-cleanup failure: after the real final retry result has registered the UID as retried, this aligned same-pair result takes the branch that increments _retriedExecutions. The summary therefore counts assembly cleanup as another retry-driven test run. Preserve the ordering fix, but mark or handle cleanup outcomes so they do not increase the retry execution count.

                    result = [.. result, AlignAttemptNumber(assemblyCleanupResult, result)];

src/Adapter/MSTestAdapter.PlatformServices/Execution/UnitTestRunner.RunSingleTest.cs:278

• Aligning a class-cleanup failure to the final retry pair makes ReportGenericTestResult treat it as another result in that retry attempt. Because the UID is already in _retriedUids, the same-pair branch increments _retriedExecutions, so a test that ran twice and then had ClassCleanup fail is reported as having two extra runs instead of one. Cleanup results need to preserve ordering without contributing to retry execution accounting (likely via explicit result-kind metadata or equivalent terminal-state handling).

This issue also appears on line 306 of the same file.

                        result = [.. result, AlignAttemptNumber(cleanupResult, result)];

• Files reviewed: 43/43 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

Suppressed comments (5)

src/TestFramework/TestFramework/Attributes/TestMethod/RetryResult.cs:24

• This states that every attempt reaches the test host, but the VSTest recorder explicitly drops superseded results in TestResultRecorderExtensions.cs:65-71; only Microsoft.Testing.Platform receives the full history. Qualify the behavior so consumers do not expect VSTest tooling to expose AllResults.
  src/Platform/Microsoft.Testing.Platform/OutputDevice/TerminalOutputDevice.DataConsumption.cs:1
• This changed file is still BOM-less, but .editorconfig:67 requires UTF-8 with BOM for every C# file. Restore the BOM so the file complies with the repository encoding rule.

// Copyright (c) Microsoft Corporation. All rights reserved.

src/TestFramework/TestFramework/Attributes/TestMethod/RetryBaseAttribute.cs:34

• The earlier attempts are not reported to VSTest: TestResultRecorderExtensions.cs:65-71 drops superseded results before calling the VSTest recorder. Qualify this as Microsoft.Testing.Platform behavior so custom retry authors are not promised per-attempt tooling on both hosts.
  src/TestFramework/TestFramework/Attributes/TestMethod/RetryAttribute.cs:88
• This is not true for the VSTest path, which deliberately drops superseded results in TestResultRecorderExtensions.cs:65-71. Qualify the sentence as Microsoft.Testing.Platform behavior to keep this public override's documentation accurate.
  src/TestFramework/TestFramework/Attributes/TestMethod/TestResult.cs:225
• This claim conflicts with the VSTest behavior documented below and implemented in TestResultRecorderExtensions.cs:65-71: VSTest receives only the final result, while Microsoft.Testing.Platform receives every attempt. Qualify this remark so it does not describe the platform-specific behavior as universal.

• Files reviewed: 43/43 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[Copilot]

Review details

• Files reviewed: 43/43 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[github-actions]

🧪 Test quality grade — PR #10295

No new or modified test methods were identified in the changed regions
of this PR. Nothing to grade.

_{Re-run with /grade-tests.}

🤖 Automated content by GitHub Copilot. Generated by the Grade Tests on PR (on open / sync) workflow. · sonnet46 40 AIC · ⌖ 4.02 AIC · ⊞ 11.8K · [◷]( · ◷)