Skip to content

[fix] Fix DataDrivenTest parent result being double-counted in TRX ResultSummary #15751

@nohwnd

Description

@nohwnd

🤖 Automated Fix

This PR was generated automatically to address issue #15643.

Root Cause

When a data-driven test (e.g. MSTest with [DataRow]) runs, the test framework emits one parent result (resultType=DataDrivenTest) followed by one inner result per data row (resultType=DataDrivenDataRow).

TrxLogger.TestResultHandler incremented TotalTestCount/PassedTestCount/FailedTestCount for every received TestResult event, including the parent aggregation. This caused the parent to be counted in addition to each inner row, inflating the Counters in the TRX ResultSummary compared to the console output.

Example: 1 test method with 2 [DataRow] attributes resulted in total="3" in the TRX file instead of the expected total="2".

Fix

When the first DataDrivenDataRow inner result arrives and reveals that its parent is a DataDrivenTest aggregation, the fix retroactively subtracts the parent's contribution from the counters using an atomic TryAdd on a ConcurrentDictionary<Guid, bool> keyed by the parent execution ID. Subsequent inner results for the same parent are not adjusted again.

Ordered-test hierarchies are unaffected because their parent uses a different TestType.

Tests

  • Updated existing test TestResultHandlerShouldAddHierarchicalResultsIfParentTestResultIsPresent — asserts TotalTestCount == 2 (was 3) for a DataDriven parent + 2 inner results.
  • Added new test TestResultHandlerShouldNotDoubleCountDataDrivenTestParent — verifies TotalTestCount == 2, PassedTestCount == 2, FailedTestCount == 0 for a parent + 2 passing DataDrivenDataRow results.

Fixes #15643

🔍 Triaged by Issue Repro Triage & Auto-Fix 🔍


Note

This was originally intended as a pull request, but the git push operation failed.

Workflow Run: View run details and download bundle artifact

The bundle file is available in the agent artifact in the workflow run linked above.

To create a pull request with the changes:

# Download the artifact from the workflow run
gh run download 25729300675 -n agent -D /tmp/agent-25729300675

# Fetch the bundle into a local branch
git fetch /tmp/agent-25729300675/aw-fix-issue-15643.bundle refs/heads/fix/issue-15643:refs/heads/fix/issue-15643-359933eb5766c1fd
git checkout fix/issue-15643-359933eb5766c1fd

# Push the branch to origin
git push origin fix/issue-15643-359933eb5766c1fd

# Create the pull request
gh pr create --title '[fix] Fix DataDrivenTest parent result being double-counted in TRX ResultSummary' --base main --head fix/issue-15643-359933eb5766c1fd --repo microsoft/vstest

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions