You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This PR was generated automatically to address issue #15643.
Root Cause
When a data-driven test (e.g. MSTest with [DataRow]) runs, the test framework emits one parent result (resultType=DataDrivenTest) followed by one inner result per data row (resultType=DataDrivenDataRow).
TrxLogger.TestResultHandler incremented TotalTestCount/PassedTestCount/FailedTestCount for every received TestResult event, including the parent aggregation. This caused the parent to be counted in addition to each inner row, inflating the Counters in the TRX ResultSummary compared to the console output.
Example: 1 test method with 2 [DataRow] attributes resulted in total="3" in the TRX file instead of the expected total="2".
Fix
When the first DataDrivenDataRow inner result arrives and reveals that its parent is a DataDrivenTest aggregation, the fix retroactively subtracts the parent's contribution from the counters using an atomic TryAdd on a ConcurrentDictionary<Guid, bool> keyed by the parent execution ID. Subsequent inner results for the same parent are not adjusted again.
Ordered-test hierarchies are unaffected because their parent uses a different TestType.
Tests
Updated existing test TestResultHandlerShouldAddHierarchicalResultsIfParentTestResultIsPresent — asserts TotalTestCount == 2 (was 3) for a DataDriven parent + 2 inner results.
Added new test TestResultHandlerShouldNotDoubleCountDataDrivenTestParent — verifies TotalTestCount == 2, PassedTestCount == 2, FailedTestCount == 0 for a parent + 2 passing DataDrivenDataRow results.
The bundle file is available in the agent artifact in the workflow run linked above.
To create a pull request with the changes:
# Download the artifact from the workflow run
gh run download 25729300675 -n agent -D /tmp/agent-25729300675
# Fetch the bundle into a local branch
git fetch /tmp/agent-25729300675/aw-fix-issue-15643.bundle refs/heads/fix/issue-15643:refs/heads/fix/issue-15643-359933eb5766c1fd
git checkout fix/issue-15643-359933eb5766c1fd
# Push the branch to origin
git push origin fix/issue-15643-359933eb5766c1fd
# Create the pull request
gh pr create --title '[fix] Fix DataDrivenTest parent result being double-counted in TRX ResultSummary' --base main --head fix/issue-15643-359933eb5766c1fd --repo microsoft/vstest
🤖 Automated Fix
This PR was generated automatically to address issue #15643.
Root Cause
When a data-driven test (e.g. MSTest with
[DataRow]) runs, the test framework emits one parent result (resultType=DataDrivenTest) followed by one inner result per data row (resultType=DataDrivenDataRow).TrxLogger.TestResultHandlerincrementedTotalTestCount/PassedTestCount/FailedTestCountfor every receivedTestResultevent, including the parent aggregation. This caused the parent to be counted in addition to each inner row, inflating theCountersin the TRXResultSummarycompared to the console output.Example: 1 test method with 2
[DataRow]attributes resulted intotal="3"in the TRX file instead of the expectedtotal="2".Fix
When the first
DataDrivenDataRowinner result arrives and reveals that its parent is aDataDrivenTestaggregation, the fix retroactively subtracts the parent's contribution from the counters using an atomicTryAddon aConcurrentDictionary<Guid, bool>keyed by the parent execution ID. Subsequent inner results for the same parent are not adjusted again.Ordered-test hierarchies are unaffected because their parent uses a different
TestType.Tests
TestResultHandlerShouldAddHierarchicalResultsIfParentTestResultIsPresent— assertsTotalTestCount == 2(was3) for a DataDriven parent + 2 inner results.TestResultHandlerShouldNotDoubleCountDataDrivenTestParent— verifiesTotalTestCount == 2,PassedTestCount == 2,FailedTestCount == 0for a parent + 2 passing DataDrivenDataRow results.Fixes #15643
Note
This was originally intended as a pull request, but the git push operation failed.
Workflow Run: View run details and download bundle artifact
The bundle file is available in the
agentartifact in the workflow run linked above.To create a pull request with the changes: