feat: cross-process test report aggregation — merged HTML report + single GitHub summary#6415
Conversation
Multiple test projects run as separate processes, producing N summary blocks and N HTML reports with no way to merge them. Each process now persists a JSON sidecar of its report data; sibling processes cooperatively rewrite one marker-delimited summary block and one merged HTML report under a file lock (last finisher leaves the complete aggregate). On by default where a shared directory is resolvable (GitHub Actions or TUNIT_AGGREGATE_DIR); TUNIT_AGGREGATE_REPORTS=off to disable, =defer for multi-step pipelines with the new tunit-report dotnet tool (TUnit.Reporting.Tool) merging at the end.
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| BestPractice | 1 medium |
| CodeStyle | 52 minor |
🟢 Metrics 364 complexity
Metric Results Complexity 364
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
Full artifact-based workflow: per-job sidecar upload (defer mode), download-artifact with merge-multiple onto one runner, tunit-report merge for the combined HTML + single summary block; plus gh run download for combining separate workflow runs.
GitHub Actions
Failure
GitHub Actions
Failure
GitHub Actions
Failure
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b71373e417
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@claude review |
|
Claude finished @thomhurst's task in 1m 58s —— View job Claude finished @thomhurst's taskTodo list
ReviewOverall this is a well-scoped design: no daemon/coordination protocol, atomic writes, and a genuinely simple last-writer-wins model over a monotonically growing input. The lock/splice/merge separation is clean and the "torn marker → append, never splice" rule in Confirmed — both Codex findings are real bugs:
Other observations (no action required, noted for awareness):
I did not run the build/test suite (no tool access to |
Single lock cycle: HtmlReporter now drives sidecar write, merged HTML and the summary region in one env-parse/lock/scan pass, calling back into GitHubReporter for the region write; previously both reporters independently parsed env, locked and re-read every sidecar. Sidecar JSON serializes once to UTF-8 bytes for both copies. Shared AtomicFile helper replaces two swap implementations. ReportTestResult is a record so merging uses 'with' instead of a drift-prone manual copy. Sidecar reader collapses eight loops into a generic ReadArray. Tool links real constants via new constants-only partials (TUnitActivitySource, ClassTimelineAttribute) instead of value-mirroring shims; merged report name and summary cap live in linked files. Duration formatting and the failures-per-group cap are shared between the per-suite and aggregated summaries.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5b1043510d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Code review (re-review after
|
Review findings (Codex + Claude): GetInt32() on a non-Int32-representable schemaVersion threw FormatException past the catch(JsonException) guard, letting one corrupt sidecar kill the whole merge; the tool's per-file read was unguarded, so a locked or unreadable sidecar exited the tool with code 2 instead of counting as skipped. schemaVersion and the int helpers now use TryGetInt32; the tool skips unreadable files like the engine does.
GitHub Actions
Failure
GitHub Actions
Failure
GitHub Actions
Failure
Code review (re-review after
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f8e61a0d71
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Sidecar reader now treats wrong-shaped nested values (groups:[1] throws InvalidOperationException, not JsonException) and non-finite numeric literals (1e9999 parses to Infinity) as malformed input instead of letting them escape or poison duration math. Merged reports re-tag tunit.test.class span tags when class names are disambiguated, so per-class timelines survive multi-TFM merges. Source-control metadata and link templates are kept only when every suite agrees — merging sidecars from different commits no longer links tests to the wrong revision. SpanData is a record to support the retag via 'with'.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 00bd7bb18b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Code review (re-review of
|
1 participant
Closes #4522. Addresses discussion #6408.
Problem
Running many test projects (solution
dotnet test, per-microservicedotnet run) spawns one process per suite. Each process emits its own HTML report and its own GitHub step summary block — N projects means N blocks and N artifacts, and failures get buried. Processes can't see each other, so nothing merges.Key idea: nobody needs to know who finishes last
GITHUB_STEP_SUMMARYis a per-step file that stays freely rewritable until the step ends (the runner reads it once, at step completion). So every finishing process persists its results, takes a cross-process file lock, and re-renders one merged block from all results present so far — the true last finisher naturally leaves the complete aggregate. Last-writer-wins over monotonically-growing input needs no coordination protocol, no daemon, no "am I last?" answer.What's included
1. JSON sidecar (foundation, on by default)
Every HTML report now gets a machine-readable sibling:
{asm}-{os}-{tfm}.tunit-report.json— a faithful, versioned serialization ofReportData(hand-writtenUtf8JsonWriter/JsonDocument, AOT-safe, forward-compatible reader). Opt out:TUNIT_DISABLE_JSON_REPORT. This is the substrate everything else consumes.2. Cooperative in-engine aggregation (ON by default where a shared dir is resolvable — GitHub Actions, or explicit
TUNIT_AGGREGATE_DIR; escape hatchTUNIT_AGGREGATE_REPORTS=off)For the single-step/many-processes layout (the #4522 scenario). Each process drops its sidecar into a shared dir (auto-derived on GHA from
RUNNER_TEMP+ run/attempt/job;TUNIT_AGGREGATE_DIRelsewhere), then under a lock regenerates:echo >>lines) is never touched. Conservative marker pairing + temp-file-and-File.Replacewrites mean even a killed process can't corrupt the file.3.
tunit-reportdotnet tool (new packageTUnit.Reporting.Tool)For layouts cooperative merging can't cover — one step per project, or matrix jobs across machines (steps/jobs each get their own summary file, so a later step can't rewrite an earlier block):
The tool compiles the engine's own serializer/merger/renderer sources via
<Compile Include>links (no MTP dependency, tiny payload), so the tool always reads exactly what the engine writes.4. Docs — new guide
docs/guides/report-aggregation+ cross-links from the HTML-report and CI/CD-reporting pages.Verified
TUnit.UnitTestsunder net8.0 + net10.0 with simulated GHA env sharing one summary file — got exactly one marked block ("2 suites", 482 tests, per-runtime labels), pre-existing foreign summary content preserved, merged HTML rendered both suites.tunit-report mergeon crafted sidecars: merged HTML + summary with failure grouping, flaky section, source links, artifact links.PackAsTool); added toGetPackageProjectsModuleandTUnit.slnx.Design notes
defer(documented). Plain local runs (no resolvable shared dir) silently no-op.RunEngineTestsModulesetsofffor its child processes — hundreds of short-lived, deliberately-failing suites would otherwise re-merge the job report on every exit.SourceLinkstemplates weren't persisted in the sidecar (merged reports would have lost clickable source links). Sidecar round-trip now covers them.TUNIT_DISABLE_HTML_REPORTERdisables sidecars and therefore aggregation.ReportData.TotalDurationMscame out as ~4ms for a 2.6s run when the session span is present — the merged outputs sidestep it by deriving wall-clock from test timestamps, but the per-suite HTML header may be affected on main too.