Impact Efficiency Report: progressive disclosure + fix artificial/missing data + recommendations

[Copilot]

The generated report (e.g. issue #39811) was a wall of text with no collapsing, used "artificially depressed" to describe a computable metric, inflated the Human Work count by counting likely-agentic PRs as human due to attribution gaps, and provided no guidance on how to improve accuracy in future cycles.

Progressive disclosure

• Executive Summary stays plain at the top (always visible)
• Every other section (Summary, Agentic Work by Objective, Which Workflows Drove That Impact, Top Outcomes, Unmapped Outcomes, Interpretation, Data Quality, Recommendations, Human Work) is wrapped in <details><summary> with a label that includes the section's key number

Scope-adjusted IE replaces "artificially depressed"

• Interpretation section now instructs the agent to compute and report a scope-adjusted IE: IE denominated only by AIC from workflows that produced at least one analyzed outcome, presented alongside the full-denominator IE
• The word "artificially" is prohibited; instead the denominator composition is explained concretely

Human Work attribution gap

• Before counting unattributed merged PRs as human, the agent now filters PRs with bot-style title patterns ([docs], [linter-miner], [fix], etc.) and bot authors into a separate "Likely agentic (attribution gap)" sub-table
• The confirmed-human count breakdown is updated to reflect the reclassification

N/A → — for unknown per-category AIC/IE

• Shows — (not N/A) when workflow attribution is missing, with a precise note that attribution is structurally absent — not computed as zero

Actionable Recommendations

• New Recommendations section placed between Data Quality and Human Work
• Each recommendation is emitted only when the cycle's own data confirms the gap (no generic advice)
• Covers six gap categories: workflow attribution (direct workflow_run_id stamping), linked-issue coverage, objective label coverage (lists the specific unmapped labels observed), AIC per-outcome attribution (minimal dataset join needed), PR dataset cap, and likely-agentic reclassification
• Each item includes the concrete gap metric with numbers, the expected effect on report accuracy (e.g. "would convert up to N excluded PRs into analyzable outcomes"), and the owner/mechanism (workflow name, script, PR template)
• Sorted by expected impact on report accuracy, highest first

[mnkiefer]

@copilot Add actionable recommendations to improve the accuracy and evidence for this report. It should be based on deterministic outcomes and objective values and attributed workflows can be directly attributed.

[Copilot]

@copilot Add actionable recommendations to improve the accuracy and evidence for this report. It should be based on deterministic outcomes and objective values and attributed workflows can be directly...

Added a Recommendations section (commit a2c6694-follow-on) placed between Data Quality and Human Work. Each recommendation is:

• Evidence-grounded — only emitted when the cycle's own data confirms the gap (e.g. a concrete exclusion count, an unattributed outcome count, a capped dataset)
• Specific: includes the gap metric with numbers, the expected effect on accuracy (e.g. "would convert up to N excluded PRs into analyzable outcomes"), and the owner/mechanism (workflow name, script, PR template)

The six covered gap categories are: workflow attribution (direct workflow_run_id stamping), linked-issue coverage, objective label coverage (lists the specific unmapped labels), AIC per-outcome attribution (minimal dataset join needed), PR dataset cap, and likely-agentic reclassification.

[mnkiefer]

@copilot Revert ‎.github/workflows/example-failure-category-filter.lock.yml file changes.

[Copilot]

@copilot Revert ‎.github/workflows/example-failure-category-filter.lock.yml file changes.

Reverted in the latest commit. The make recompile step had moved the zizmor: ignore comment from a standalone line to an inline position — the file is now restored to its original state.

[Copilot]

Pull request overview

Improves the “Impact Efficiency Report” workflow instructions to generate more readable and actionable GitHub issues by introducing progressive disclosure formatting, refining metric language/definitions, correcting attribution-related counting guidance, and adding evidence-grounded recommendations.

Changes:

• Adds progressive disclosure guidance for report sections using collapsible <details><summary> blocks while keeping the Executive Summary always visible.
• Replaces “artificially depressed” framing with a scope-adjusted IE concept and clearer denominator explanation, plus — for structurally-unknown AIC/IE.
• Updates Human Work guidance to split “likely agentic (attribution gap)” from confirmed human, and adds a Recommendations section driven by observed gaps.

Show a summary per file

File	Description
.github/workflows/objective-impact-report.md	Updates report-generation instructions: progressive disclosure, scope-adjusted IE guidance, attribution-gap handling, and evidence-based recommendations.
.github/workflows/objective-impact-report.lock.yml	Regenerates compiled lock metadata to reflect the markdown workflow changes.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 2/2 changed files
• Comments generated: 4

[mnkiefer]

@copilot Recompile workflow

[Copilot]

@copilot Recompile workflow

Done — objective-impact-report.lock.yml recompiled in the latest commit.

[github-actions]

@copilot fix the failing validate-yaml check, then rerun CI and recompile the workflow lock if needed.

Generated by 👨‍🍳 PR Sous Chef · ◷

[github-actions]

Hey @copilot-swe-agent 👋 — great work on the impact report improvements! The progressive disclosure structure, scope-adjusted IE, likely-agentic reclassification, and the new Recommendations section are all clear wins for report usability and accuracy.

One thing to address before merging:

• No test coverage — the diff only touches the workflow prompt spec (objective-impact-report.md) and its auto-generated lock file. If this project has a prompt-validation test suite or a snapshot/golden-file test that exercises the report spec (e.g. checking that required section headers are present, that prohibited strings like "artificially" are absent, or that the <details> wrapper pattern is consistent), those should be updated or added to cover the new sections and wording constraints introduced here.

If you'd like a hand, you can assign this prompt to your coding agent:

Review the test suite for the objective-impact-report workflow spec in .github/workflows/objective-impact-report.md.
Add or update tests to cover the changes introduced in the current branch:
1. Verify the word "artificially" does not appear anywhere in the spec.
2. Verify every section after the Executive Summary is wrapped in a <details><summary>...</summary>...</details> block.
3. Verify the new "Recommendations" section is present and appears between "Data Quality" and "Human Work".
4. Verify the "Likely agentic (attribution gap)" sub-table instructions are present in the Human Work section.
If no automated test harness exists for .md workflow specs, create a minimal shell or Node script under scripts/ that performs these static checks and note it in the PR description.

Generated by ✅ Contribution Check · ◷

[github-actions]

Please add or update test coverage for .github/workflows/objective-impact-report.md, then rerun CI if needed.

Generated by 👨‍🍳 PR Sous Chef · ◷