Skip to content

feat: add test-coverage prompt with artifact-reading guidance - #18273

Merged
pelikhan merged 4 commits into
mainfrom
copilot/update-create-agentic-workflow
Feb 25, 2026
Merged

feat: add test-coverage prompt with artifact-reading guidance#18273
pelikhan merged 4 commits into
mainfrom
copilot/update-create-agentic-workflow

Conversation

Copilot AI commented Feb 25, 2026

Copy link
Copy Markdown
Contributor
  • Create .github/aw/test-coverage.md with artifact-reading guidance (preferred: gh run download / actions toolset; fallback: run tests)
  • Update .github/agents/agentic-workflows.agent.md to link the new prompt
  • Optimize test-coverage.md for token usage (206 → 54 lines)
  • Restore test-coverage routing entry and ### Analyze Test Coverage section in agent file
  • Fix frontmatter template: triggers:on:, network: defaults: truenetwork: defaults
Original prompt

This section details on the original issue you should resolve

<issue_title>[plan] Surface artifact-reading pattern in create-agentic-workflow prompt for test coverage scenarios</issue_title>
<issue_description>## Objective

Update the create-agentic-workflow prompt to guide the agent toward reading pre-computed coverage artifacts (via gh run download or the actions toolset) instead of re-running the full test suite inside the workflow.

Context

From Agent Persona Exploration discussion github/gh-aw#18250, QA-1 (Test Coverage PR Analyzer) scored 4.2/5.0. The agent doesn't proactively distinguish between "read coverage from existing CI artifacts" vs "run coverage yourself", leading to suggestions of slow, resource-intensive test reruns.

Approach

  1. Locate the create-agentic-workflow prompt (likely in skills/ or .github/workflows/)
  2. Add guidance that distinguishes two patterns:
    • Preferred: Fetch pre-computed coverage artifacts from a previous CI run using gh run download or the actions toolset
    • Fallback: Run tests to compute fresh coverage (only when no prior artifacts exist)
  3. Add an example snippet showing gh run download --name coverage-report pattern
  4. Optionally add a reference workflow for test coverage PR analysis

Files to Modify

  • Update: the create-agentic-workflow prompt/skill file with artifact-reading guidance
  • Optionally create: .github/workflows/coverage-pr-analyzer.md as a reference workflow

Acceptance Criteria

  • Prompt guidance distinguishes artifact-reading vs test-running approaches
  • Agent recommends gh run download or actions toolset for coverage artifacts by default
  • Guidance is clear enough that agent scores ≥ 4.5 on test coverage scenarios

Generated by Plan Command for issue #discussion #18250

  • expires on Feb 27, 2026, 4:46 AM UTC

Comments on the Issue (you are @copilot in this section)

@pelikhan @copilot create a prompt at .github/aw/... and link it from .github/agents/agentic-workflows.agent.md

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Adds .github/aw/test-coverage.md with guidance that distinguishes:
- Preferred: fetch pre-computed coverage artifacts via gh run download or actions toolset
- Fallback: run tests when no prior artifacts exist

Links the new prompt from .github/agents/agentic-workflows.agent.md.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Update create-agentic-workflow prompt for coverage artifacts feat: add test-coverage prompt with artifact-reading guidance Feb 25, 2026
@pelikhan
pelikhan marked this pull request as ready for review February 25, 2026 05:40
Copilot AI review requested due to automatic review settings February 25, 2026 05:40
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot optimize for token usage and still provide clear guidance for the authoring agent

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds dedicated guidance for creating agentic workflows that analyze test coverage, with an explicit preference for reading existing CI coverage artifacts (via gh run download or GitHub actions toolset) instead of re-running tests.

Changes:

  • Introduces a new .github/aw/test-coverage.md prompt covering artifact-first strategy, fallback behavior, formats, and an end-to-end example.
  • Updates the dispatcher agent documentation to route “test coverage” requests to the new prompt and links it in Available Prompts.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
.github/aw/test-coverage.md New test coverage workflow guidance emphasizing artifact-first coverage analysis.
.github/agents/agentic-workflows.agent.md Adds routing + documentation entry pointing to the new test-coverage prompt.
Comments suppressed due to low confidence (1)

.github/aw/test-coverage.md:152

  • The complete example workflow repeats the same frontmatter issues as the earlier template (triggers: instead of on: and network: defaults: true). Since this is presented as a copy/paste-ready workflow, it’s likely to produce non-working workflows unless corrected to the documented schema.
---
engine: copilot
triggers:
  pull_request:
    types: [opened, synchronize]
permissions:
  pull-requests: write
  actions: read
network:
  defaults: true
tools:

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/aw/test-coverage.md Outdated

```bash
# Find the latest successful CI run for this commit/branch
gh run list --branch "$BRANCH" --workflow ci.yml --status success --limit 1 --json databaseId -q '.[0].databaseId'

Copilot AI Feb 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the gh run download example, RUN_ID is used but never set. This makes the snippet non-runnable as written. Consider capturing the output of the preceding gh run list ... -q '.[0].databaseId' into RUN_ID (or inline it) so the download step is self-contained.

Suggested change
gh run list --branch "$BRANCH" --workflow ci.yml --status success --limit 1 --json databaseId -q '.[0].databaseId'
RUN_ID=$(gh run list --branch "$BRANCH" --workflow ci.yml --status success --limit 1 --json databaseId -q '.[0].databaseId')

Copilot uses AI. Check for mistakes.
Comment thread .github/aw/test-coverage.md Outdated

```yaml
engine: copilot
triggers:

Copilot AI Feb 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The frontmatter template uses triggers: for the GitHub Actions event configuration, but this repo’s agentic workflow format uses on: (required). See .github/aw/github-agentic-workflows.md:80 and the example at lines 12-23. Using triggers: in guidance will lead to invalid workflows.

This issue also appears on line 142 of the same file.

Suggested change
triggers:
on:

Copilot uses AI. Check for mistakes.
Comment thread .github/aw/test-coverage.md Outdated
Comment on lines +114 to +116
defaults: true
# Add language ecosystem if running tests as fallback:
# egosystems: [node] # or python, go, etc.

Copilot AI Feb 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The network: examples appear to use a non-existent shape (defaults: true) and reference egosystems, which looks like a typo and is not part of the documented schema. In this repo, network should be either network: defaults or network: { allowed: [defaults, <ecosystem>...] } (e.g. add node/python/go only when falling back to running tests). As written, the template may compile to invalid config or mislead users.

Suggested change
defaults: true
# Add language ecosystem if running tests as fallback:
# egosystems: [node] # or python, go, etc.
allowed: [defaults] # base; extend if running tests as fallback
# Add language ecosystems if running tests as fallback:
# allowed: [defaults, node] # or python, go, etc.

Copilot uses AI. Check for mistakes.
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Feb 25, 2026

Copy link
Copy Markdown
Contributor Author

@copilot optimize for token usage and still provide clear guidance for the authoring agent

Done in 7271a4b — reduced from 206 to 54 lines (~74% cut) by:

  • Collapsing the two separate pattern sections into a single copy-paste "Coverage Data Strategy" block
  • Merging the artifact detection steps into the strategy block
  • Removing the redundant full example workflow and anti-patterns section
  • Keeping the frontmatter template and a compact fallback table

All key guidance (preferred artifact path, fallback condition, required permissions/toolsets) is preserved.

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot bring back reference to test coverage and link to file

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Feb 25, 2026

Copy link
Copy Markdown
Contributor Author

@copilot bring back reference to test coverage and link to file

Restored in da6ed5c — added back the routing bullet (- **Analyzing test coverage**: Routes to \test-coverage` prompt) and the ### Analyze Test Coveragesection with prompt file link and use cases. Also fixed two schema issues in the frontmatter template while I was there:triggers:on:andnetwork: defaults: truenetwork: defaults`.

@pelikhan
pelikhan merged commit 0399c09 into main Feb 25, 2026
29 checks passed
@pelikhan
pelikhan deleted the copilot/update-create-agentic-workflow branch February 25, 2026 06:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[plan] Surface artifact-reading pattern in create-agentic-workflow prompt for test coverage scenarios

3 participants