Skip to content

Add visual regression reference prompt (.github/aw/visual-regression.md) - #18272

Merged
pelikhan merged 5 commits into
mainfrom
copilot/add-visual-regression-workflow
Feb 25, 2026
Merged

Add visual regression reference prompt (.github/aw/visual-regression.md)#18272
pelikhan merged 5 commits into
mainfrom
copilot/add-visual-regression-workflow

Conversation

Copilot AI commented Feb 25, 2026

Copy link
Copy Markdown
Contributor
  • Create .github/aw/visual-regression.md as a reference prompt demonstrating visual regression testing with playwright + cache-memory
  • Rewrite as a proper .github/aw/ agent prompt: simple name:/description: frontmatter, example workflow shown as a code block
  • Link visual-regression.md in .github/aw/create-agentic-workflow.md at the playwright browser automation hint
  • Simplify and condense instructions in visual-regression.md (144 → 59 lines)
Original prompt

This section details on the original issue you should resolve

<issue_title>[plan] Add visual regression reference workflow using playwright + cache-memory</issue_title>
<issue_description>## Objective

Add a reference workflow that demonstrates visual regression testing using playwright + cache-memory for baseline screenshot storage. This addresses the lowest-scoring scenario (FE-1, 3.8/5.0) from the Agent Persona Exploration discussion github/gh-aw#18250.

Context

The agent correctly suggests playwright for screenshots but leaves baseline comparison storage unaddressed. Without a reference workflow showing how to store "before" screenshots using cache-memory or artifacts, visual regression workflows are incomplete.

Approach

  1. Create .github/workflows/visual-regression.md as a reference workflow that:
    • Triggers on pull_request
    • Uses playwright for capturing screenshots
    • Uses cache-memory to persist baseline screenshots between runs
    • Compares current screenshots against stored baselines
    • Uses safe-outputs to post a comment with the diff report
  2. Follow the pattern of existing reference workflows like auto-triage-issues.md
  3. Ensure filesystem-safe timestamp formats (no colons) in cache-memory filenames

Files to Create/Modify

  • Create: .github/workflows/visual-regression.md (and recompile to generate .lock.yml)

Acceptance Criteria

  • Reference workflow exists and compiles successfully
  • Workflow demonstrates baseline storage with cache-memory
  • Workflow posts visual diff report via safe-outputs
  • Workflow follows security best practices (minimal permissions, safe-outputs for writes)
  • The agentic-workflows agent can use this as a reference when asked about visual regression

Generated by Plan Command for issue #discussion #18250

  • expires on Feb 27, 2026, 4:46 AM UTC

Comments on the Issue (you are @copilot in this section)

@pelikhan Create .github/aw/visual-regression.md

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Add visual regression reference workflow using playwright and cache-memory Add visual regression reference workflow (.github/aw/visual-regression.md) Feb 25, 2026
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot turn into a prompt

…rence

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Feb 25, 2026

Copy link
Copy Markdown
Contributor Author

@copilot turn into a prompt

Done in d96ebeb. The file is now a proper .github/aw/ prompt with simple name:/description: frontmatter — no workflow-specific YAML (on:, engine:, tools:, etc.). The body contains the complete example workflow as a code block (frontmatter + concise agent prompt) plus a brief "Key Design Decisions" section explaining the pattern choices.

Copilot AI changed the title Add visual regression reference workflow (.github/aw/visual-regression.md) Add visual regression reference prompt (.github/aw/visual-regression.md) Feb 25, 2026
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot link this file in the agent prompt

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Feb 25, 2026

Copy link
Copy Markdown
Contributor Author

@copilot link this file in the agent prompt

Done in abc03a6. Added a reference to .github/aw/visual-regression.md in create-agentic-workflow.md at the playwright browser automation hint (line 211), so the agent is directed to that reference whenever a visual regression testing task is detected.

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot make the instructions simpler , more condensed

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Feb 25, 2026

Copy link
Copy Markdown
Contributor Author

@copilot make the instructions simpler , more condensed

Done in 7c7bcf8. The file is now 59 lines (down from 144) — the agent prompt body is reduced to four short paragraphs, verbose step-by-step sections are removed, and the "Key Design Decisions" list is trimmed to the five most important points.

@pelikhan
pelikhan marked this pull request as ready for review February 25, 2026 06:23
Copilot AI review requested due to automatic review settings February 25, 2026 06:23
@pelikhan
pelikhan merged commit e324355 into main Feb 25, 2026
@pelikhan
pelikhan deleted the copilot/add-visual-regression-workflow branch February 25, 2026 06:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new reference prompt for visual regression testing workflows, and links it from the workflow-authoring guidance so agents can reuse the pattern when asked about screenshot comparison across PRs.

Changes:

  • Added .github/aw/visual-regression.md reference prompt describing a Playwright + cache-memory baseline workflow pattern.
  • Linked the new reference from .github/aw/create-agentic-workflow.md at the browser automation hint.
  • Extended the dispatcher agent documentation to include a “test coverage” routing section.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
.github/aw/visual-regression.md Introduces a reference prompt + example frontmatter for screenshot baselines using cache-memory.
.github/aw/create-agentic-workflow.md Adds a pointer to the new visual regression reference prompt for browser-automation use cases.
.github/agents/agentic-workflows.agent.md Documents an additional routing option for test coverage analysis.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- "date *"
safe-outputs:
add-comment:
max: 1

Copilot AI Feb 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example workflow text says to use the noop safe-output when nothing changes, but safe-outputs: only configures add-comment. Add noop: to the safe-outputs config (or remove the instruction to use noop) so the workflow doesn’t fail with “safe output not enabled”.

Suggested change
max: 1
max: 1
noop: {}

Copilot uses AI. Check for mistakes.
Comment on lines +122 to +131
### Analyze Test Coverage
**Load when**: The workflow reads, analyzes, or reports test coverage — whether triggered by a PR, a schedule, or a slash command. Always consult this prompt before designing the coverage data strategy.

**Prompt file**: https://github.com/github/gh-aw/blob/main/.github/aw/test-coverage.md

**Use cases**:
- "Create a workflow that comments coverage on PRs"
- "Analyze coverage trends over time"
- "Add a coverage gate that blocks PRs below a threshold"

Copilot AI Feb 25, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This agent now routes “Analyzing test coverage” to .github/aw/test-coverage.md, but that prompt file doesn’t exist in the repository. Either add the missing .github/aw/test-coverage.md prompt (and ensure it’s linked consistently elsewhere), or remove/adjust this route to point at an existing prompt.

Suggested change
### Analyze Test Coverage
**Load when**: The workflow reads, analyzes, or reports test coverage — whether triggered by a PR, a schedule, or a slash command. Always consult this prompt before designing the coverage data strategy.
**Prompt file**: https://github.com/github/gh-aw/blob/main/.github/aw/test-coverage.md
**Use cases**:
- "Create a workflow that comments coverage on PRs"
- "Analyze coverage trends over time"
- "Add a coverage gate that blocks PRs below a threshold"

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[plan] Add visual regression reference workflow using playwright + cache-memory

3 participants