Skip to content

fix(smoke): target ADO agent-definitions repo for PR/git-write smokes#1045

Merged
jamesadevine merged 1 commit into
mainfrom
fix/smoke-pr-repos
Jun 15, 2026
Merged

fix(smoke): target ADO agent-definitions repo for PR/git-write smokes#1045
jamesadevine merged 1 commit into
mainfrom
fix/smoke-pr-repos

Conversation

@jamesadevine

Copy link
Copy Markdown
Collaborator

Problem

The daily safe-output smoke suite is registered as Azure DevOps pipelines in msazuresphere/AgentPlayground, but the pipeline YAML lives in GitHub (githubnext/ado-aw). For PR / git-write smokes, repository: "self" in the agent prompt resolves to $(Build.Repository.Name) at runtime, which is the GitHub repo. Stage 3 then calls the ADO Git REST APIs against a name that does not exist on the ADO side, so every PR-targeting smoke 404s before any variable-group / perma-PR prereq can help.

Fix

Update the 7 affected smoke fixtures so they explicitly target the AgentPlayground ADO repo agent-definitions:

  • add-pr-comment, reply-to-pr-comment, resolve-pr-thread, submit-pr-review, update-pr
  • create-branch, create-git-tag

For each fixture:

  • declare the ADO repo resource via repos: [agent-definitions=agent-definitions]
  • whitelist the alias in the per-tool allowed-repositories: list
  • change the prompt parameter from repository: "self" to repository: "agent-definitions"

Lock files are regenerated to include the agent-definitions repo resource and a matching - checkout: agent-definitions step.

Documentation

tests/safe-outputs/REGISTERED.md now has an ADO repo targeting section explaining the convention, plus per-row notes calling out which fixtures target agent-definitions.

Follow-up

create-pull-request.md is not included here. It needs a separate redesign that synthesises a working-tree commit inside the ADO repo (the current prompt assumes self's working tree contains an editable file). The README and table flag it as not yet exercised against AgentPlayground.

Why this is unblocking

The variable group ado-aw-daily-smoke and the perma-PR / perma-thread in agent-definitions are now meaningful: they target a repo that Stage 3 can actually reach. Without this PR, those prereqs would be wired against a repo Stage 3 cannot see.

Compiled with ado-aw v0.36.0 --force (inside the ado-aw source checkout).

The pipeline YAML for the daily safe-output smoke suite lives in
GitHub (githubnext/ado-aw), so repository: "self" in the
PR-targeting smoke prompts resolves to the GitHub source repo at
runtime. Stage 3 safe-output executors then call the ADO Git REST
APIs against a name that doesn't exist on the ADO side, 404-ing
every PR / git-write smoke before any prereq can help.

Fix the 7 affected fixtures:
  add-pr-comment, reply-to-pr-comment, resolve-pr-thread,
  submit-pr-review, update-pr, create-branch, create-git-tag

For each:
  - declare ADO repo resource via repos: [agent-definitions=agent-definitions]
  - whitelist the alias in the per-tool allowed-repositories: list
  - change prompt parameter from repository: "self"
    to repository: "agent-definitions"

Recompile lock files. Document the limitation + the new ADO-repo
targeting convention in tests/safe-outputs/REGISTERED.md.

create-pull-request remains untouched (needs a separate redesign
that synthesises a working-tree commit inside the ADO repo); the
README flag and table note explain the limitation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jamesadevine jamesadevine merged commit aae77f6 into main Jun 15, 2026
5 checks passed
@jamesadevine jamesadevine deleted the fix/smoke-pr-repos branch June 15, 2026 22:24
@github-actions

Copy link
Copy Markdown
Contributor

🔍 Rust PR Review

Summary: Fix is correct and consistently applied — this approach is sound.

Findings

✅ What Looks Good

  • Root-cause fix is accurate: repository: "self" resolving to $(Build.Repository.Name) (the GitHub repo) on an ADO token that can only see ADO repos is exactly the right diagnosis. Switching to an explicit ADO repo resource is the correct fix.
  • Consistency across all 7 fixtures: Every affected fixture gets the same three-part treatment — repos: declaration, allowed-repositories: allowlist, and prompt parameter change from "self" to "agent-definitions". No partial fixes.
  • Allowlist is tight: allowed-repositories: [agent-definitions] is correctly scoped to the single target repo rather than a permissive wildcard.
  • Path migration in lock files is thorough: All occurrences of $(Build.SourcesDirectory) are consistently updated to $(Build.SourcesDirectory)/$(Build.Repository.Name) to account for ADO's multi-checkout subdirectory layout — applies to working directories, safe-outputs server startup, runtime imports, threat analysis paths, and Stage 3 execute.
  • create-pull-request.md exclusion is clearly documented: The REGISTERED.md entry and the new "ADO repo targeting" section both call out why it can't be trivially updated, with a pointer to the follow-up redesign.
  • Version bumps are coherent: ado-aw 0.35.3 → 0.36.0, copilot CLI 1.0.60 → 1.0.62, AWF 0.25.65 → 0.27.3, MCPG v0.3.23 → v0.3.25 — all updated uniformly across all recompiled lock files with no version skew between fixtures.

⚠️ Suggestions

  • resolve-pr-thread setup section not visible in diff: The fixture's setup: block (which presumably creates/seeds the thread to be resolved) was not changed. If any bash in that block calls ADO Git REST APIs with an implicit repo reference, it would have the same self-vs-ADO-repo problem. Worth auditing the full setup section to confirm it uses explicit ADO API identifiers (project/repo) rather than relying on $(Build.Repository.Name).

  • Node.js task/version change is a buried side-effect: The recompile changed UseNode@1 (22.x) to NodeTool@0 (20.x) across all lock files. The PR description doesn't mention this. NodeTool@0 is the correct standard ADO task for standalone (non-1ES) pipelines, but the downgrade from 22.x to 20.x is a visible behaviour change. If any runtime script uses Node 22 features (.findLast(), etc.) this could silently regress. Confirm the version change is intentional as part of the 0.36.0 compiler defaults.

Generated by Rust PR Reviewer for issue #1045 · 206.9 AIC · ⌖ 12.6 AIC · ⊞ 33.1K ·

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant