fix(smoke-claude): raise turn budget to 8 and fix add_comment usage#5328
Conversation
The Smoke Claude job was failing with "403 Maximum LLM invocations
exceeded (5 / 5)". The agent burned one of only five api-proxy runs on
an empty-args `add_comment` schema probe (`safeoutputs add_comment .`
with JSON piped via stdin yields `args: {}`, rejected by MCP), and the
cumulative cap left no room to retry.
Two-pronged fix:
- Raise `max-turns` 5 -> 8 for headroom (expands to --max-turns,
GH_AW_MAX_TURNS, and apiProxy maxRuns).
- Add explicit prompt guidance to pass `add_comment` arguments inline as
a single JSON object and never pipe JSON via stdin or pass `.`.
Recompiled the lock and re-ran the smoke post-processing script. Updated
smoke-claude-workflow.test.ts assertions accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (1 files)
Coverage comparison generated by |
There was a problem hiding this comment.
Pull request overview
This PR adjusts the Smoke Claude workflow configuration to reduce CI failures caused by exhausting the LLM invocation/turn budget before completing required safeoutputs actions.
Changes:
- Increased the configured turn budget from 5 → 8 in the source workflow and corresponding compiled lock workflow.
- Added prompt guidance to avoid empty-args
safeoutputs add_comment“schema probe” invocations by passing arguments inline. - Updated CI tests to assert the new turn budget and the presence of the new prompt guidance.
Show a summary per file
| File | Description |
|---|---|
scripts/ci/smoke-claude-workflow.test.ts |
Updates assertions for max-turns: 8 / --max-turns 8 and checks the new inline-JSON guidance text. |
.github/workflows/smoke-claude.md |
Raises max-turns and adds guidance for correct safeoutputs add_comment invocation. |
.github/workflows/smoke-claude.lock.yml |
Regenerated lock workflow reflecting maxRuns: 8, --max-turns 8, and GH_AW_MAX_TURNS: 8. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 3/3 changed files
- Comments generated: 2
| The JSON contains: `result` (PASS/FAIL), `api_status`, `gh_check`, `file_status`, `event`, `pr_number`. | ||
|
|
||
| - If `event` is `pull_request`: call `add_comment` with `item_number` set to `pr_number` and a body listing each check result plus the overall `result`; then call `add_labels` with `["smoke-claude"]` only if `result` is `PASS`. | ||
| - Pass arguments inline as a single JSON object, e.g. `safeoutputs add_comment '{"item_number": <pr_number>, "body": "<markdown>"}'`. Do NOT pipe JSON via stdin and do NOT pass `.` (or any placeholder) as the argument — that sends empty arguments and the call is rejected as a schema probe, wasting a turn. |
|
@copilot address review feedback |
|
✅ Copilot review passed with no inline comments. @lpcox Add the |
|
✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓 |
|
🚀 Security Guard has started processing this pull request |
|
✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓 |
|
🔌 Smoke Services — All services reachable! ✅ |
|
✅ Smoke Claude passed |
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤 |
|
✅ Smoke Gemini completed. All facets verified. 💎 Smoke test completed with FAIL status. Connectivity issues and missing MCP tools reported. |
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅ |
|
Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded. |
|
✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓 |
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟 |
|
🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅ |
|
✅ Build Test Suite completed successfully! |
🤖 Smoke Test ResultsPR: fix(smoke-claude): raise turn budget to 8 and fix add_comment usage — @lpcox
Overall: FAIL — pre-step template variables were not substituted; tests 2 & 3 unverifiable.
|
Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) Overall: PASS
|
Smoke Test: Claude Engine Validation
Overall result: PASS
|
Smoke Test: Copilot BYOK (Direct Mode) ✅ PASSTests:
Status: Running in direct BYOK mode ( cc @lpcox
|
|
@lpcox
|
|
Merged PRs reviewed: ✅ GitHub page title contains "GitHub" Overall status: PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "registry.npmjs.org"See Network Configuration for more information.
|
Chroot Version Comparison Results
Overall: ❌ Not all tests passed — Python and Node.js versions differ between host and chroot environments.
|
Smoke Test: API Proxy OpenTelemetry Tracing
All 5 scenarios pass. OTEL integration is functioning correctly.
|
Smoke Test — Copilot PAT Auth
PR: "fix(smoke-claude): raise turn budget to 8 and fix add_comment usage"
|
Smoke Test Results — FAIL ❌
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS
|
Smoke Test: Gemini Engine Validation
Overall status: FAIL Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "localhost"See Network Configuration for more information.
|
Problem
The Smoke Claude CI job has been failing with:
The workflow was capped at
max-turns: 5, which gh-aw expands into three coupled limits:--max-turns 5(Claude harness),GH_AW_MAX_TURNS: 5(env), and the api-proxymaxRuns: 5hard cap. The agent wasted one of those five invocations on an empty-argsadd_commentschema probe — it ransafeoutputs add_comment .with the JSON piped via stdin, which yieldsargs: {}and is rejected by the MCP server (-32602 Empty arguments are not allowed). Combined with other minor waste, the agent hit the cumulative cap before ever landing a successfuladd_comment. Harness retries can't recover becausemaxRunsis cumulative per run.Fix
Two-pronged:
max-turns5 → 8.add_commentarguments inline as a single JSON object (e.g.safeoutputs add_comment '{"item_number": <pr_number>, "body": "<markdown>"}') and never pipe JSON via stdin or pass./a placeholder (which sends empty args and wastes a turn).Changes
.github/workflows/smoke-claude.md—max-turns: 8+ new prompt bullet..github/workflows/smoke-claude.lock.yml— recompiled withgh aw compile(v0.80.6) and re-run throughscripts/ci/postprocess-smoke-workflows.ts. Net change:maxRuns 5→8,--max-turns 5→8,GH_AW_MAX_TURNS 5→8. Prompt body is runtime-imported from the.md, so the new guidance takes effect via the.mdedit.scripts/ci/smoke-claude-workflow.test.ts— updated assertions tomax-turns: 8/--max-turns 8and added an assertion for the new guidance.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com