fix(smoke-claude): raise max-turns to 2 to eliminate 96% failure rate#5162
Conversation
- smoke-claude.md: change max-turns from 1 to 2 (matches prompt body intent) - Recompile: --max-turns 2 and maxRuns 2 propagated to lock file - smoke-claude-workflow.test.ts: update assertions to expect max-turns 2 and fix pre-existing SHA regex to match v-tag format for gh-aw-actions/setup
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (4 files)
Coverage comparison generated by |
There was a problem hiding this comment.
Pull request overview
This PR updates the smoke-claude agentic workflow configuration to allow two Claude turns so the workflow can complete a full agent cycle without hitting the api-proxy invocation limit.
Changes:
- Bump
smoke-claudefrontmattermax-turnsfrom 1 → 2. - Recompile the locked workflow to propagate
--max-turns 2andapiProxy.maxRuns: 2. - Update CI assertions to match the new
max-turnsand the newly emittedsetupreference format.
Show a summary per file
| File | Description |
|---|---|
scripts/ci/smoke-claude-workflow.test.ts |
Updates test expectations for max-turns: 2 / --max-turns 2 and the setup reference format. |
.github/workflows/smoke-claude.md |
Raises workflow max-turns to 2 in the source workflow. |
.github/workflows/smoke-claude.lock.yml |
Regenerated lock workflow reflecting maxRuns: 2, --max-turns 2, and other compiler output changes. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 3/3 changed files
- Comments generated: 6
| # gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"c482acab5279c38a7cbe07846e3813673b3ef559c2a0e744b97de8e88ef14896","body_hash":"6e05820005e43b82d8112bc60ced8e13336596ae671ecac69e6c5ac691485b71","compiler_version":"v0.79.6","agent_id":"claude","agent_model":"claude-haiku-4-5","engine_versions":{"claude":"2.1.168"}} | ||
| # gh-aw-manifest: {"version":1,"secrets":["ANTHROPIC_API_KEY","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/checkout","sha":"df4cb1c069e1874edd31b4311f1884172cec0e10","version":"v6.0.3"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e","version":"v6.4.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"5c2fe865bb4dc46e1450f6ee0d0541d759aea73a","version":"v0.79.6"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.2","digest":"sha256:f88e5b17b6b7a600117bc121114d6ce2155c88c983c0c939c5df884f730fa1d6","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.2@sha256:f88e5b17b6b7a600117bc121114d6ce2155c88c983c0c939c5df884f730fa1d6"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.2","digest":"sha256:ee39841d980878ebbb87592903b06d31a1af500c71525c9616f7e8e2a27041a4","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.2@sha256:ee39841d980878ebbb87592903b06d31a1af500c71525c9616f7e8e2a27041a4"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.2","digest":"sha256:2e3a717e5f19a654cd9a2263beb52012b56bcb68562ec5ae2e42f9d156b49591","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.2@sha256:2e3a717e5f19a654cd9a2263beb52012b56bcb68562ec5ae2e42f9d156b49591"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.3.1","digest":"sha256:287fad0236959f3b3d9936ea1ef8d5b4f135ef2a5f5789713495cbbef191e60c","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.3.1@sha256:287fad0236959f3b3d9936ea1ef8d5b4f135ef2a5f5789713495cbbef191e60c"}]} | ||
| # gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"a08c320bc1b492fd4827ebbdc1fff37ca664404dc3f30ca87b38733896486989","body_hash":"6e05820005e43b82d8112bc60ced8e13336596ae671ecac69e6c5ac691485b71","compiler_version":"v0.79.8","agent_id":"claude","agent_model":"claude-haiku-4-5","engine_versions":{"claude":"2.1.168"}} | ||
| # gh-aw-manifest: {"version":1,"secrets":["ANTHROPIC_API_KEY","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/checkout","sha":"df4cb1c069e1874edd31b4311f1884172cec0e10","version":"v6.0.3"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e","version":"v6.4.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"v0.79.8","version":"v0.79.8"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.2","digest":"sha256:f88e5b17b6b7a600117bc121114d6ce2155c88c983c0c939c5df884f730fa1d6","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.2@sha256:f88e5b17b6b7a600117bc121114d6ce2155c88c983c0c939c5df884f730fa1d6"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.2","digest":"sha256:ee39841d980878ebbb87592903b06d31a1af500c71525c9616f7e8e2a27041a4","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.2@sha256:ee39841d980878ebbb87592903b06d31a1af500c71525c9616f7e8e2a27041a4"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.2","digest":"sha256:2e3a717e5f19a654cd9a2263beb52012b56bcb68562ec5ae2e42f9d156b49591","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.2@sha256:2e3a717e5f19a654cd9a2263beb52012b56bcb68562ec5ae2e42f9d156b49591"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.3.1","digest":"sha256:287fad0236959f3b3d9936ea1ef8d5b4f135ef2a5f5789713495cbbef191e60c","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.3.1@sha256:287fad0236959f3b3d9936ea1ef8d5b4f135ef2a5f5789713495cbbef191e60c"}]} |
| - name: Setup Scripts | ||
| id: setup | ||
| uses: github/gh-aw-actions/setup@5c2fe865bb4dc46e1450f6ee0d0541d759aea73a # v0.79.6 | ||
| uses: github/gh-aw-actions/setup@v0.79.8 | ||
| with: |
| - name: Setup Scripts | ||
| id: setup | ||
| uses: github/gh-aw-actions/setup@5c2fe865bb4dc46e1450f6ee0d0541d759aea73a # v0.79.6 | ||
| uses: github/gh-aw-actions/setup@v0.79.8 | ||
| with: |
| - name: Setup Scripts | ||
| id: setup | ||
| uses: github/gh-aw-actions/setup@5c2fe865bb4dc46e1450f6ee0d0541d759aea73a # v0.79.6 | ||
| uses: github/gh-aw-actions/setup@v0.79.8 | ||
| with: |
| - name: Setup Scripts | ||
| id: setup | ||
| uses: github/gh-aw-actions/setup@5c2fe865bb4dc46e1450f6ee0d0541d759aea73a # v0.79.6 | ||
| uses: github/gh-aw-actions/setup@v0.79.8 | ||
| with: |
| expect(lock).not.toContain('<< ENVEOF'); | ||
| expect(lock).toContain('Report turn usage'); | ||
| expect(lock).toContain('target: 1'); | ||
| expect(lock).toMatch(/github\/gh-aw-actions\/setup@[a-f0-9]{40} # v\d+\.\d+\.\d+/); | ||
| expect(lock).toMatch(/github\/gh-aw-actions\/setup@v\d+\.\d+\.\d+/); | ||
| expect(lock).not.toContain('mcp__playwright__browser_navigate'); |
|
@copilot address review feedback |
Done. All four |
|
❌ Smoke Claude failed No user request was provided in this turn — the message contained only system reminders listing available skills and project context, with no task to act on. No GitHub action required. |
|
❌ Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed... |
|
❌ Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed... |
|
🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅ |
|
📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅ |
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤 |
|
🔌 Smoke Services — All services reachable! ✅ |
|
❌ Contribution Check failed. Please review the logs for details. |
|
✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓 |
|
Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded. |
|
🚀 Security Guard has started processing this pull request |
|
✅ Smoke Gemini completed. All facets verified. 💎 |
|
✅ Build Test Suite completed successfully! |
|
✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟 |
|
Smoke Test: Copilot BYOK Direct Mode — ✅ PASS
Mode: Direct BYOK (sidecar pattern). Real credentials held by api-proxy, placeholder injected into agent.
|
|
PR: fix(smoke-claude): raise max-turns to 2 to eliminate 96% failure rate Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "registry.npmjs.org"See Network Configuration for more information.
|
🔬 Smoke Test ResultsPR: fix(smoke-claude): raise max-turns to 2 to eliminate 96% failure rate
Overall: FAIL — pre-computed step outputs were not injected (workflow template variables unresolved).
|
Smoke Test: API Proxy OpenTelemetry Tracing
All scenarios passed.
|
Chroot Version Comparison Results
Result: ❌ Not all tests passed. Python and Node.js versions differ between host and chroot environments. The
|
Smoke Test Results
Overall status: PASS Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
network:
allowed:
- defaults
- "localhost"See Network Configuration for more information.
|
Smoke Test Results — Services Connectivity
Overall: FAIL —
|
🔬 Smoke Test Results — Auth mode: PAT (COPILOT_GITHUB_TOKEN)
Overall: PASS cc
|
🏗️ Build Test Suite Results
Overall: 8/8 ecosystems passed — ✅ PASS
|
smoke-claudewas failing ~96% of runs becausemax-turns: 1caused the api-proxymaxRunsto be compiled as1, but Claude Code needs 2 LLM API calls even for a single agentic cycle (one for tool execution, one for the final response). Every run hit a 429Maximum LLM invocations exceeded (2 / 1)on the second call.Changes
smoke-claude.md:max-turns: 1→max-turns: 2— consistent with the prompt body which already statedmax-turns: 2as the hard capsmoke-claude.lock.yml: recompiled —--max-turns 2,maxRuns: 2(auto-derived frommax-turns),--build-local,--session-state-dirall reflectedsmoke-claude-workflow.test.ts: updatedmax-turnsassertions to2; fixed pre-existing regex that expected a 40-char commit SHA forgh-aw-actions/setupbut the compiler now emits a version tag (@v0.79.8)Not addressed (deferred)
The issue also recommended tightening
network.allowedto Anthropic domains only. Investigation showedanthropicis not a valid ecosystem identifier and the 55 domains are auto-injected by the compiler based on the Claude engine — they cannot be reduced via thenetwork:frontmatter block alone. Tracked as a separate follow-up.