Skip to content

[aw-failures] [aw] Copilot CLI exits 127 — node missing inside AWF chroot (chronic Daily News failure on main) #40074

@github-actions

Description

@github-actions

Problem statement

Fix the AWF chroot Node.js bootstrap: when chroot mode is enabled, the Copilot CLI engine cannot find node inside /host and the agent job exits 127 before a single turn runs. The entrypoint prints [entrypoint][ERROR] Copilot CLI requires Node.js, but 'node' is not available inside AWF chroot. and Command completed with exit code: 127. Distinct from #39946 (Copilot exits 1 with output and turns>0); here turns=0, tool_types=0, token_usage=0 — the engine never starts.

Affected workflows and run IDs

  1. Daily News§27751720901 (main, schedule, 2026-06-18 09:59 UTC) — engine copilot / claude-sonnet-4.6.
  2. Daily News§27760141601 (branch copilot/debug-daily-news-failure, workflow_dispatch) — same signature.

Chronic: Daily News has failed on main every scheduled run for 14+ consecutive days with no successful baseline — 27681588513, 27610721754, 27540688430, 27408472399, 27338873553, 27268191506, 27198178835, 27130454325, 27008097775, 26944476134, ... all failure.

Probable root cause

  1. The chroot entrypoint resolves node via GH_AW_NODE_BIN then command -v node; inside /host neither is populated, so the harness exits 127 with node runtime missing on this runner — check runtimes.node in workflow YAML. The host runner has Node, but the toolcache bin dirs are not on PATH inside the chroot for this workflow.
  2. audit-diff of two failing runs (§27751720901 vs §27681588513) shows zero drift: run1_token_usage=0, run2_token_usage=0, no firewall anomalies — confirming the engine never executes any LLM turn, consistent across the chronic streak.

Proposed remediation

  1. In chroot mode, ensure the host Node.js install (/opt/hostedtoolcache/node/**/bin, /home/runner/work/_tool/**, nvm path) is discovered and bind-mounted/added to PATH inside /host before launching copilot_harness.cjs, or set GH_AW_NODE_BIN to the resolved host node.
  2. Alternatively, pin runtimes.node in the Daily News workflow YAML so the chroot bootstrap installs/locates a known node path.
  3. Surface the exit-127 node missing entrypoint error as a classified failure class (engine_runtime_missing) so chronic streaks are flagged rather than reported as a bare exit 127.

Success criteria / verification

  1. A re-run of Daily News on main reaches at least one LLM turn (token_usage>0) and the agent job no longer exits 127.
  2. node is resolvable inside the AWF chroot for Copilot-engine workflows; the Copilot CLI requires Node.js ... not available inside AWF chroot entrypoint error no longer appears.
  3. Daily News produces at least one successful (or correctly-classified non-127) run on main within a subsequent 48h window.

Parent report: #39883. Analyzed runs: 27751720901, 27760141601. Comparator (drift): 27751720901 → 27681588513.
Related to #39883

Generated by 🔍 [aw] Failure Investigator (6h) ·

  • expires on Jun 25, 2026, 6:03 AM UTC-08:00

Resolved — Daily News agent step recovered.

As of §27799081331 (2026-06-19 01:06 UTC) the Daily News agent job runs to completion: audit reports 42 turns / 2,394,762 tokens and the agent job conclusion is success. audit-diff vs the prior failing run §27760141601 (the run cited in this issue) shows turns 0 → 42, token_usage 0 → 2.39M — the exit-127 "node missing inside AWF chroot / engine never starts" signature is gone, and this issue's success criteria (token_usage>0, ≥1 LLM turn, no exit 127) are met.

Daily News is still red overall, but for a new, unrelated cause: the push_repo_memory job rejects the orphan memory/daily-news branch's unsigned first commit under the verified signatures ruleset (GH013). That is tracked separately in #aw_memfix.

Closing as fixed. (Original chroot/node-bootstrap analysis preserved in this issue's edit history.)

Generated by 🔍 [aw] Failure Investigator (6h) · 205.4 AIC · ⌖ 12.6 AIC · ⊞ 4.9K ·

Metadata

Metadata

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions