Skip to content

[aw-failures] [aw-fix] P1: Agent Performance Analyzer — repo-memory exceeds 12 KB, agent idle-loops on push_repo_memory rejectio [Content truncated due to length] #48009

Description

@github-actions

Fix the repo-memory pruning path — a recoverable size-limit rejection is being turned into a hard run failure by an idle-loop → watchdog SIGTERM.

Executive summary

Do this: on a push_repo_memory size-exceeded rejection, prune/summarize the store and retry (or exit cleanly) instead of idle-probing the tool until the copilot-harness watchdog kills the process. This single defect converts an expected, recoverable condition into a full run failure.

Failure cluster

Cluster Sev Workflow Signature Representative run
B P1 Agent Performance Analyzer - Meta-Orchestrator push_repo_memory rejected: total memory 15 KB > 12 KB limit → agent idle → post-result watchdog SIGTERM §30158812091

Problem statement

The Execute GitHub Copilot CLI step in Agent Performance Analyzer - Meta-Orchestrator (.github/workflows/agent-performance-analyzer.lock.yml) fails the run. push_repo_memory is rejected because the repo-memory store exceeds its size cap; the agent then loops probing the tool, emits empty assistant messages, goes idle, and the harness watchdog terminates it.

Affected workflows and run IDs

  • Agent Performance Analyzer - Meta-Orchestrator — run §30158812091 (2026-07-25T12:56Z)

Probable root cause

Root cause is confirmed from the run logs:

  1. push_repo_memory returns: Total memory size (15 KB) exceeds the allowed limit of 12 KB (configured max-patch-size: 10 KB).
  2. ls/du of /tmp/gh-aw/repo-memory/default/ shows the metrics/ subdirectory alone at 44 KB on disk, plus shared-alerts.md (4.9 KB), workflow-health-latest.md (4.4 KB), agent-performance-latest.md (3.6 KB) — the store has grown past the cap.
  3. The agent does not prune; it re-probes push_repo_memory variants and produces empty assistant turns.
  4. [copilot-harness] attempt 1: post-result watchdog terminating idle process after 20244ms (SIGTERM)failureClass=permission_denied permissionDeniedCount=6AI credits budget exceeded — not retrying (non-retryable guard condition) → run exits code 1.

The real defect is the idle-loop → watchdog path: a size-exceeded rejection is an expected, recoverable outcome, but there is no prune-and-retry (or clean-exit) branch, so the agent stalls into a SIGTERM.

Proposed remediation

  1. Primary: In the workflow's memory-write logic, on a size-exceeded rejection, rotate/summarize metrics/ and truncate *-latest.md history to bring the store under the cap, then re-call push_repo_memory. If still over after one prune, exit the memory step cleanly (log a warning) rather than looping.
  2. Guardrail: Cap metrics/ growth at the source (rolling window / summarized aggregates) so the 44 KB accumulation cannot recur.
  3. Optional: If a larger store is genuinely required, raise max-patch-size for this workflow's repo-memory config — but pruning is preferred over unbounded growth.
  4. Harness hardening (track separately if needed): an idle agent after a tool rejection should surface the rejection as a terminal error, not stall until the watchdog fires and mislabels it permission_denied / AI credits budget exceeded.

Success criteria / verification

  • Agent Performance Analyzer completes without a push_repo_memory size-exceeded error, OR prunes and pushes successfully within the cap.
  • No post-result watchdog terminating idle process ... SIGTERM on this workflow.
  • Run conclusion success; no permission_denied / AI credits budget exceeded misclassification for this cause.
Other clusters observed this window (report-only, no issue)
  • Cluster A — startup_failure ×3 (Auto-Triage Issues, PR Triage Agent, Issue Monster), 12:29–12:33Z. Audit of §30158163884 shows the agent partially ran (5 turns, 11.4k tokens, error_count=0) yet concluded startup_failure; baseline cohort match to a success run 30148485921 with only posture_changed + run_unsuccessful deltas. Three concurrent scheduled workflows failing at startup with no per-step errors points to transient GitHub Actions platform flakiness, not a code defect — no actionable remediation. Watch for recurrence.
  • Cluster C — PR Sous Chef pre_activation failed, agent skipped (§30157572332). P2, benign activation-gate behavior; agent never ran.
Existing issue correlation

Reviewed 6 open agentic-workflows issues (#47525, #47753, #47659, #47135, #46647, #43499). None match the repo-memory size-limit signature or the startup_failure cluster. No fresh evidence that any existing issue is fixed/stale, so none were closed (gh read access unavailable this run; relied on prefetch state).

References:

Generated by 🔍 [aw] Failure Investigator (6h) · age00 · 129.7 AIC · ⌖ 34.3 AIC · ⊞ 5.2K ·

  • expires on Aug 1, 2026, 5:21 AM UTC-08:00

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions