Executive Summary
Overall gh-aw health for the last 24h is stable but degraded at the edges, with a 1.95% AI-call error rate (309 of 15,812 gen_ai spans errored). The failures are chronic (spread from ~20:19 to ~22:39 UTC, not a single burst) and split into three real problem classes: (1) a handful of model aliases that fail 100% of the time, (2) long-duration Copilot model timeouts (single error spans up to ~13 min), and (3) a large attribution/instrumentation gap — 97% of erroring AI spans cannot be tied back to a workflow. The errors and logs datasets are empty for the window, so reliability triage currently depends entirely on the spans dataset.
Telemetry source: Sentry spans dataset (org github, project gh-aw), last 24h, via list_events (this MCP build lacks search_events / get_trace_details). Emit-side naming cross-checked against the current run's /tmp/gh-aw/otel.jsonl mirror.
Top Reliability Findings
| Priority |
Workflow |
Problem |
Evidence |
Next Action |
| P1 |
(unattributed) |
Model alias kiwi fails 100% (24/24), immediate ~210-224ms rejections |
span.op:gen_ai gen_ai.request.model:kiwi span.status:error → 24/24 error; trace 9944168b..., 7b75bc02... |
Treat kiwi as an invalid/unresolved model alias; fix or remove it in the owning workflow config |
| P1 |
(unattributed) |
Model alias small fails 100% (14/14), ~256-262ms rejections |
gen_ai.request.model:small span.status:error → 14/14 error; trace 9988caa2..., 3c48d555... |
Same as above — resolve small to a valid model id |
| P1 |
(unattributed) |
Literal unexpanded expression ${{ experiments.subagent_model }} sent as a model name, fails 100% (4/4) |
grouped count: model = ${{ experiments.subagent_model }}, 4/4 error |
Templating/substitution bug — the expression is not being expanded before the model call |
| P2 |
PR Sous Chef |
Long Copilot model timeouts / hangs on copilot/gpt-5.4; retry ladder of error spans up to 407s |
trace da8567c364d2d5b38cba5dbe15b11bb6: error spans 407011ms, 285749ms, 61012ms, 35720ms, 7407ms interleaved with ok spans |
Add/verify a per-call timeout + capped retry for the Copilot engine; alert on gen_ai spans > N min |
| P4 |
(unattributed) |
Truncation / runaway-token detection is blind |
gen_ai.response.finish_reasons:length → no results; field unpopulated on sampled error spans |
Inconclusive, not "no truncation" — see instrumentation gap below |
| P5 |
all |
97% of erroring AI spans have no workflow attribution |
grouped: gh-aw.workflow.name = null for 299/309 error spans (only Delight=5, PR Sous Chef=5 attributed) |
Propagate gh-aw.workflow.name onto gen_ai spans (esp. Copilot-engine auto-instrumented spans) |
Representative Traces
View representative traces
Recommendations
- Fix the 100%-failure model aliases first (smallest, highest ROI).
kiwi (24/24), small (14/14), and the unexpanded ${{ experiments.subagent_model }} (4/4) are guaranteed failures — 42 wasted calls/day. Resolve these to valid model ids in the owning workflow(s) and fix the expression-expansion path that lets a raw ${{ ... }} string reach a model call.
- Propagate
gh-aw.workflow.name (and gen_ai.request.model) onto gen_ai spans. 299/309 error spans and 10,911/15,812 total gen_ai spans lack these attributes, so failures can't be attributed. This is the single biggest triage blocker; without it findings 1 and the Copilot timeouts can't be pinned to a workflow.
- Bound Copilot-engine model calls. Add/verify a per-request timeout and capped retry so
copilot/gpt-5.4 calls can't hang for 6-13 minutes, and add an alert on gen_ai spans exceeding a duration threshold.
- Capture
span.status_message on failed gen_ai spans so error class (429/5xx/invalid-model/timeout) is visible without opening each trace.
Notes
View notes
- Inconclusive runtime outcome + confirmed instrumentation gap on core fields.
span.status_message, gen_ai.response.finish_reasons, and (on 97% of error spans) gh-aw.workflow.name are missing/empty. Failures above are confirmed via span.status:error, but the cause class and truncation state are not directly observable from traces.
- Emit-side cross-check (per
actions/setup/js/send_otlp_span.cjs semantics, verified against this run's /tmp/gh-aw/otel.jsonl): workflow identity is emitted as gh-aw.workflow.name, runtime version as resource service.version (2.1.216; not a release attribute — Sentry release mapping is backend-dependent), and gen_ai.response.finish_reasons is emitted only when runtime metrics include stopReason — consistent with the empty finish_reasons:length result. The null-workflow.name gen_ai spans appear to be Copilot-engine auto-instrumented spans that do not inherit the gh-aw resource attributes; the fix is emit-side propagation, not a query change.
- Dataset availability:
spans = populated (15,812 gen_ai spans/24h). errors = empty. logs = empty. Empty error/log datasets are reported as an explicit observability finding, not silently skipped.
- Tooling limitation: this Sentry MCP build exposes
list_events only (no search_events, no get_trace_details); trace continuity was verified via list_events filtered by trace:<id>.
- Duplicate-span note (low signal): invalid-model errors appear as span pairs at identical timestamps (e.g. two
kiwi spans per call) — possibly double instrumentation (request + stream); noted, not investigated.
References:
Generated by 🚨 Daily Reliability Review · age00 154.9 AIC · ⌖ 42.9 AIC · ⊞ 5.3K · ◷
Executive Summary
Overall gh-aw health for the last 24h is stable but degraded at the edges, with a 1.95% AI-call error rate (309 of 15,812
gen_aispans errored). The failures are chronic (spread from ~20:19 to ~22:39 UTC, not a single burst) and split into three real problem classes: (1) a handful of model aliases that fail 100% of the time, (2) long-duration Copilot model timeouts (single error spans up to ~13 min), and (3) a large attribution/instrumentation gap — 97% of erroring AI spans cannot be tied back to a workflow. Theerrorsandlogsdatasets are empty for the window, so reliability triage currently depends entirely on thespansdataset.Telemetry source: Sentry
spansdataset (orggithub, projectgh-aw), last 24h, vialist_events(this MCP build lackssearch_events/get_trace_details). Emit-side naming cross-checked against the current run's/tmp/gh-aw/otel.jsonlmirror.Top Reliability Findings
kiwifails 100% (24/24), immediate ~210-224ms rejectionsspan.op:gen_ai gen_ai.request.model:kiwi span.status:error→ 24/24 error; trace9944168b...,7b75bc02...kiwias an invalid/unresolved model alias; fix or remove it in the owning workflow configsmallfails 100% (14/14), ~256-262ms rejectionsgen_ai.request.model:small span.status:error→ 14/14 error; trace9988caa2...,3c48d555...smallto a valid model id${{ experiments.subagent_model }}sent as a model name, fails 100% (4/4)${{ experiments.subagent_model }}, 4/4 errorcopilot/gpt-5.4; retry ladder of error spans up to 407sda8567c364d2d5b38cba5dbe15b11bb6: error spans 407011ms, 285749ms, 61012ms, 35720ms, 7407ms interleaved with ok spansgen_ai.response.finish_reasons:length→ no results; field unpopulated on sampled error spansgh-aw.workflow.name= null for 299/309 error spans (only Delight=5, PR Sous Chef=5 attributed)gh-aw.workflow.nameontogen_aispans (esp. Copilot-engine auto-instrumented spans)Representative Traces
View representative traces
da8567c364d2d5b38cba5dbe15b11bb6(PR Sous Chef,copilot/gpt-5.4): verified continuity — errorgen_aispans of 407011ms / 285749ms / 61012ms / 35720ms / 7407ms interleaved with ok spans (19845ms, 2746ms). Pattern = long hangs then progressively shorter retries.span.status_messagewas empty on all error spans (no failure reason captured).kiwi) — traces9944168b0ccf4aaadca68a8ec8b843cc,7b75bc02e4dbee6de129d017d5ee4125: allkiwispans error in ~210-224ms (fast provider rejection).small) — traces9988caa21c1e8e151618e7b8338077fb,3c48d5559095ca47ebee9241c196105a: allsmallspans error in ~256-262ms.Recommendations
kiwi(24/24),small(14/14), and the unexpanded${{ experiments.subagent_model }}(4/4) are guaranteed failures — 42 wasted calls/day. Resolve these to valid model ids in the owning workflow(s) and fix the expression-expansion path that lets a raw${{ ... }}string reach a model call.gh-aw.workflow.name(andgen_ai.request.model) ontogen_aispans. 299/309 error spans and 10,911/15,812 totalgen_aispans lack these attributes, so failures can't be attributed. This is the single biggest triage blocker; without it findings 1 and the Copilot timeouts can't be pinned to a workflow.copilot/gpt-5.4calls can't hang for 6-13 minutes, and add an alert ongen_aispans exceeding a duration threshold.span.status_messageon failedgen_aispans so error class (429/5xx/invalid-model/timeout) is visible without opening each trace.Notes
View notes
span.status_message,gen_ai.response.finish_reasons, and (on 97% of error spans)gh-aw.workflow.nameare missing/empty. Failures above are confirmed viaspan.status:error, but the cause class and truncation state are not directly observable from traces.actions/setup/js/send_otlp_span.cjssemantics, verified against this run's/tmp/gh-aw/otel.jsonl): workflow identity is emitted asgh-aw.workflow.name, runtime version as resourceservice.version(2.1.216; not areleaseattribute — Sentryreleasemapping is backend-dependent), andgen_ai.response.finish_reasonsis emitted only when runtime metrics includestopReason— consistent with the emptyfinish_reasons:lengthresult. The null-workflow.namegen_aispans appear to be Copilot-engine auto-instrumented spans that do not inherit the gh-aw resource attributes; the fix is emit-side propagation, not a query change.spans= populated (15,812 gen_ai spans/24h).errors= empty.logs= empty. Empty error/log datasets are reported as an explicit observability finding, not silently skipped.list_eventsonly (nosearch_events, noget_trace_details); trace continuity was verified vialist_eventsfiltered bytrace:<id>.kiwispans per call) — possibly double instrumentation (request + stream); noted, not investigated.References:
kiwi100% failure)