[OTLP Validation] Run 26737661601 — OTLP Data Quality Validation Report

[github-actions]

A. Executive Summary

Overall status: WARN

The single setup span emitted for this workflow run (gh-aw.agent.setup) is correctly shaped, fully attributed, and being actively exported to Sentry. No export error files were found. However, six CONNECT attempts were made to Sentry's OTLP ingest over ~35 seconds — more than expected for a single successful export — indicating likely retry activity. Because all connections returned HTTP 200 (TCP tunnel established), the actual HTTPS response codes are not visible in firewall logs; we cannot confirm whether each POST succeeded or was rejected by Sentry.

Missing conclusion and agent spans are expected: the workflow agent job is still running and those spans are emitted only on job conclusion.

---

B. Trace Completeness

Item	Value
Validation window	2026-06-01 05:50:46 – 05:50:48 UTC (span times)
Expected service.name	gh-aw.otlp-data-quality-validator
Spans in JSONL mirror	1 (gh-aw.agent.setup)
Backend span count	Unknown — no MCP backend tool available
Duplicate spans (same traceId+spanId)	0
Unique trace IDs	1 (e9add4ab48412025ae82a3553dc67a61) ✅
Missing spans (expected at this stage)	conclusion, agent — not yet emitted
Export error count	0 (no otlp-export-errors.jsonl or .count file)

Confidence: High for local-mirror completeness (single span at setup stage); backend visibility unverified — no MCP query tool available.

---

C. Span Hierarchy Validation

Check	Result	Notes
Setup span has global parentSpanId	✅ Pass	7bc093fe6b27c124 (per spec §9.3)
Conclusion spans parent under setup	⏳ Pending	Not yet emitted
Agent spans parent under conclusion	⏳ Pending	Not yet emitted
Span naming pattern gh-aw.<job>.<op>	✅ Pass	gh-aw.agent.setup
Root span (no parent) in mirror	⚠️ Not present	Root activation span from a different job; not in this job's JSONL

Note: spec §9.3 confirms that setup spans share a global parent span ID. The absence of the root span in this job's mirror is expected — root spans are emitted by the activation job.

---

D. Attribute Contract Validation

Setup Span Required Attributes (spec §10.1)

Attribute	Present	Value
gh-aw.job.name	✅	agent
gh-aw.workflow.name	✅	OTLP Data Quality Validator
gh-aw.run.id	✅	26737661601
gh-aw.run.attempt	✅	1
gh-aw.run.actor	✅	mnkiefer
gh-aw.repository	✅	github/gh-aw
gh-aw.staged	✅	false

All 7 required setup attributes present. ✅

Conclusion Span Required Attributes (spec §10.2)

Not yet emitted — pending workflow completion. ⏳

Agent Span GenAI Attributes (spec §10.3)

Not yet emitted. Note: gen_ai.system: github_models and gh-aw.engine.id: copilot appear on the setup span as supplemental attributes — correct placement per convention.

Resource Attributes (spec §11.1)

Attribute	Present	Value
service.name	✅	gh-aw.otlp-data-quality-validator
service.version	✅	1.0.55
github.repository	✅	github/gh-aw
github.run_id	✅	26737661601
github.run_attempt	✅	1
github.actions.run_url	✅	https://github.com/github/gh-aw/actions/runs/26737661601

All 6 required resource attributes present. ✅

Instrumentation Scope

Field	Value	Status
scope.name	gh-aw	✅
scope.version	1.0.55	✅ matches service.version

---

E. Export and Fan-Out Health

Item	Status	Details
JSONL mirror write	✅ OK	/tmp/gh-aw/otel.jsonl, 1 line
Sentry OTLP export	⚠️ Probable — unverifiable	6 × CONNECT HTTP 200 to o205451.ingest.us.sentry.io:443
Other endpoints (Grafana, Datadog)	Not observed	No CONNECT tunnels to other OTLP hosts
Export error log	✅ Empty	No otlp-export-errors.jsonl present
Multi-endpoint fan-out	N/A	Only one endpoint active in this run

Sentry connection pattern: 6 CONNECT requests spanning 05:52:11–05:52:46 UTC (≈35 s gap). With maxRetries=2 and base delay 100 ms + exponential backoff, 3 attempts per export cycle take <1 s total. Six connections over 35 s suggests either: (a) multiple separate export invocations from retries or parallel jobs, or (b) the proxy tunnel is being re-established per-request. The absence of any export error entries is reassuring but doesn't exclude silent 4xx/5xx responses inside the TLS tunnel.

Note: The CONNECT-tunnel firewall logs show TCP-layer success only; HTTPS-level responses (e.g., 401 Unauthorized, 429 Too Many Requests) are not visible. Cross-stage reconciliation against the Sentry backend is not possible without a backend MCP tool.

---

F. Root-Cause Hypothesis

Primary observation: higher-than-expected CONNECT count to Sentry

Rank	Hypothesis	Evidence
1	Multiple export invocations (parallel agent sub-jobs or workflow steps each emitting a setup span separately)	6 connections over 35 s; timing gaps consistent with retries or parallel callers
2	Individual HTTPS requests each open a new CONNECT tunnel (proxy behavior)	Proxy rewrites Authorization → x-sentry-auth; each fetch call may open a new tunnel
3	Sentry returning 4xx inside tunnel, triggering retries	No error log entries, but error log silence is not definitive for in-tunnel failures
4	Network jitter causing retry of otherwise successful exports	No evidence; low probability given clean mirror

No confirmed data loss. No confirmed export failure.

---

G. Recommended Fixes (Prioritized)

1. Add HTTPS-level response code logging — The current export implementation logs failures to otlp-export-errors.jsonl only on caught exceptions or non-2xx status codes. If Sentry silently resets after the CONNECT tunnel is established, the error may go unlogged. Consider logging the raw HTTP status code for every Sentry POST attempt regardless of outcome.

2. Normalize CONNECT count per span emission — If 6 CONNECT tunnels are expected for a single-span run (e.g., 3 retries × 2 header attempts), document this in the implementation. If unexpected, add a per-run CONNECT counter to the conclusion span attributes (gh-aw.otlp.connect_count) to aid future validation.

3. Validate conclusion + agent spans post-run — Re-run this validator as a post-conclusion check or add a second validation step after the agent job completes to confirm conclusion span attributes (spec §10.2) and gh-aw.otlp.export_errors counter.

4. Fix workflow prompt mirror path — The task prompt references /tmp/gh-aw/agent/otel.jsonl; spec §7 (line 261) and send_otlp_span.cjs both use /tmp/gh-aw/otel.jsonl. Update the prompt to use the canonical path.

---

H. Validation Queries

# Span summary from JSONL mirror
jq -c '.resourceSpans[].scopeSpans[].spans[] | {name, traceId, spanId, parentSpanId, kind, status}' \
  /tmp/gh-aw/otel.jsonl

# Unique trace IDs (expect 1)
jq -r '.resourceSpans[].scopeSpans[].spans[].traceId' /tmp/gh-aw/otel.jsonl | sort -u | wc -l

# Required setup attributes
jq -c '.resourceSpans[].scopeSpans[].spans[] | select(.name | endswith(".setup")) |
  {name, attrs: [.attributes[]? | {(.key): .value}] | add}' /tmp/gh-aw/otel.jsonl

# Resource attributes
jq -c '.resourceSpans[].resource.attributes[] | {(.key): .value}' /tmp/gh-aw/otel.jsonl | sort -u

# Instrumentation scope
jq -c '.resourceSpans[].scopeSpans[].scope' /tmp/gh-aw/otel.jsonl

# Sentry CONNECT count from firewall audit log
jq -c 'select(.host | test("sentry")) | {timestamp, method, status}' \
  /tmp/gh-aw/sandbox/firewall/logs/audit.jsonl

# Export errors
cat /tmp/gh-aw/agent/otlp-export-errors.jsonl 2>/dev/null || echo "(none)"
cat /tmp/gh-aw/agent/otlp-export-errors.count 2>/dev/null || echo "0"

---

Validated against specs/otel-observability-spec.md v0.3.0 · Run ID: 26737661601 · gh-aw v1.0.55

Generated by 🧭 OTLP Data Quality Validator · sonnet46 1.4M · ◷

• expires on Jun 8, 2026, 5:57 AM UTC

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #36408

View newer issue

---

This action was performed automatically by the OTLP Data Quality Validator workflow.