fix(api-proxy): copy token-tracker-shared + otel modules into image (fixes AIC=0) by lpcox · Pull Request #5254 · github/gh-aw-firewall

lpcox · 2026-06-18T16:03:52Z

Root cause of AIC=0

The api-proxy Docker image copies source files individually by name (no bundler — see Dockerfile). PR #4780 ("extract shared token-tracker budget helpers", first released in v0.27.3) introduced token-tracker-shared.js, required by token-tracker-http.js and token-tracker-ws.js, but never added it to the Dockerfile COPY list.

In the running container:

require('./token-tracker') → token-tracker-http.js → require('./token-tracker-shared') throws MODULE_NOT_FOUND.
The graceful-degradation guard in proxy-request.js catches MODULE_NOT_FOUND and stubs trackTokenUsage to a no-op.
The container boots and proxies traffic normally — but no token-usage.jsonl is ever written, so AI-credit accounting reports 0 for every run since v0.27.3.
models.json still appears (written by model-discovery.js, which doesn't depend on the missing module), which is why the log dir looked healthy.

Reproduction (deterministic)

$ mv token-tracker-shared.js /tmp/ && node -e "require('./token-tracker')"
require threw: MODULE_NOT_FOUND - Cannot find module './token-tracker-shared'
=> proxy-request.js stubs trackTokenUsage to no-op (token tracking SILENTLY DISABLED)
model-discovery loads fine => models.json still written

Matches every observation from the real example run (github/gh-aw run 27768989380, image api-proxy:0.27.4): 73 upstream Copilot CONNECTs from the api-proxy, valid responses with usage returned to the CLI, models.json present, token-usage.jsonl absent.

Audit: 3 files missing

Computing the full require closure from server.js against the COPY list found three silently-disabled modules:

file	required by	subsystem silently disabled
`token-tracker-shared.js`	token-tracker-http/ws	token tracking (AIC)
`otel-exporters.js`	otel.js	OTEL tracing
`otel-serialization.js`	otel.js	OTEL tracing

Fix

Add all three files to the Dockerfile COPY list.
Add dockerfile-copy-coverage.test.js: computes the require closure from server.js and asserts every local module is present in the Dockerfile. This is the second occurrence of this exact bug class (OIDC modules were missed the same way previously), so the guard fails fast in CI.

Tests

New guard test passes against the fix and fails when any required file is removed (verified both directions).
Full api-proxy suite green: 1228 passed.

Relationship to #5253

#5253 fixes a separate cache-split fidelity bug in usage parsing. This PR is the actual AIC=0 root cause — without it, the parser fixed in #5253 is never even invoked inside the container.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

PR #4780 (released in v0.27.3) extracted token-tracker-shared.js but did not add it to the Dockerfile COPY list. Because the image copies files individually by name (no bundler), require('./token-tracker-shared') threw MODULE_NOT_FOUND in the container. The graceful-degradation guard in proxy-request.js then silently stubbed trackTokenUsage to a no-op, so the container booted and served traffic normally but never wrote token-usage.jsonl -- causing AI-credit accounting to report 0 for every run since v0.27.3. models.json still appeared because model-discovery.js does not depend on the missing module. otel-exporters.js and otel-serialization.js (required by otel.js) were likewise missing and silently disabled OTEL tracing via its own graceful-degradation guard. Add all three files to the COPY list and add a guard test that computes the require closure from server.js and asserts every local module is present in the Dockerfile, preventing this recurring class of bug. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-18T16:05:23Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	97.57%	97.61%	📈 +0.04%
Statements	97.50%	97.54%	📈 +0.04%
Functions	98.84%	98.84%	➡️ +0.00%
Branches	92.95%	92.98%	📈 +0.03%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/workdir-setup.ts`	92.7% → 94.5% (+1.82%)	92.7% → 94.5% (+1.82%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

Fixes the api-proxy container image’s incomplete COPY list so that runtime-required modules (token tracking + OpenTelemetry) are actually present in the built image, preventing silent subsystem disablement (notably the AIC=0 regression).

Changes:

Add previously-missing runtime modules (token-tracker-shared.js, otel-exporters.js, otel-serialization.js) to the api-proxy Docker image COPY list.
Add a Jest guard test that computes the require() closure from server.js and asserts every reachable local module is covered by the Dockerfile COPY instructions.

Show a summary per file

File	Description
`containers/api-proxy/Dockerfile`	Extends the explicit `COPY` list to include token-tracker shared helpers and OTEL support modules required at runtime.
`containers/api-proxy/dockerfile-copy-coverage.test.js`	Adds a CI guard to detect missing Dockerfile-copied modules by analyzing local `require()` reachability from `server.js`.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 2/2 changed files
Comments generated: 1

+    const missing = [...closure]
+      .map((abs) => path.relative(ROOT, abs))
+      .filter((rel) => !rel.startsWith('node_modules'))
+      .filter((rel) => !isCopied(rel))
+      .sort();


lpcox · 2026-06-18T16:10:16Z

@copilot address review feedback

github-actions · 2026-06-18T16:10:44Z

⏳ Copilot review left inline comments.

@lpcox To proceed:

Ask @copilot to address the review feedback (reply to this comment or the review thread)
Once the fix is pushed, add the ready-for-aw label to trigger agentic CI smoke tests

…e-copy-coverage test

Copilot · 2026-06-18T16:12:12Z

@copilot address review feedback

Fixed in the latest commit: path.relative() output is now normalized to POSIX separators via .split(path.sep).join('/') before comparing against Dockerfile paths.

github-actions · 2026-06-18T16:15:05Z

❌ Smoke Claude failed

github-actions · 2026-06-18T16:15:10Z

✅ Contribution Check completed successfully!

github-actions · 2026-06-18T16:16:48Z

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

github-actions · 2026-06-18T16:16:48Z

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

github-actions · 2026-06-18T16:16:51Z

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

github-actions · 2026-06-18T16:16:52Z

✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

github-actions · 2026-06-18T16:16:55Z

🔌 Smoke Services — All services reachable! ✅

github-actions · 2026-06-18T16:17:08Z

❌ Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed...

github-actions · 2026-06-18T16:17:21Z

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

github-actions · 2026-06-18T16:17:22Z

❌ Smoke Copilot BYOK reports failed. BYOK mode investigation needed...

github-actions · 2026-06-18T16:17:23Z

✅ Build Test Suite completed successfully!

github-actions · 2026-06-18T16:17:50Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

github-actions · 2026-06-18T16:17:51Z

✅ Smoke Gemini completed. All facets verified. 💎

Testing safeoutputs

github-actions · 2026-06-18T16:23:53Z

🔬 Smoke Test: Copilot PAT Auth — PASS

Test	Result
GitHub MCP connectivity	✅
GitHub.com HTTP	✅
File write/read	✅

PR: fix(api-proxy): copy token-tracker-shared + otel modules into image (fixes AIC=0)
Author: @lpcox
Auth mode: PAT (COPILOT_GITHUB_TOKEN)

Overall: ✅ PASS

🔑 PAT report filed by Smoke Copilot PAT

github-actions · 2026-06-18T16:24:32Z

@lpcox

fix(api-proxy): copy token-tracker-shared + otel modules into image (fixes AIC=0) - ✅
fix(api-proxy): use copilot_usage token_details for accurate cache split - ✅

MCP tool connectivity - ✅
GitHub.com connectivity - ✅
File I/O in sandbox - ✅
Direct BYOK mode via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) - ✅

PASS

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

github-actions · 2026-06-18T16:25:01Z

🔍 Smoke Test: API Proxy OpenTelemetry Tracing

Scenario	Result	Detail
1. Module Loading	✅	`otel.js` loads cleanly; exports: `startRequestSpan`, `setTokenAttributes`, `setBudgetAttributes`, `endSpan`, `endSpanError`, `shutdown`, `isEnabled`
2. Test Suite	✅	59 passed, 0 failed (2 suites: `otel.test.js`, `otel-fanout.test.js`)
3. Env Var Forwarding	✅	`api-proxy-service-config.ts` forwards `GH_AW_OTLP_ENDPOINTS`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, `GITHUB_AW_OTEL_TRACE_ID`, `GITHUB_AW_OTEL_PARENT_SPAN_ID`, `OTEL_SERVICE_NAME`
4. Token Tracker Integration	✅	`onUsage` callback present in `token-tracker-http.js` (lines 283 & 324) as OTEL hook point
5. OTEL Diagnostics	✅	Span exported to `/tmp/gh-aw/otel.jsonl`; parent context propagated (`traceId`+`parentSpanId` set from workflow trace)

All scenarios pass. The fix in this PR (adding otel-exporters.js and otel-serialization.js to the Dockerfile COPY list) unblocks OTEL tracing that was silently disabled since those modules were missing from the image.

📡 OTel tracing validated by Smoke OTel Tracing

github-actions · 2026-06-18T16:25:08Z

Reviewed merged PRs:

Add comprehensive gVisor firewall comparison workflow
refactor: extract provider env var constants to a shared module
Checks:
GitHub title ✅
Temp file ✅
Discussion ✅
Build ✅
Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-06-18T16:25:13Z

Chroot Smoke Test Results

Runtime	Host Version	Chroot Version	Match?
Python	Python 3.12.13	Python 3.12.3	❌
Node.js	v24.16.0	v22.22.3	❌
Go	go1.22.12	go1.22.12	✅

Overall: ❌ Not all tests passed — Python and Node.js versions differ between host and chroot environments.

Tested by Smoke Chroot

github-actions · 2026-06-18T16:25:24Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	All passed	✅ PASS
Node.js	execa	✅	All passed	✅ PASS
Node.js	p-limit	✅	All passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #5254 · 177.1 AIC · ⊞ 33.8K · ◷

github-actions · 2026-06-18T16:26:07Z

Smoke Test: Services Connectivity

Check	Result
Redis PING	❌ no response
PostgreSQL pg_isready	❌ no response
PostgreSQL SELECT 1	❌ no response

Overall: FAIL — host.docker.internal is unreachable on both port 6379 and 5432 from this runner environment.

🔌 Service connectivity validated by Smoke Services

github-actions · 2026-06-18T16:27:19Z

Smoke Test: Gemini Engine Validation - PASS

GitHub MCP: ✅
- fix(api-proxy): copy token-tracker-shared + otel modules into image (fixes AIC=0)
- fix(api-proxy): use copilot_usage token_details for accurate cache split
GitHub.com Connectivity: ✅ (Status 200)
File Writing: ✅
Bash Tools: ✅

Overall status: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

github-actions · 2026-06-18T16:34:07Z

@lpcox

GitHub MCP testing: ✅
GitHub.com HTTP: ✅
File write/read: ✅
Direct BYOK inference: ✅

Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra

Overall: PASS

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

github-actions · 2026-06-18T16:57:59Z

🤖 Smoke Test Results — PASS

Test	Result
GitHub MCP connectivity	✅
github.com HTTP	✅ 200
File write/read	✅

PR: fix(api-proxy): copy token-tracker-shared + otel modules into image (fixes AIC=0)
Author: @lpcox

Overall: PASS

📰 BREAKING: Report filed by Smoke Copilot

Copilot AI review requested due to automatic review settings June 18, 2026 16:03

Copilot started reviewing on behalf of lpcox June 18, 2026 16:04 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Comment thread containers/api-proxy/dockerfile-copy-coverage.test.js

Comment on lines +111 to +115

const missing = [...closure]

.map((abs) => path.relative(ROOT, abs))

.filter((rel) => !rel.startsWith('node_modules'))

.filter((rel) => !isCopied(rel))

.sort();

Copilot started work on behalf of lpcox June 18, 2026 16:11 View session

fix(test): normalize path.relative() to POSIX separators in dockerfil…

9ad6ead

…e-copy-coverage test

Copilot finished work on behalf of lpcox June 18, 2026 16:12

lpcox added the ready-for-aw label Jun 18, 2026

lpcox had a problem deploying to aoai-model June 18, 2026 16:17 — with GitHub Actions Failure

github-actions Bot added the smoke-copilot-pat label Jun 18, 2026

github-actions Bot added the smoke-copilot-byok-aoai-apikey label Jun 18, 2026

github-actions Bot added the smoke-codex label Jun 18, 2026

github-actions Bot mentioned this pull request Jun 18, 2026

[aw] Smoke Copilot failed #5258

Closed

github-actions Bot added the build-test label Jun 18, 2026

github-actions Bot added the smoke-gemini label Jun 18, 2026

lpcox temporarily deployed to aoai-model June 18, 2026 16:33 — with GitHub Actions Inactive

github-actions Bot added the smoke-copilot-byok-aoai-entra label Jun 18, 2026

lpcox temporarily deployed to aoai-model June 18, 2026 16:34 — with GitHub Actions Inactive

github-actions Bot mentioned this pull request Jun 18, 2026

[aw] Smoke Copilot BYOK AOAI (Entra) failed #5261

Closed

github-actions Bot added the smoke-copilot label Jun 18, 2026

lpcox merged commit d4ef2b7 into main Jun 18, 2026
94 of 105 checks passed

lpcox deleted the fix/api-proxy-copy-token-tracker-shared branch June 18, 2026 18:14

This was referenced Jun 18, 2026

[aw] No-Op Runs #5231

Closed

ci(smoke): add token-usage sanity checks to smoke workflows #5264

Merged

Conversation

lpcox commented Jun 18, 2026

Root cause of AIC=0

Reproduction (deterministic)

Audit: 3 files missing

Fix

Tests

Relationship to #5253

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Coverage Check Passed

Overall Coverage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

lpcox commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

Copilot AI commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 18, 2026

🔬 Smoke Test: Copilot PAT Auth — PASS

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

🔍 Smoke Test: API Proxy OpenTelemetry Tracing

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Chroot Smoke Test Results

Uh oh!

github-actions Bot commented Jun 18, 2026

🏗️ Build Test Suite Results

Uh oh!

github-actions Bot commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading

github-actions Bot commented Jun 18, 2026 •

edited

Loading