Overview
The file pkg/cli/token_usage.go has grown to 1141 lines, making it difficult to maintain and test. This task involves refactoring it into smaller, focused files with improved test coverage.
Current State
- File:
pkg/cli/token_usage.go
- Size: 1141 lines
- Test file:
pkg/cli/token_usage_test.go (782 lines, ratio ~0.69)
- Complexity: High — the file mixes data models, file parsing, subagent attribution, steering-event detection, AIC computation, and summary aggregation
Full File Analysis
Logical Groups Found
Data models / structs (lines 29–131):
TokenCoreMetrics, TokenUsageEntry, AmbientContextMetrics, TokenUsageSummary, ModelTokenUsage, ModelTokenUsageRow, SubagentModelRequest, SubagentModelActual, agentUsageEntry, proxyEventsEntry
Token-usage JSONL parsing (lines 136–360):
parseTokenUsageFile, extractAmbientContextMetrics, parseTokenUsageTimestamp, findTokenUsageFile
Agent-usage JSON parsing (lines 360–500):
findAgentUsageFile, parseAgentUsageFile
AIC (AI Credits) computation (lines 543–777):
findUsageJSONLFiles, extractUsageRecord, usageNumericValue, usageStringValue, isFinite, sumAICFromUsageJSONLFiles, processOneUsageJSONLFile, analyzeTokenUsageAICOnly
Steering events detection (lines 778–894):
countAPIProxySteeringEvents, findAPIProxyEventsFile, scanSteeringEntries, parseAPIProxySteeringEvents, containsSteeringKeyword, isSteeringEvent
Subagent attribution (lines 896–1072):
augmentSubagentModelAttribution, addTokenUsageWarning, extractSubagentModelRequests, findAgentStdioFile, correlateToolCallsWithTokenDelta
Summary methods / AIC population (lines 1079–1141):
TotalTokens, AvgDurationMs, ModelRows, populateAIC
Top-level orchestrator (lines 501–542):
Refactoring Strategy
Proposed File Splits
-
token_usage_models.go
- Types:
TokenCoreMetrics, TokenUsageEntry, AmbientContextMetrics, TokenUsageSummary, ModelTokenUsage, ModelTokenUsageRow, SubagentModelRequest, SubagentModelActual
- Methods:
TotalTokens, AvgDurationMs, ModelRows
- Responsibility: Shared data types and summary helpers
- Estimated LOC: ~200
-
token_usage_parser.go
- Functions:
parseTokenUsageFile, extractAmbientContextMetrics, parseTokenUsageTimestamp, findTokenUsageFile, parseAgentUsageFile, findAgentUsageFile, agentUsageEntry
- Responsibility: Parse JSONL/JSON log files into domain structs
- Estimated LOC: ~300
-
token_usage_aic.go
- Functions:
findUsageJSONLFiles, extractUsageRecord, usageNumericValue, usageStringValue, isFinite, sumAICFromUsageJSONLFiles, processOneUsageJSONLFile, analyzeTokenUsageAICOnly, populateAIC
- Responsibility: AI Credits computation from raw usage JSONL
- Estimated LOC: ~260
-
token_usage_steering.go
- Types:
proxyEventsEntry
- Functions:
countAPIProxySteeringEvents, findAPIProxyEventsFile, scanSteeringEntries, parseAPIProxySteeringEvents, containsSteeringKeyword, isSteeringEvent
- Responsibility: Detect and count steering events from proxy event logs
- Estimated LOC: ~130
-
token_usage_subagent.go
- Functions:
augmentSubagentModelAttribution, addTokenUsageWarning, extractSubagentModelRequests, findAgentStdioFile, correlateToolCallsWithTokenDelta
- Responsibility: Attribute token usage to subagents from stdio logs
- Estimated LOC: ~200
-
token_usage.go (shrunk to orchestrator + constants)
- Functions:
analyzeTokenUsage
- Constants and package-level vars
- Responsibility: Entry point that wires together all sub-packages
- Estimated LOC: ~80
Shared Utilities
No new utility files needed; helpers (usageNumericValue, usageStringValue, isFinite) should move into token_usage_aic.go alongside their callers.
Interface Abstractions
- Consider a
logFileLocator interface to abstract find*File helpers, making it easier to test with in-memory paths.
Test Coverage Plan
Add or redistribute tests across new files:
-
token_usage_models_test.go
- Test
TotalTokens, AvgDurationMs, ModelRows with various inputs
- Target coverage: >80%
-
token_usage_parser_test.go
- Test JSONL parsing with valid, malformed, and empty files
- Test
extractAmbientContextMetrics edge cases (missing first entry, etc.)
- Target coverage: >80%
-
token_usage_aic_test.go
- Test
sumAICFromUsageJSONLFiles with single and multi-file scenarios
- Test
processOneUsageJSONLFile with various key name variants
- Target coverage: >80%
-
token_usage_steering_test.go
- Test
scanSteeringEntries with token and timeout events, noise lines
- Test
isSteeringEvent with all known event names
- Target coverage: >80%
-
token_usage_subagent_test.go
- Test
extractSubagentModelRequests with realistic stdio log snippets
- Test
augmentSubagentModelAttribution with mismatch scenarios
- Target coverage: >80%
Implementation Guidelines
- Preserve Behavior: Ensure all existing functionality works identically
- Maintain Exports: Keep public API unchanged (exported functions/types)
- Add Tests First: Write tests for each new file before refactoring
- Incremental Changes: Split one module at a time
- Run Tests Frequently: Verify
make test-unit passes after each split
- Update Imports: Ensure all import paths are correct
- Document Changes: Add comments explaining module boundaries
Acceptance Criteria
Additional Context
- Repository Guidelines: Follow patterns in
.github/agents/developer.instructions.agent.md
- Code Organization: Prefer many small files grouped by functionality
- Testing: Match existing test patterns in
pkg/cli/*_test.go
Priority: Medium
Effort: Medium (well-defined boundaries, no external API changes needed)
Expected Impact: Improved maintainability, easier testing, reduced complexity per file
Generated by 🧹 Daily File Diet · sonnet46 79.5 AIC · ⌖ 7.02 AIC · ⊞ 6.8K · ◷
Overview
The file
pkg/cli/token_usage.gohas grown to 1141 lines, making it difficult to maintain and test. This task involves refactoring it into smaller, focused files with improved test coverage.Current State
pkg/cli/token_usage.gopkg/cli/token_usage_test.go(782 lines, ratio ~0.69)Full File Analysis
Logical Groups Found
Data models / structs (lines 29–131):
TokenCoreMetrics,TokenUsageEntry,AmbientContextMetrics,TokenUsageSummary,ModelTokenUsage,ModelTokenUsageRow,SubagentModelRequest,SubagentModelActual,agentUsageEntry,proxyEventsEntryToken-usage JSONL parsing (lines 136–360):
parseTokenUsageFile,extractAmbientContextMetrics,parseTokenUsageTimestamp,findTokenUsageFileAgent-usage JSON parsing (lines 360–500):
findAgentUsageFile,parseAgentUsageFileAIC (AI Credits) computation (lines 543–777):
findUsageJSONLFiles,extractUsageRecord,usageNumericValue,usageStringValue,isFinite,sumAICFromUsageJSONLFiles,processOneUsageJSONLFile,analyzeTokenUsageAICOnlySteering events detection (lines 778–894):
countAPIProxySteeringEvents,findAPIProxyEventsFile,scanSteeringEntries,parseAPIProxySteeringEvents,containsSteeringKeyword,isSteeringEventSubagent attribution (lines 896–1072):
augmentSubagentModelAttribution,addTokenUsageWarning,extractSubagentModelRequests,findAgentStdioFile,correlateToolCallsWithTokenDeltaSummary methods / AIC population (lines 1079–1141):
TotalTokens,AvgDurationMs,ModelRows,populateAICTop-level orchestrator (lines 501–542):
analyzeTokenUsageRefactoring Strategy
Proposed File Splits
token_usage_models.goTokenCoreMetrics,TokenUsageEntry,AmbientContextMetrics,TokenUsageSummary,ModelTokenUsage,ModelTokenUsageRow,SubagentModelRequest,SubagentModelActualTotalTokens,AvgDurationMs,ModelRowstoken_usage_parser.goparseTokenUsageFile,extractAmbientContextMetrics,parseTokenUsageTimestamp,findTokenUsageFile,parseAgentUsageFile,findAgentUsageFile,agentUsageEntrytoken_usage_aic.gofindUsageJSONLFiles,extractUsageRecord,usageNumericValue,usageStringValue,isFinite,sumAICFromUsageJSONLFiles,processOneUsageJSONLFile,analyzeTokenUsageAICOnly,populateAICtoken_usage_steering.goproxyEventsEntrycountAPIProxySteeringEvents,findAPIProxyEventsFile,scanSteeringEntries,parseAPIProxySteeringEvents,containsSteeringKeyword,isSteeringEventtoken_usage_subagent.goaugmentSubagentModelAttribution,addTokenUsageWarning,extractSubagentModelRequests,findAgentStdioFile,correlateToolCallsWithTokenDeltatoken_usage.go(shrunk to orchestrator + constants)analyzeTokenUsageShared Utilities
No new utility files needed; helpers (
usageNumericValue,usageStringValue,isFinite) should move intotoken_usage_aic.goalongside their callers.Interface Abstractions
logFileLocatorinterface to abstractfind*Filehelpers, making it easier to test with in-memory paths.Test Coverage Plan
Add or redistribute tests across new files:
token_usage_models_test.goTotalTokens,AvgDurationMs,ModelRowswith various inputstoken_usage_parser_test.goextractAmbientContextMetricsedge cases (missing first entry, etc.)token_usage_aic_test.gosumAICFromUsageJSONLFileswith single and multi-file scenariosprocessOneUsageJSONLFilewith various key name variantstoken_usage_steering_test.goscanSteeringEntrieswith token and timeout events, noise linesisSteeringEventwith all known event namestoken_usage_subagent_test.goextractSubagentModelRequestswith realistic stdio log snippetsaugmentSubagentModelAttributionwith mismatch scenariosImplementation Guidelines
make test-unitpasses after each splitAcceptance Criteria
token_usage.gois split into 5–6 focused filesmake test-unit)make lint)make build)Additional Context
.github/agents/developer.instructions.agent.mdpkg/cli/*_test.goPriority: Medium
Effort: Medium (well-defined boundaries, no external API changes needed)
Expected Impact: Improved maintainability, easier testing, reduced complexity per file