feat(utils): collect per-run token/time/step usage from SDK traces#130
Merged
Conversation
Fixed-width line wrapping is for code, not prose; hard-wrapped Markdown reads poorly and fights soft-wrapping renderers. Flip the contexts authoring rule to the same no-hard-wrap style already used for GitHub bodies: one paragraph or list item per physical line. - write-contexts.md (both .agents and .claude mirrors): replace the "hard-wrap at ~76 columns" rule with the no-hard-wrap prose rule. - github-writing.md: its prose rule now also governs repository contexts pages, not only GitHub bodies. - Reflow existing hard-wrapped contexts pages to one line per paragraph or list item. Line breaks only -- no prose was added, removed, or changed (verified by whitespace-normalized comparison). Tables, code fences, and mermaid blocks are untouched. tests/test_contexts.py stays green: headings, Contents anchors, and index links are unaffected by unwrapping. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Design contract for quantmind/utils/usage.py (not yet implemented): a per-flow-run readout of token usage, wall/busy time, and LLM step count, built by collecting the spans the Agents SDK already emits rather than hand-rolling a second accounting path. - SpanCollector: one process-global TracingProcessor, added (never set/replacing) via add_trace_processor, bucketing spans by trace_id. - usage_scope(name): mints a trace_id and wraps the flow in trace() so a run's N+1 Runner.run calls collapse into one tree, then reads and evicts that trace's spans on exit into a RunUsage. - RunUsage / UsageStep: frozen value readouts; requests == llm-steps; wall vs busy seconds expose fan-out parallelism. Scope: flow LLM calls (paper structure + summary). Tokens/time/steps only -- no cost, no budget enforcement. Registered in the design index. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
usage_scope wraps a flow run and reports one RunUsage -- input/output/total tokens, an llm-steps count (requests), and wall vs busy time -- by reading the spans the Agents SDK already emits, not by hand-rolling a second accounting path. - SpanCollector: one process-global TracingProcessor, added (never replacing) via add_trace_processor, bucketing finished spans by trace_id. - usage_scope(name): mints a trace_id and opens one trace() so a run's N+1 Runner.run calls (fan-out researchers + reducer, plus any fallback retry) nest into a single tree; on exit it reads and evicts that trace's spans and aggregates them into RunUsage. - Aggregation counts only leaf generation/response spans (task/turn aggregate spans are skipped to avoid double-counting), resolves each step's label from its nearest agent span, and tolerates both usage-dict shapes (input/output and prompt/completion) plus response.usage. - Observation only: no cost, no budget, no enforcement. Adds quantmind/utils/usage.py, offline tests (also verified end-to-end against real SDK spans), and examples/utils/usage_scope.py. Updates the utils guidance in both skill mirrors (logger-only -> logger plus shared SDK-wrapping leaf helpers such as structured_output and usage). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
keli-wen
force-pushed
the
feat/flow-usage-observability
branch
2 times, most recently
from
July 22, 2026 15:49
939780e to
f24ab17
Compare
This was referenced Jul 22, 2026
Wrap PaperFlow(PaperStructureCfg).build inside usage_scope and print the per-run usage. The "Example output" block is a real run on OpenRouter's deepseek-v4-flash: OpenRouter accepts the strict json_schema request, so it is a single clean call (requests=1). The docstring notes the two other shapes without inventing numbers -- a json-object-only provider adds a rejected strict attempt before its json_object fallback (requests=2), and a fan-out flow such as the summary map-reduce shows one step per researcher plus the reducer, where wall drops below busy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
keli-wen
force-pushed
the
feat/flow-usage-observability
branch
from
July 22, 2026 16:37
f24ab17 to
fef588b
Compare
wanghaoxue0
self-requested a review
July 22, 2026 19:23
This was referenced Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
usage_scope, a smallquantmind/utils/helper that reports token, time, and LLM-step usage for one flow run by collecting the spans the Agents SDK already emits — no hand-rolled accounting, no cost, no budget enforcement. This delivers the tokens/time/steps observability half of #127.usage_scope(name)opens one SDKtrace(a mintedtrace_id) so a run'sN+1Runner.runcalls — a paper summary's fan-out researchers plus the reducer, plus any structured-output fallback retry — nest into a single tree; on exit it reads and evicts that trace's spans into a frozenRunUsage:RunUsagecarriesinput/output/total_tokens,requests(the llm-steps count),wall_seconds(end-to-end latency) versusbusy_seconds(sum of per-step durations — the gap shows how much the fan-out actually overlapped, the signal for tuning concurrency), and a per-stepUsageSteplist (label, model, tokens, ISO timestamps, and span ids for a waterfall).Design decisions: a process-global
SpanCollectoris registered additively viaadd_trace_processor(it never replaces the SDK's processors and never touches the default backend exporter — whether spans also reach an external dashboard stays the caller's own global tracing config) and buckets spans bytrace_id, so concurrent scopes andbatch_runfan-out stay isolated. Aggregation counts only leafgeneration/responsespans (skippingtask/turnaggregate spans to avoid double-counting), resolves each step's label from its nearestagentspan, and tolerates both usage-dict shapes (input/outputandprompt/completion) plusresponse.usage. Cost, pricing, and budget enforcement are intentionally out of scope. The full rationale lives incontexts/design/utils/usage.md.This PR also bundles a small, orthogonal docs-convention change that writing the design context motivated: contexts pages no longer hard-wrap prose at a fixed width (fixed-width wrapping is for code, not prose), matching the existing no-hard-wrap rule for GitHub bodies. The rule is flipped in
write-contexts.md(both skill mirrors) andgithub-writing.md, and the existing hard-wrapped contexts pages are reflowed to one paragraph or list item per line — line breaks only, no prose added, removed, or changed (verified by whitespace-normalized comparison). The three commits are cleanly separable for commit-by-commit review, and I am happy to split the convention change into its own PR if you prefer.Related Issue
Part of #127 (surface token usage and cost for runs and
batch_run) — this PR delivers the tokens/time/steps observability; cost and budget are deliberately left out. Also part of #71 (Agents SDK migration).Verification
bash scripts/verify.sh— green:ruff format --check,ruff check,basedpyright,lint-imports, andpytest --cov(405 passed, 85.66% total coverage;quantmind/utils/usage.pyat 95%).trace()plusagent_span/generation_spanflowing throughadd_trace_processor→SpanCollector→ aggregation): three steps, correct labels and models, correct token sums, and wall-versus-busy timing.Checklist
type(scope): summary.bash scripts/verify.shpasses.