Skip to content

feat(utils): collect per-run token/time/step usage from SDK traces#130

Merged
keli-wen merged 4 commits into
masterfrom
feat/flow-usage-observability
Jul 22, 2026
Merged

feat(utils): collect per-run token/time/step usage from SDK traces#130
keli-wen merged 4 commits into
masterfrom
feat/flow-usage-observability

Conversation

@keli-wen

Copy link
Copy Markdown
Contributor

Summary

Adds usage_scope, a small quantmind/utils/ helper that reports token, time, and LLM-step usage for one flow run by collecting the spans the Agents SDK already emits — no hand-rolled accounting, no cost, no budget enforcement. This delivers the tokens/time/steps observability half of #127.

usage_scope(name) opens one SDK trace (a minted trace_id) so a run's N+1 Runner.run calls — a paper summary's fan-out researchers plus the reducer, plus any structured-output fallback retry — nest into a single tree; on exit it reads and evicts that trace's spans into a frozen RunUsage:

from quantmind.utils.usage import usage_scope

with usage_scope("quantmind.paper") as run:
    tree = await PaperFlow(cfg).build(input)

print(run.usage.total_tokens, run.usage.requests, run.usage.wall_seconds)
for step in run.usage.steps:
    print(step.label, step.model, step.total_tokens, step.duration_seconds)

RunUsage carries input/output/total_tokens, requests (the llm-steps count), wall_seconds (end-to-end latency) versus busy_seconds (sum of per-step durations — the gap shows how much the fan-out actually overlapped, the signal for tuning concurrency), and a per-step UsageStep list (label, model, tokens, ISO timestamps, and span ids for a waterfall).

Design decisions: a process-global SpanCollector is registered additively via add_trace_processor (it never replaces the SDK's processors and never touches the default backend exporter — whether spans also reach an external dashboard stays the caller's own global tracing config) and buckets spans by trace_id, so concurrent scopes and batch_run fan-out stay isolated. Aggregation counts only leaf generation/response spans (skipping task/turn aggregate spans to avoid double-counting), resolves each step's label from its nearest agent span, and tolerates both usage-dict shapes (input/output and prompt/completion) plus response.usage. Cost, pricing, and budget enforcement are intentionally out of scope. The full rationale lives in contexts/design/utils/usage.md.

This PR also bundles a small, orthogonal docs-convention change that writing the design context motivated: contexts pages no longer hard-wrap prose at a fixed width (fixed-width wrapping is for code, not prose), matching the existing no-hard-wrap rule for GitHub bodies. The rule is flipped in write-contexts.md (both skill mirrors) and github-writing.md, and the existing hard-wrapped contexts pages are reflowed to one paragraph or list item per line — line breaks only, no prose added, removed, or changed (verified by whitespace-normalized comparison). The three commits are cleanly separable for commit-by-commit review, and I am happy to split the convention change into its own PR if you prefer.

Related Issue

Part of #127 (surface token usage and cost for runs and batch_run) — this PR delivers the tokens/time/steps observability; cost and budget are deliberately left out. Also part of #71 (Agents SDK migration).

Verification

  • bash scripts/verify.sh — green: ruff format --check, ruff check, basedpyright, lint-imports, and pytest --cov (405 passed, 85.66% total coverage; quantmind/utils/usage.py at 95%).
  • End-to-end wiring exercised with real Agents SDK spans (real trace() plus agent_span/generation_span flowing through add_trace_processorSpanCollector → aggregation): three steps, correct labels and models, correct token sums, and wall-versus-busy timing.
  • No live-network smoke test applies: the helper reads local traces and makes no network call of its own.

Checklist

  • The title uses English Conventional Commit format: type(scope): summary.
  • The related issue or design discussion is linked when applicable.
  • bash scripts/verify.sh passes.
  • Every applicable live-network component smoke test passes, or this PR states why none applies.
  • Public behavior has focused tests, an example, and documentation where applicable.
  • The PR is complete, small, and contains no unrelated changes. (Bundles an orthogonal contexts-wrapping convention change; can be split on request.)

keli-wen and others added 3 commits July 22, 2026 21:55
Fixed-width line wrapping is for code, not prose; hard-wrapped Markdown
reads poorly and fights soft-wrapping renderers. Flip the contexts
authoring rule to the same no-hard-wrap style already used for GitHub
bodies: one paragraph or list item per physical line.

- write-contexts.md (both .agents and .claude mirrors): replace the
  "hard-wrap at ~76 columns" rule with the no-hard-wrap prose rule.
- github-writing.md: its prose rule now also governs repository contexts
  pages, not only GitHub bodies.
- Reflow existing hard-wrapped contexts pages to one line per paragraph
  or list item. Line breaks only -- no prose was added, removed, or
  changed (verified by whitespace-normalized comparison). Tables, code
  fences, and mermaid blocks are untouched.

tests/test_contexts.py stays green: headings, Contents anchors, and
index links are unaffected by unwrapping.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Design contract for quantmind/utils/usage.py (not yet implemented): a
per-flow-run readout of token usage, wall/busy time, and LLM step count,
built by collecting the spans the Agents SDK already emits rather than
hand-rolling a second accounting path.

- SpanCollector: one process-global TracingProcessor, added (never
  set/replacing) via add_trace_processor, bucketing spans by trace_id.
- usage_scope(name): mints a trace_id and wraps the flow in trace() so a
  run's N+1 Runner.run calls collapse into one tree, then reads and
  evicts that trace's spans on exit into a RunUsage.
- RunUsage / UsageStep: frozen value readouts; requests == llm-steps;
  wall vs busy seconds expose fan-out parallelism.

Scope: flow LLM calls (paper structure + summary). Tokens/time/steps
only -- no cost, no budget enforcement. Registered in the design index.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
usage_scope wraps a flow run and reports one RunUsage -- input/output/total
tokens, an llm-steps count (requests), and wall vs busy time -- by reading the
spans the Agents SDK already emits, not by hand-rolling a second accounting path.

- SpanCollector: one process-global TracingProcessor, added (never replacing)
  via add_trace_processor, bucketing finished spans by trace_id.
- usage_scope(name): mints a trace_id and opens one trace() so a run's N+1
  Runner.run calls (fan-out researchers + reducer, plus any fallback retry) nest
  into a single tree; on exit it reads and evicts that trace's spans and
  aggregates them into RunUsage.
- Aggregation counts only leaf generation/response spans (task/turn aggregate
  spans are skipped to avoid double-counting), resolves each step's label from
  its nearest agent span, and tolerates both usage-dict shapes (input/output and
  prompt/completion) plus response.usage.
- Observation only: no cost, no budget, no enforcement.

Adds quantmind/utils/usage.py, offline tests (also verified end-to-end against
real SDK spans), and examples/utils/usage_scope.py. Updates the utils guidance
in both skill mirrors (logger-only -> logger plus shared SDK-wrapping leaf
helpers such as structured_output and usage).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@keli-wen keli-wen added type: feature Adds a new capability or observable behavior area: harness Contributor and agent controls, CI, hooks, skills, and rulesets area: contexts Repository information routing and discovery under contexts/ area: utils The narrowly owned quantmind/utils/ surface labels Jul 22, 2026
@keli-wen keli-wen self-assigned this Jul 22, 2026
@keli-wen
keli-wen force-pushed the feat/flow-usage-observability branch 2 times, most recently from 939780e to f24ab17 Compare July 22, 2026 15:49
Wrap PaperFlow(PaperStructureCfg).build inside usage_scope and print the per-run
usage. The "Example output" block is a real run on OpenRouter's deepseek-v4-flash:
OpenRouter accepts the strict json_schema request, so it is a single clean call
(requests=1).

The docstring notes the two other shapes without inventing numbers -- a
json-object-only provider adds a rejected strict attempt before its json_object
fallback (requests=2), and a fan-out flow such as the summary map-reduce shows
one step per researcher plus the reducer, where wall drops below busy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@keli-wen
keli-wen force-pushed the feat/flow-usage-observability branch from f24ab17 to fef588b Compare July 22, 2026 16:37
@wanghaoxue0
wanghaoxue0 self-requested a review July 22, 2026 19:23

@wanghaoxue0 wanghaoxue0 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all good!

@wanghaoxue0 wanghaoxue0 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all good!

@keli-wen
keli-wen merged commit 1edcb12 into master Jul 22, 2026
1 check passed
@keli-wen
keli-wen deleted the feat/flow-usage-observability branch July 22, 2026 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: contexts Repository information routing and discovery under contexts/ area: harness Contributor and agent controls, CI, hooks, skills, and rulesets area: utils The narrowly owned quantmind/utils/ surface type: feature Adds a new capability or observable behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants