Bug: trace usage semantics differ by provider for tool-call runs

[Diluka]

Version

@voltagent/core: 2.8.0

Describe the bug

Trace-level usage semantics appear to differ by provider/model during tool-call runs.

For OpenAI/GPT tool-call traces, the root/agent span appears to record cumulative totalUsage across the full streamText run. For Anthropic/Claude tool-call traces, the root/agent span appears to record usage / last-step-or-provider usage instead.

This is a separate issue from trace summary totalTokens double-counting cached tokens. The problem here is that the same trace/root usage fields can mean different things depending on provider.

Why this is a problem

We listen to the same VoltAgent observability hooks and read the same fields:

usage.prompt_tokens
usage.completion_tokens
usage.total_tokens
usage.cached_tokens

But when only the model/provider changes, the semantics appear to change:

• OpenAI/GPT: trace/root usage looks cumulative across multiple model requests inside a tool-calling agent run.
• Anthropic/Claude: trace/root usage can look like last-step/provider usage even when the trace contains multiple tool calls and multiple conversation steps.

That makes provider comparisons and billing/usage dashboards very hard to reason about. A field named trace/root usage.* should have one consistent meaning across providers.

Evidence from traces

OpenAI/GPT trace with tool calls

The trace contains tool calls and multiple conversation step persists:

tool calls: currentDateTime, getBrandDetails
persisted step groups: 2, 2, 1

Usage values:

root usage.prompt_tokens      = 161251
root usage.completion_tokens  = 349
root usage.total_tokens       = 161600
root usage.cached_tokens      = 160768

llm.usage.prompt_tokens       = 53909
llm.usage.completion_tokens   = 122
llm.usage.total_tokens        = 54031
llm.usage.cached_tokens       = 53760

The root prompt usage is roughly 3x the LLM span prompt usage, which is consistent with cumulative usage across multiple requests/steps:

161251 ~= 3 * 53909

Anthropic/Claude trace with tool calls

The trace also contains tool calls and multiple conversation step persists:

tool calls: currentDateTime, getBrandDetails, getDataSourceConnectionStatus, querySocialMediaPostsAndComments x2
persisted step groups: 4, 4, 2, 1

Usage values:

root usage.prompt_tokens      = 74865
root usage.completion_tokens  = 274
root usage.total_tokens       = 75139
root usage.cached_tokens      = 74610

llm.usage.prompt_tokens       = 74865
llm.usage.completion_tokens   = 274
llm.usage.total_tokens        = 75139
llm.usage.cached_tokens       = 74610

Here root usage equals LLM span usage, despite the run containing multiple tool calls/steps. This suggests the trace/root usage is not cumulative in the same way as the OpenAI/GPT trace.

Source-level cause

In @voltagent/core@2.8.0, resolveFinishUsage() normally prefers totalUsage, but switches to usage for Anthropic/cache cases.

Relevant source files:

packages/core/src/utils/usage-normalizer.ts
packages/core/src/agent/agent.ts

The relevant behavior is:

providerMetadata has anthropic -> shouldUseLastStepUsage = true
resolveFinishUsage(...) -> returns usage instead of totalUsage
recordRootSpanUsageAndProviderCost(...) -> writes that value to the root span

This makes Anthropic traces use a different root usage semantics than non-Anthropic traces.

Expected behavior

Trace/root usage fields should have consistent semantics across providers.

If usage.prompt_tokens on the root span means cumulative usage for the full agent run, then it should mean that for Anthropic as well.

If Anthropic needs special cache-aware handling, VoltAgent should expose separate fields instead of changing the meaning of root usage.*, for example:

usage.prompt_tokens              // cumulative trace/run usage
usage.completion_tokens          // cumulative trace/run usage
usage.total_tokens               // cumulative trace/run usage
usage.last_step.prompt_tokens    // final/provider step usage
usage.cache_read_tokens
usage.cache_creation_tokens

Actual behavior

The same root usage fields appear to represent cumulative usage for GPT/OpenAI traces but last-step/provider usage for Claude/Anthropic traces.

Related

Related but separate from #1359, which is about exported trace summary totalTokens double-counting cached/reasoning token breakdowns.

Impact on onEnd hook consumers

This also affects application code that consumes usage from the agent onEnd hook.

For example, we currently track token usage from:

onEnd: async ({ agent, output, context, error }) => {
  const usage = {
    inputTokens: output?.usage?.promptTokens || 0,
    outputTokens: output?.usage?.completionTokens || 0,
    totalTokens: output?.usage?.totalTokens || 0,
    reasoningTokens: output?.usage?.reasoningTokens || 0,
    cachedInputTokens: output?.usage?.cachedInputTokens || 0,
  };
}

Because output.usage appears to use different semantics depending on provider/model, this makes our own token accounting incorrect when switching models while listening to the same hook.

The expectation is not necessarily that output.usage must be cumulative or non-cumulative; the important requirement is that it must be consistent across providers:

• If output.usage represents cumulative usage for the full agent run, it should do so for all providers.
• If output.usage represents last-step/provider usage, it should do so for all providers.
• If both are useful, they should be exposed as separate fields with explicit names.

Right now the same hook field can mean different things for OpenAI vs Anthropic tool-call runs, which is very hard for downstream applications to handle reliably.

Code evidence permalinks

The following permalinks are from the @voltagent/core@2.8.0 tag commit 0c1a1a4a3b4ad01b5a62ae91b4370f8e685cbba3.

1. Anthropic/cache cases are explicitly routed to usage instead of totalUsage:

• shouldUseLastStepUsage() returns true when providerMetadata has anthropic
• resolveFinishUsage() returns usage ?? totalUsage when shouldUseLastStepUsage(...) is true; otherwise it returns totalUsage ?? usage

2. streamText passes both finalResult.usage and finalResult.totalUsage into resolveFinishUsage():

• providerUsage is read from finalResult.usage, then resolveFinishUsage({ usage: providerUsage, totalUsage: finalResult.totalUsage }) is called

3. The resolved value is what gets written to the root trace usage and exposed to onEnd:

• recordRootSpanUsageAndProviderCost(..., usageForFinish, ...) writes the resolved usage to the root trace context
• const usage = convertUsage(usageForFinish)
• onEnd receives output: { text, usage, ... }, so hook consumers see this provider-dependent resolved usage

4. setTraceContextUsage() maps the resolved usage into root span attributes:

• traceContext.setUsage({ promptTokens, completionTokens, totalTokens, cachedTokens, reasoningTokens })

These links show why the same onEnd hook field and root span usage.* attributes can resolve from totalUsage for non-Anthropic providers but from usage for Anthropic/cache cases.

[buyicoder]

I took a quick pass through current main to see whether this is likely to be a small fix or a contract/design fix.

The provider-dependent behavior looks intentionally encoded in two places:

• packages/core/src/utils/usage-normalizer.ts: shouldUseLastStepUsage() returns true for Anthropic provider metadata and for raw cache fields (cache_creation_input_tokens / cache_read_input_tokens). When true, resolveFinishUsage() returns usage ?? totalUsage instead of totalUsage ?? usage.
• normalizeFinishUsageStream() also rewrites a finish part's totalUsage to the last finish-step usage for those Anthropic/cache cases.

The tests in packages/core/src/utils/usage-normalizer.spec.ts lock that behavior in: "prefers last-step usage when provider metadata indicates anthropic", "prefers last-step usage when usage includes cache fields", and the stream normalization tests all expect the Anthropic/cache path to replace cumulative totalUsage with the last-step usage.

So I would be cautious about a direct one-line flip to always prefer totalUsage: it would probably make root/onEnd usage cumulative again, but it would also undo the cache-aware behavior those tests currently protect.

A safer fix path might be to split the contract explicitly, e.g. keep one field for cumulative run usage and a separate field for provider/last-step/cache-detail usage, then write root span attributes and onEnd.output.usage from the same documented source across providers. The regression test should probably cover the two traces described here: one OpenAI multi-step tool run and one Anthropic multi-step tool run, asserting that the root span and onEnd usage semantics match even when cache detail fields are present.

[omeraplak]

Thanks for the thorough write-up and source-level pointers. I opened #1366 to make the root/onEnd usage semantics consistent across providers.

The change removes the Anthropic/cache-specific last-step override. resolveFinishUsage now consistently prefers aggregate totalUsage when AI SDK provides it, and the stream normalizer preserves finish totalUsage instead of rewriting it based on provider metadata.

I also added regression coverage for the Anthropic/cache cases so tool-call runs keep aggregate usage consistently.

[buyicoder]

Thanks for the quick fix and for linking the resolution. #1366 looks good to me.