Skip to content

feat(compare): add counterfactual pricing#238

Open
ozymandiashh wants to merge 1 commit intogetagentseal:mainfrom
ozymandiashh:feat/counterfactual-pricing
Open

feat(compare): add counterfactual pricing#238
ozymandiashh wants to merge 1 commit intogetagentseal:mainfrom
ozymandiashh:feat/counterfactual-pricing

Conversation

@ozymandiashh
Copy link
Copy Markdown
Contributor

Summary

This adds a pricing-only what-if mode to codeburn compare: codeburn compare --reprice <model>. The normal interactive model comparison stays unchanged, but users can now ask how their recorded usage would have priced if every call had used a different model.

The calculation keeps the original usage shape: input tokens, output tokens, cache creation, cache reads, web-search requests, and fast-mode flags are reused exactly as recorded. Only the model pricing table changes. That makes the output useful for budget planning while keeping the scope honest: it does not claim to simulate model quality, different reasoning behavior, or shorter/longer outputs.

What changed

  • add codeburn compare --reprice <model> as a non-interactive what-if pricing mode
  • add --json for scripted/reporting workflows when --reprice is used
  • validate the target model before parsing sessions, using the existing pricing and alias lookup
  • report actual spend, what-if spend, savings or added cost, and percent difference
  • surface the largest session impacts, plus project and source-model breakdowns
  • skip <synthetic> calls so provider bookkeeping entries are not repriced as real API calls
  • preserve fast-mode pricing semantics for the target model
  • document the new workflow and add an Unreleased changelog entry

Example

codeburn compare --reprice claude-sonnet-4-5
codeburn compare -p week --provider claude --reprice gpt-5.3-codex
codeburn compare --reprice gpt-4o-mini --json

Output shape

The text output is meant for quick budget decisions:

  • total actual vs what-if spend
  • net savings or added cost
  • top session impacts sorted by absolute difference
  • project-level impact
  • source-model impact

The JSON output returns the same structure in USD fields so downstream scripts do not have to parse localized currency strings.

Validation

  • npx vitest run tests/reprice.test.ts
  • npx vitest run
  • npm run build
  • node dist/cli.js compare --help
  • node dist/cli.js compare --reprice gpt-4o-mini -p today --json
  • node dist/cli.js compare --json

Note: full npx tsc --noEmit is still blocked on origin/main by existing src/providers/copilot.ts type errors; this branch does not touch that provider.

@ozymandiashh ozymandiashh marked this pull request as ready for review May 5, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant