feat(core): add claude provider using @anthropic-ai/claude-agent-sdk

[christso]

Summary

Replace the current claude-code provider (subprocess scraper) with a new claude provider using the official @anthropic-ai/claude-agent-sdk. This follows the same pattern as the copilot provider (PR #211) which replaced the subprocess-based copilot-cli with the typed SDK.

Motivation

The current claude-code provider spawns the Claude CLI as a subprocess and scrapes stdout. The Agent SDK provides:

• Structured messages via AsyncGenerator<SDKMessage> - typed SDKAssistantMessage, SDKResultMessage
• Token usage & cost from SDKResultMessage.usage, modelUsage, total_cost_usd
• Duration from SDKResultMessage.duration_ms
• Tool call tracking via assistant message content blocks or PostToolUse hooks
• Permission control via permissionMode: 'bypassPermissions' (no interactive prompts)
• Model selection via options.model
• Budget control via options.maxBudgetUsd and options.maxTurns
• Abort support via options.abortController

Competitive Advantage

AgentV's SDK-based providers extract structured tool call traces as outputMessages — enabling tool_trajectory and tool_call_f1 evaluation. Most eval frameworks treat coding agents as text-in/text-out black boxes. AgentV's SDK integrations give evaluators access to:

• Every tool call the agent made (name, arguments, result)
• Token usage breakdown (input, output, cached)
• Cost in USD (from the SDK, not estimated)
• Duration (SDK-reported, not wall-clock)
• Turn count and model breakdown

This structured data is what makes AgentV uniquely suited for evaluating coding agents — you can evaluate HOW the agent worked, not just its final output.

SDK API Overview

import { query } from '@anthropic-ai/claude-agent-sdk';

const q = query({
  prompt: 'What is 2+2?',
  options: {
    model: 'claude-sonnet-4-5-20250929',
    permissionMode: 'bypassPermissions',
    allowDangerouslySkipPermissions: true,
    cwd: '/path/to/workspace',
    systemPrompt: 'custom system prompt',
    maxTurns: 50,
    maxBudgetUsd: 1.0,
    abortController: new AbortController(),
  }
});

for await (const message of q) {
  if (message.type === 'assistant') {
    // message.message contains APIAssistantMessage with content blocks
    // Extract tool_use blocks → ToolCall[]
  }
  if (message.type === 'result') {
    // message.result - final text
    // message.usage - { input_tokens, output_tokens, cache_* }
    // message.modelUsage - per-model breakdown with costUSD
    // message.total_cost_usd - total cost
    // message.duration_ms - total duration
    // message.num_turns - number of turns
  }
}

Implementation Plan

Follow the same pattern as the copilot provider (see PR #211):

Files to modify

1. types.ts - Rename claude-code → claude in ProviderKind, add claude-code as alias
2. targets.ts - Update resolved config and target resolution
3. index.ts - Wire new provider, remove old
4. targets-validator.ts - Update settings validation

Files to create

1. claude.ts - New provider using @anthropic-ai/claude-agent-sdk
   • Lazy-load SDK (same pattern as copilot provider)
   • Use query() with permissionMode: 'bypassPermissions'
   • Map SDKAssistantMessage content blocks → ToolCall[] and OutputMessage[]
   • Extract usage, total_cost_usd, duration_ms from SDKResultMessage
2. claude-log-tracker.ts - Log tracker (rename from claude-code variant)
3. claude.test.ts - Unit tests with mocked SDK

Files to delete

1. claude-code.ts - Old subprocess provider
2. claude-code-log-tracker.ts - Old log tracker
3. claude-code.test.ts - Old tests (if exists)

Dependencies

• Add @anthropic-ai/claude-agent-sdk to packages/core/package.json and apps/cli/package.json
• Add to CLI's tsup.config.ts external list

Key Design Decisions

• Naming: claude (canonical), claude-code (alias for backward compat)
• Permissions: Use bypassPermissions + allowDangerouslySkipPermissions for unattended eval
• Tool calls: Extract from SDKAssistantMessage.message.content blocks (type tool_use / tool_result)
• Streaming: Iterate the async generator, collect all messages, extract result at end
• Structured output: Map every tool_use content block to ToolCall for trajectory evaluation

References

• TypeScript Agent SDK docs
• PR feat(core): add copilot-sdk provider #211 - copilot provider implementation (pattern to follow)

[christso]

Agent-Ready Implementation Plan (Final)

Branch

git checkout -b feat/213-claude-agent-sdk-provider

Prerequisites

cd packages/core && bun add @anthropic-ai/claude-agent-sdk
cd apps/cli && bun add @anthropic-ai/claude-agent-sdk

Package confirmed: @anthropic-ai/claude-agent-sdk v0.2.47+ (npm, docs, Promptfoo reference). Use the stable V1 API (async generator with query()). A V2 preview exists but is not stable yet. The old @anthropic-ai/claude-code package is deprecated — use @anthropic-ai/claude-agent-sdk.

Reference files to read first

1. packages/core/src/evaluation/providers/copilot-sdk.ts — Primary pattern reference (PR feat(core): add copilot-sdk provider #211)
2. packages/core/src/evaluation/providers/copilot-sdk-log-tracker.ts — Log tracker pattern
3. packages/core/src/evaluation/providers/types.ts — All type definitions (ProviderKind, Provider, ProviderRequest, ProviderResponse, ToolCall, OutputMessage)
4. packages/core/src/evaluation/providers/targets.ts — Config resolution pattern (CopilotSdkResolvedConfig, resolveCopilotSdkConfig(), resolveTargetDefinition() switch)
5. packages/core/src/evaluation/providers/index.ts — Provider factory (createProvider() switch)
6. packages/core/src/evaluation/validation/targets-validator.ts — Settings validation (COPILOT_SDK_SETTINGS, getKnownSettings())
7. packages/core/src/evaluation/providers/claude-code.ts — Current implementation to replace
8. packages/core/src/evaluation/providers/claude-code-log-tracker.ts — Current log tracker to replace
9. apps/cli/tsup.config.ts — External dependency configuration
10. packages/core/test/evaluation/providers/copilot-sdk.test.ts — Test pattern to follow

Step-by-step implementation

Step 1: Update packages/core/src/evaluation/providers/types.ts

• In ProviderKind union: replace 'claude-code' with 'claude'
• In KNOWN_PROVIDERS array: replace 'claude-code' with 'claude'
• In AGENT_PROVIDER_KINDS array: replace 'claude-code' with 'claude'
• In PROVIDER_ALIASES map: add 'claude-code': 'claude' and 'claude-sdk': 'claude' for backward compat
• Add SDK-specific optional fields to TargetDefinition if needed: maxBudgetUsd, maxTurns

Step 2: Update packages/core/src/evaluation/providers/targets.ts

• Add ClaudeResolvedConfig interface:

  export interface ClaudeResolvedConfig {
    model?: string;
    apiKey?: string;
    cwd?: string;
    workspaceTemplate?: string;
    maxTurns?: number;
    maxBudgetUsd?: number;
    systemPrompt?: string;
    timeoutMs?: number;
    logDir?: string;
    logFormat?: 'summary' | 'json';
  }

• Add resolveClaudeConfig() function following resolveCopilotSdkConfig() pattern exactly — use resolveString()/resolveOptionalString() for env var ${{ VAR }} syntax
• Add ResolvedTarget variant: { kind: 'claude'; config: ClaudeResolvedConfig; ... }
• Add case 'claude': in resolveTargetDefinition() switch calling resolveClaudeConfig()

Step 3: Create packages/core/src/evaluation/providers/claude.ts

Follow copilot-sdk.ts structure exactly:

• Lazy SDK loading (module-level cached function, never top-level import)
• Class ClaudeProvider implements Provider with kind = 'claude' as const
• invoke() method:
  1. Call loadSdk() lazily
  2. Build prompt via buildPromptDocument(request, inputFiles) from ./preread.js
  3. Resolve working directory via resolveCwd() pattern
  4. Call sdk.query({ prompt, options: { model, permissionMode: 'bypassPermissions', allowDangerouslySkipPermissions: true, cwd, systemPrompt, maxTurns, maxBudgetUsd, abortController } })
  5. Iterate async generator collecting messages:
     • message.type === 'assistant' → extract message.message.content blocks → find type: 'tool_use' blocks → map to ToolCall[] and OutputMessage[]
     • message.type === 'result' → extract message.usage (token counts), message.total_cost_usd, message.duration_ms, message.num_turns
  6. Build final OutputMessage[] array
  7. Return ProviderResponse { raw, outputMessages, tokenUsage: { input, output, cached }, costUsd, durationMs, startTime, endTime }
• Stream logging to .agentv/logs/claude/ directory (same pattern as copilot-sdk)
• Abort signal forwarding from request.signal

Step 4: Create packages/core/src/evaluation/providers/claude-log-tracker.ts

Copy copilot-sdk-log-tracker.ts and rename all references. Same global log entry tracking pattern.

Step 5: Update packages/core/src/evaluation/providers/index.ts

• Import ClaudeProvider from './claude.js'
• Add case 'claude': in createProvider() switch
• Export ClaudeResolvedConfig from targets
• Export log tracker functions
• Remove imports for claude-code.ts and claude-code-log-tracker.ts

Step 6: Update packages/core/src/evaluation/validation/targets-validator.ts

• Add CLAUDE_SETTINGS = new Set(['model', 'api_key', 'cwd', 'workspace_template', 'max_turns', 'max_budget_usd', 'system_prompt', 'timeout_ms', 'log_dir', 'log_format'])
• Add case 'claude': in getKnownSettings() returning CLAUDE_SETTINGS
• Remove old claude-code case and settings

Step 7: Update apps/cli/tsup.config.ts

Add '@anthropic-ai/claude-agent-sdk' to the external array.

Step 8: Delete old files

• packages/core/src/evaluation/providers/claude-code.ts
• packages/core/src/evaluation/providers/claude-code-log-tracker.ts
• packages/core/test/evaluation/providers/claude-code.test.ts (if exists)

Step 9: Create packages/core/test/evaluation/providers/claude.test.ts

Follow packages/core/test/evaluation/providers/copilot-sdk.test.ts pattern. Mock the SDK's query() function.

Step 10: Update references

• Update examples/features/.agentv/targets.yaml if it references claude-code → claude
• Update docs referencing the old provider name

Verification checklist

bun run typecheck
bun run lint
bun run test
bun run build