feat(core): add OpenAI Codex SDK provider replacing CLI subprocess

[christso]

Summary

Replace the current codex provider (CLI subprocess via spawn) with a new provider using the official @openai/codex SDK. This follows the same pattern as the copilot provider (PR #211) and the planned claude provider (#213).

Motivation

The current Codex provider spawns codex CLI as a subprocess and scrapes JSONL output. The SDK provides:

• Structured responses with typed messages
• Token usage from response metadata
• Cost tracking from SDK-reported usage
• Tool call extraction as structured data (not scraped text)
• Abort/timeout control via SDK options

Competitive Advantage

Like the Claude and Copilot SDK providers, this gives AgentV structured access to agent internals — tool calls, token usage, cost — enabling tool_trajectory, tool_call_f1, and execution_metrics evaluators to work out of the box. Most eval frameworks treat Codex as a text-in/text-out black box.

Implementation Plan

Follow the same pattern as copilot provider (PR #211):

Files to modify

1. types.ts - Update ProviderKind to add codex-sdk or alias existing codex
2. targets.ts - Update resolved config and target resolution
3. index.ts - Wire new provider
4. targets-validator.ts - Update settings validation

Files to create

1. codex-sdk.ts - New provider using @openai/codex SDK
   • Lazy-load SDK (same pattern as copilot/claude providers)
   • Extract structured tool calls → ToolCall[] and OutputMessage[]
   • Extract token usage, cost, duration from SDK response
2. codex-sdk.test.ts - Unit tests with mocked SDK

Files to update/delete

1. codex.ts - Keep as fallback or deprecate
2. codex-log-tracker.ts - Update for SDK events

Dependencies

• Add @openai/codex to packages/core/package.json
• Add to CLI tsup.config.ts external list

Key Design Decisions

• Naming: codex (canonical using SDK), old CLI provider available as codex-cli if needed
• Tool calls: Extract structured tool use data for trajectory evaluation
• Backward compat: Existing codex configs should work with the new provider

Supersedes

This supersedes #99 (execution metrics for Codex) — the SDK provider will return metrics natively.

References

• PR feat(core): add copilot-sdk provider #211 - copilot provider implementation (pattern to follow)
• feat(core): add claude provider using @anthropic-ai/claude-agent-sdk #213 - claude provider using Agent SDK (same pattern)
• Current implementation: packages/core/src/evaluation/providers/codex.ts

[christso]

Agent-Ready Implementation Plan (Final)

Branch

git checkout -b feat/236-codex-sdk-provider

Prerequisites

cd packages/core && bun add @openai/codex-sdk
cd apps/cli && bun add @openai/codex-sdk

Package name: @openai/codex-sdk (npm, source). Promptfoo also uses this package for their Codex SDK provider.

Reference files to read first

1. packages/core/src/evaluation/providers/copilot-sdk.ts — Primary pattern reference (PR feat(core): add copilot-sdk provider #211)
2. packages/core/src/evaluation/providers/copilot-sdk-log-tracker.ts — Log tracker pattern
3. packages/core/src/evaluation/providers/types.ts — Types: ProviderKind, Provider, ProviderResponse, ToolCall, OutputMessage
4. packages/core/src/evaluation/providers/targets.ts — Config resolution: CopilotSdkResolvedConfig, resolveCopilotSdkConfig()
5. packages/core/src/evaluation/providers/index.ts — Provider factory: createProvider() switch
6. packages/core/src/evaluation/validation/targets-validator.ts — Settings validation pattern
7. packages/core/src/evaluation/providers/codex.ts — Current CLI subprocess implementation to understand
8. packages/core/src/evaluation/providers/codex-log-tracker.ts — Current log tracker
9. packages/core/test/evaluation/providers/copilot-sdk.test.ts — Test pattern
10. apps/cli/tsup.config.ts — External deps config

SDK API (confirmed from npm/GitHub)

import { Codex } from '@openai/codex-sdk';

const codex = new Codex({
  env: { OPENAI_API_KEY: apiKey },
  config: { model: 'o4-mini' }
});

const thread = codex.startThread({
  workingDirectory: cwd,
  skipGitRepoCheck: true,
});

// Buffered response
const turn = await thread.run(prompt);
// turn.finalResponse — final text
// turn.items — array of items (tool calls, messages)

// Streaming response (preferred for tool call extraction)
const { events } = await thread.runStreamed(prompt);
for await (const event of events) {
  // event.type: "item.completed" | "turn.completed"
  // event.item — tool call or message item
  // event.usage — token usage
}

Step-by-step implementation

Step 1: Update packages/core/src/evaluation/providers/types.ts

• Keep 'codex' in ProviderKind (already exists) — SDK replaces CLI under the same kind
• In PROVIDER_ALIASES: add 'codex-cli': 'codex' for backward compat
• Verify 'codex' is in AGENT_PROVIDER_KINDS

Step 2: Update packages/core/src/evaluation/providers/targets.ts

• Modify existing CodexResolvedConfig to add SDK-specific fields:

  export interface CodexResolvedConfig {
    model?: string;
    apiKey?: string;       // OPENAI_API_KEY
    cwd?: string;
    workspaceTemplate?: string;
    timeoutMs?: number;
    logDir?: string;
    logFormat?: 'summary' | 'json';
  }

• Update resolveCodexConfig() to handle new fields
• Update the ResolvedTarget variant for 'codex'

Step 3: Rewrite packages/core/src/evaluation/providers/codex.ts

Replace CLI subprocess with SDK. Follow copilot-sdk.ts pattern:

• Lazy SDK loading: const { Codex } = await import('@openai/codex-sdk')
• Class CodexSdkProvider implements Provider with kind = 'codex' as const
• invoke() method:
  1. Lazy-load @openai/codex-sdk
  2. Build prompt via buildPromptDocument(request, inputFiles)
  3. Create new Codex({ env, config }) + codex.startThread({ workingDirectory })
  4. Use thread.runStreamed(prompt) — iterate events, collect tool calls + messages
  5. Extract ToolCall[] from event.item objects (tool_use items)
  6. Build OutputMessage[] from collected messages
  7. Extract token usage and cost from event.usage or turn result
  8. Return ProviderResponse { raw, outputMessages, tokenUsage, costUsd, durationMs, startTime, endTime }
• Ephemeral threads: create a new thread per invocation, discard after (same as Promptfoo pattern)
• Stream logging to .agentv/logs/codex/
• Abort signal forwarding

Step 4: Update packages/core/src/evaluation/providers/codex-log-tracker.ts

Update for SDK events if the log format changes. Same global tracking pattern.

Step 5: Update packages/core/src/evaluation/providers/index.ts

• Update import to use new SDK provider class
• Keep case 'codex': in createProvider() switch

Step 6: Update packages/core/src/evaluation/validation/targets-validator.ts

• Update CODEX_SETTINGS set to include SDK-specific fields

Step 7: Update apps/cli/tsup.config.ts

Add '@openai/codex-sdk' to the external array.

Step 8: Create/update packages/core/test/evaluation/providers/codex.test.ts

• Mock @openai/codex-sdk module
• Test: basic invocation, tool call extraction, token usage, cost, duration
• Test: error handling, timeout, abort signal
• Test: backward compat with existing codex configs

Verification checklist

bun run typecheck
bun run lint
bun run test
bun run build

Notes

• This supersedes feat: add execution metrics support to Codex provider #99 — SDK provider returns execution metrics natively
• Keep old CLI provider as fallback if needed (rename to codex-cli.ts), similar to how copilot-cli was handled in PR feat(core): add copilot-sdk provider #211
• Ephemeral threads per eval case (no thread reuse across cases)