feat: add execution metrics support to Pi Coding Agent provider

[christso]

Summary

The Pi Coding Agent provider should return execution metrics (tokenUsage, costUsd, durationMs) in its ProviderResponse so they can be used by evaluators and included in evaluation results.

Current State

The provider already captures some of this data but doesn't surface it:

• usage data is extracted from Pi's agent_end event and stored in metadata.usage on output messages
• Execution duration is tracked for logging but not returned
• Cost is not calculated

Proposed Changes

In packages/core/src/evaluation/providers/pi-coding-agent.ts:

1. Parse token usage from agent_end event

   // In extractOutputMessages or new function
   const usage = agentEndEvent.usage; // { input_tokens, output_tokens, ... }
   const tokenUsage = {
     input: usage.input_tokens,
     output: usage.output_tokens,
     cached: usage.cached_tokens, // if available
   };

2. Calculate duration

   const startTime = Date.now();
   // ... execute Pi ...
   const durationMs = Date.now() - startTime;

3. Return metrics in ProviderResponse

   return {
     raw: { ... },
     outputMessages,
     tokenUsage,
     durationMs,
     // costUsd: optional, requires pricing info
   };

Files to Update

• packages/core/src/evaluation/providers/pi-coding-agent.ts

Acceptance Criteria

• tokenUsage extracted from Pi's usage data and returned in response
• durationMs calculated from wall-clock execution time
• Metrics appear in evaluation results when using Pi provider
• Unit tests for metric extraction

Labels

enhancement, good-first-issue

[christso]

Agent-Ready Implementation Plan

Branch

git checkout -b feat/98-pi-execution-metrics

Reference files to read first

1. packages/core/src/evaluation/providers/pi-coding-agent.ts — Primary file to modify
2. packages/core/src/evaluation/providers/types.ts — ProviderResponse, ProviderTokenUsage
3. packages/core/src/evaluation/providers/copilot-sdk.ts — Reference for how other providers return metrics
4. packages/core/test/evaluation/providers/ — Test patterns

Step-by-step implementation

Step 1: Read the current Pi provider implementation

Understand how pi-coding-agent.ts currently:

• Spawns the Pi agent
• Parses events (look for agent_end event)
• Extracts output messages
• Returns the response

Step 2: Extract token usage from agent_end event

The issue states usage data is already extracted but stored in metadata.usage. Surface it:

// In the response building section:
const usage = agentEndEvent?.usage;
const tokenUsage: ProviderTokenUsage | undefined = usage ? {
  input: usage.input_tokens,
  output: usage.output_tokens,
  cached: usage.cached_tokens,  // if available
} : undefined;

Step 3: Add duration tracking

const startTime = new Date();
// ... execute Pi agent ...
const endTime = new Date();
const durationMs = endTime.getTime() - startTime.getTime();

Step 4: Return metrics in ProviderResponse

return {
  raw: rawResponse,
  outputMessages,
  tokenUsage,
  durationMs,
  startTime: startTime.toISOString(),
  endTime: endTime.toISOString(),
  // costUsd: optional — requires Pi pricing data
};

Step 5: Add unit tests

In packages/core/test/evaluation/providers/pi-coding-agent.test.ts:

• Test: tokenUsage extracted from agent_end event
• Test: durationMs calculated correctly
• Test: metrics included in ProviderResponse
• Test: missing usage data handled gracefully (tokenUsage = undefined)

Verification checklist

bun run typecheck
bun run lint
bun run test
bun run build

Notes

• This is a small, focused change — only modifying pi-coding-agent.ts
• Do NOT change the Pi Agent SDK provider (pi-agent-sdk.ts) — that's a separate provider
• Cost tracking is optional since Pi pricing may not be available
• The issue is labeled "good-first-issue" — keep the scope minimal