feat: evaluate multiple targets

[christso]

Problem

Prompts behave differently depending on the provider. For example, VS Code uses runSubagent too, while Copilot CLI uses task agent.
Also, Copilot CLI has GitHub MCP built-in by default, while VS Code has githubRepo tool built-in.

Proposal

Matrix Evaluation

In the eval YAML, allow users to define multiple targets and automatically run every case against every target, producing a comparison matrix:

name: cross-provider-eval
version: "1.0"

targets:
  - copilot-cli
  - vscode
  - claude

cases:
  - id: refund-request
    input: "Help me get a refund"
    evaluators:
      - type: llm_judge
        prompt: evaluators/helpful.md

Running this produces a targets × cases matrix:

                | copilot-cli | vscode | claude |
refund-request  |    0.9 ✓    | 0.85 ✓ | 0.7 ✗  |
greeting        |    1.0 ✓    | 1.0 ✓  | 0.95 ✓ |
edge-case       |    0.6 ✗    | 0.8 ✓  | 0.9 ✓  |

CLI Usage

# Run against all targets defined in the eval
agentv run evals/cross-provider.yaml

# Run against specific targets only
agentv run --target copilot-cli --target claude evals/

# Compare results across targets
agentv compare --by-target results.jsonl

Multi-Prompt Testing

Also support testing multiple prompt variants against the same cases:

prompts:
  - file://prompts/v1-concise.md
  - file://prompts/v2-detailed.md

targets:
  - copilot-cli

cases:
  - id: test-1
    input: "..."

This produces a prompts × cases (or prompts × targets × cases) matrix for A/B testing prompt versions.

Results Schema

Each result includes which target and prompt variant produced it:

{
  "caseId": "refund-request",
  "target": "copilot-cli",
  "prompt": "v1-concise",
  "score": 0.9,
  "pass": true
}

Why This Matters

• Cross-provider testing is essential for portable agent prompts
• Matrix view makes it easy to spot provider-specific regressions
• Prompt A/B testing helps optimize agent instructions
• The compare command already exists — this extends it naturally

Acceptance Criteria

• Multiple targets in eval YAML, all cases run against all targets
• Matrix output in terminal showing targets × cases
• --target flag to filter to specific targets
• Optional multi-prompt support for A/B testing
• Results include target and prompt metadata
• Documentation with comparison examples

[christso]

Agent-Ready Implementation Plan (Final)

Branch

git checkout -b feat/179-matrix-evaluation

Reference files to read first

1. apps/cli/src/commands/eval/run-eval.ts — Primary file to modify. Main eval runner loop
2. apps/cli/src/commands/eval/commands/run.ts — CLI flag definitions (cmd-ts). --target is currently option({ type: string }) with default 'default'
3. packages/core/src/evaluation/yaml-parser.ts — YAML parsing, how execution.target is loaded
4. packages/core/src/evaluation/providers/targets.ts — resolveTargetDefinition()
5. packages/core/src/evaluation/providers/targets-file.ts — readTargetDefinitions(), listTargetNames()
6. packages/core/src/evaluation/types.ts — EvalSuite, EvalCase, EvaluationResult
7. apps/cli/src/commands/eval/jsonl-writer.ts — Result output format

Design decisions

• execution.targets (plural, array) at both root and per-case level under execution
• --target flag accepts multiple values (additive, not breaking)
• Multi-prompt support is deferred — not in scope for initial PR

YAML schema

# Root level — default targets for all cases
execution:
  targets:
    - copilot
    - claude

cases:
  - id: general-test
    input: "Hello"
    # Runs against copilot AND claude (from root)

  - id: copilot-only-test
    input: "GitHub specific task"
    execution:
      targets:
        - copilot         # Override — only run against copilot

Precedence:

• execution.targets (plural, array) — multiple targets, new behavior
• execution.target (singular, string) — single target, existing behavior
• If both specified, targets takes precedence
• CLI --target values override YAML execution.targets

Step-by-step implementation

Step 1: Support targets array in YAML

Update YAML parsing in packages/core/src/evaluation/yaml-parser.ts:

• Parse execution.targets as string[] (optional) alongside existing execution.target
• If targets present, use it; else fall back to target (singular)
• Same at per-case level: case.execution.targets overrides root

Step 2: Update --target CLI flag to accept multiple values

In apps/cli/src/commands/eval/commands/run.ts, change from:

target: option({ type: string, long: 'target', defaultValue: () => 'default' })

to multioption (cmd-ts). Single --target copilot still works. Multiple --target copilot --target claude adds more. This is not a breaking change.

Step 3: Update eval runner for multi-target execution

In the runner loop:

const targetNames: string[] =
  cliTargets.length > 0 ? cliTargets :           // --target flags override
  suite.execution?.targets ??                      // YAML targets array
  [suite.execution?.target ?? 'default'];          // single target fallback

for (const targetName of targetNames) {
  const target = await resolveAndCreateProvider(targetDef, env);
  for (const evalCase of cases) {
    const result = await runCase(evalCase, target, evaluators);
    result.target = targetName;
    writer.addResult(result);
  }
}

Step 4: Add target field to results

Ensure each EvaluationResult includes which target produced it. JSONL output: one line per case-target pair.

Step 5: Add matrix output display

Create apps/cli/src/commands/eval/matrix-display.ts:

                  | copilot | claude |
refund-request    |  0.9 ✓  | 0.85 ✓ |
greeting          |  1.0 ✓  | 0.7 ✗  |

Display after eval completes when multiple targets were used. Terminal-only (informational).

Step 6: Add tests

• Test: single target in YAML → current behavior unchanged
• Test: execution.targets array → all cases run against all targets
• Test: per-case execution.targets overrides root
• Test: --target CLI flag overrides YAML targets
• Test: multiple --target flags work
• Test: results include target metadata

Step 7: Add example

Create examples/features/matrix-evaluation/evals/dataset.yaml.

Verification checklist

bun run typecheck
bun run lint
bun run test
bun run build