Showcase: evaluator conformance harness (compatibility + consistency fixtures, CI gate)

[christso]

Goal

Provide a reusable pattern (wrapper + fixtures) to validate evaluator/judge health:

• compatibility (can execute, returns valid structured output)
• consistency (stable on unambiguous fixtures across multiple runs)

This is intentionally NOT a core feature; it should be an example/showcase that teams can copy and adapt.

Motivation (concrete failures this catches)

• Provider/dependency/runtime misconfig causes evaluator to error (e.g., missing deps, unsupported runtime imports).
• LLM judge returns malformed JSON, missing fields, or out-of-range scores.
• Judge prompt/model change causes drift: previously PASS fixtures start failing (or vice versa).
• Stochastic judge flips decisions on clearly unambiguous cases.

Proposal

Add a new example/showcase (or extend an existing one) that includes:

• a small fixture dataset with labeled expectations:
  • unambiguous-pass cases (must always score 1)
  • unambiguous-fail cases (must always score 0)
  • ambiguous cases (allowed to vary, but bounded)
• a wrapper script that:
  • runs the evaluator N times per fixture (or runs agentv eval N times)
  • validates schema and bounds (compatibility)
  • computes flip-rate / agreement / variance (consistency)
  • exits 0/1 for CI gating

Suggested CLI flags (wrapper)

• --runs N
• --max-flip-rate X (for unambiguous fixtures, typically 0)
• --require-schema (default true)
• --output for structured CI JSON

Acceptance Criteria

• Can be used in CI to prevent shipping evaluator prompt/code changes that:
  • break output schema
  • introduce regressions on fixtures
  • increase instability beyond configured thresholds

Related

• Stability-aware evaluation: multi-run sampling + variance/disagreement penalty aggregation #117 Stability-aware evaluation: multi-run sampling + variance/disagreement penalty aggregation
• Showcase: strengthen export-screening CI gating (multi-sample runs + stability-aware thresholds) #118 Showcase: strengthen export-screening CI gating (multi-sample runs + stability-aware thresholds)

[christso]

Agent-Ready Implementation Plan

Branch

git checkout -b feat/119-evaluator-conformance-harness

Reference files to read first

1. examples/showcase/ — Existing showcases for directory/file conventions
2. examples/showcase/export-screening/ — Related CI gating showcase
3. packages/eval/src/index.ts — defineCodeJudge API
4. evals/ — Internal evals directory (the framework's own evals)

Step-by-step implementation

Step 1: Create showcase directory

examples/showcase/evaluator-conformance/
├── fixtures/
│   ├── unambiguous-pass.yaml    # Cases that must always score 1
│   ├── unambiguous-fail.yaml    # Cases that must always score 0
│   └── ambiguous.yaml           # Cases allowed to vary within bounds
├── harness/
│   └── conformance-check.ts     # Main conformance harness script
├── evals/
│   └── dataset.yaml             # Sample eval using the harness
└── README.md

Step 2: Create fixture files

fixtures/unambiguous-pass.yaml:

- id: obvious-pass-1
  input: "What is 2+2?"
  candidate_answer: "4"
  expected_score: 1
  tolerance: 0

- id: obvious-pass-2
  input: "Say hello"
  candidate_answer: "Hello! How can I help you today?"
  expected_score: 1
  tolerance: 0

fixtures/unambiguous-fail.yaml:

- id: obvious-fail-1
  input: "What is 2+2?"
  candidate_answer: "The meaning of life is 42"
  expected_score: 0
  tolerance: 0

- id: obvious-fail-2
  input: "Translate to French: Hello"
  candidate_answer: "SELECT * FROM users"
  expected_score: 0
  tolerance: 0

fixtures/ambiguous.yaml:

- id: ambiguous-1
  input: "Is this response good?"
  candidate_answer: "It depends on context..."
  expected_score: 0.5
  tolerance: 0.3  # Score can vary ±0.3

Step 3: Create harness/conformance-check.ts

#!/usr/bin/env bun

// Parse args: --runs N, --max-flip-rate X, --require-schema, --output
const args = parseArgs();
const runs = args.runs ?? 3;
const maxFlipRate = args.maxFlipRate ?? 0;

// Load fixtures
const passFixtures = loadFixtures('fixtures/unambiguous-pass.yaml');
const failFixtures = loadFixtures('fixtures/unambiguous-fail.yaml');
const ambiguousFixtures = loadFixtures('fixtures/ambiguous.yaml');

// Run evaluator N times per fixture
for (let run = 0; run < runs; run++) {
  for (const fixture of [...passFixtures, ...failFixtures, ...ambiguousFixtures]) {
    // Run the target evaluator against this fixture
    // Store result: { fixtureId, run, score, verdict, rawOutput }
  }
}

// Validate schema (all outputs have required fields: score, reasoning, etc.)
// Validate compatibility (all runs produced valid output, no crashes)
// Validate consistency (flip-rate on unambiguous fixtures = 0)
// Validate bounds (ambiguous fixture scores within tolerance)

// Output results
const report = {
  schema_valid: schemaCheckPassed,
  compatibility: allRunsSucceeded,
  consistency: {
    unambiguous_flip_rate: flipRate,
    max_allowed: maxFlipRate,
    passed: flipRate <= maxFlipRate,
  },
  fixtures: detailedResults,
};

console.log(JSON.stringify(report, null, 2));
process.exit(report.consistency.passed && report.compatibility ? 0 : 1);

Step 4: Create README

• Explain what the conformance harness tests (compatibility + consistency)
• Document fixture format
• Show how to add custom fixtures for your evaluator
• Provide example CI workflow

Verification checklist

cd examples/showcase/evaluator-conformance
bun run harness/conformance-check.ts --runs 3 --max-flip-rate 0

Notes

• This is a showcase/example — not core code
• The harness should be evaluator-agnostic — it tests ANY evaluator by running it against known fixtures
• Keep it simple: the harness runs agentv eval N times and checks results
• Users copy/adapt this pattern for their own evaluator testing