Showcase: strengthen export-screening CI gating (multi-sample runs + stability-aware thresholds)

[christso]

Goal

Improve the existing export-screening showcase to demonstrate more robust CI gating without requiring AgentV core changes.

This keeps the current PR scope contained while providing an actionable pattern users can adopt immediately.

Proposal

Extend examples/showcase/export-screening/evals/ci_check.ts with optional multi-sample evaluation and stability-aware gating.

New wrapper options (suggested)

• --samples N: run the eval N times (fresh eval invocation each time) and aggregate metrics across runs.
• --gate min|mean|p05|p10 (or similar): choose conservative gating strategy.
• --min-run-f1 X: require every run (or p05) to meet threshold.
• Optional: --max-stddev X or --max-variance X for the checked class.

Behavior

• Default behavior remains unchanged (single-run threshold gate), so existing docs continue to work.
• When --samples is provided:
  • wrapper runs bun agentv eval repeatedly (or expects multiple results files)
  • aggregates confusion matrices / per-class metrics
  • emits a stability-aware CI result JSON and exits non-zero on failure.

Why this belongs in a wrapper/example

• Export-screening already demonstrates ΓÇ£results.jsonl -> wrapper metrics -> CI exit codeΓÇ¥.
• Multi-run/stability primitives are proposed for core (see issue Stability-aware evaluation: multi-run sampling + variance/disagreement penalty aggregation #117), but users want robust CI gating now.
• This example is a concrete pattern that generalizes to other domains, even if the dataset is export risk.

Acceptance Criteria

• bun run ./evals/ci_check.ts --eval ./evals/dataset.yaml --samples 5 --threshold 0.95 --check-class High works.
• Output JSON includes per-run metrics + aggregate metrics + selected gating rule.
• Wrapper exits 0/1 deterministically based on selected gate.
• README updated with the new options and guidance on choosing --samples and gate types.

Related

• Stability-aware evaluation: multi-run sampling + variance/disagreement penalty aggregation #117 Stability-aware evaluation: multi-run sampling + variance/disagreement penalty aggregation

Related

• Stability-aware evaluation: multi-run sampling + variance/disagreement penalty aggregation #117 Stability-aware evaluation: multi-run sampling + variance/disagreement penalty aggregation
• Showcase: evaluator conformance harness (compatibility + consistency fixtures, CI gate) #119 Showcase: evaluator conformance harness (compatibility + consistency fixtures, CI gate)

[christso]

Scope update: The core multi-run primitive this showcase was designed to work around is now tracked in #214 (consolidated from #117). Once #214 lands, this showcase can be simplified to demonstrate the new trials config in a CI gating context rather than implementing its own multi-sample loop.

Keeping open as a lower-priority showcase item.

[christso]

Agent-Ready Implementation Plan (Final)

#214 (trials) has landed. Use the built-in execution.trials config instead of a manual multi-sample loop.

Branch

git checkout -b feat/118-export-screening-ci-gating

Reference files to read first

1. examples/showcase/export-screening/ — Existing showcase to extend
2. examples/showcase/export-screening/evals/ci_check.ts — Current CI gate wrapper
3. examples/features/trials/evals/dataset.yaml — Trials feature (use this pattern)
4. packages/core/src/evaluation/loaders/config-loader.ts — extractTrialsConfig() for trials API

Step-by-step implementation

Step 1: Read the current showcase

Understand what evals/ci_check.ts currently does (runs eval, reads results, computes metrics, exits 0/1).

Step 2: Update EVAL.yaml to use trials

execution:
  target: default
  trials:
    count: 5
    strategy: conservative
    cost_limit_usd: 2.00

Step 3: Simplify ci_check.ts

Since trials handles multi-run internally, the wrapper now focuses on:

• Reading trial results from JSONL output
• Applying gating logic (threshold, stability checks)
• Optional flags: --threshold, --check-class, --max-stddev
• Emitting CI result JSON
• Exiting 0/1

// Parse trial results
const results = readTrialResults(resultsPath);
const metrics = computeAggregateMetrics(results, checkClass);

// Gate logic
const passed = metrics.aggregateScore >= threshold;
if (maxStddev && metrics.stddev > maxStddev) passed = false;

console.log(JSON.stringify({ metrics, passed }, null, 2));
process.exit(passed ? 0 : 1);

Step 4: Update README

• Document the trials config usage
• Show example GitHub Actions workflow snippet
• Guidance on choosing trial count and gate strategy

Verification checklist

cd examples/showcase/export-screening
bun run evals/ci_check.ts --eval ./evals/dataset.yaml --threshold 0.95 --check-class High