feat: assert-set evaluator with threshold logic (N% must pass)

[christso]

Problem

The composite evaluator supports weighted_average aggregation. There is no way to express "at least N% must pass" — a common quality gate pattern.

Proposal

Add threshold aggregation mode to the existing composite evaluator:

evaluators:
  - name: flexible_gate
    type: composite
    aggregator:
      type: threshold
      threshold: 0.5  # at least 50% of child evaluators must pass
    evaluators:
      - name: latency_check
        type: latency
        max_ms: 5000
      - name: cost_check
        type: cost
        max_usd: 0.01

Score = proportion of child evaluators that pass. Pass = score >= threshold.

Design Principles Alignment

• ✅ Lightweight Core — one new aggregation mode on existing type, not a new evaluator
• ✅ Non-Breaking Extension — existing weighted_average mode unchanged
• ✅ Built-ins for Primitives Only — threshold aggregation cannot be composed from existing primitives

Acceptance Criteria

• threshold aggregation mode on composite evaluator
• Score = fraction of passing children
• Pass/fail based on threshold value
• Unit tests
• Documentation

[christso]

Agent-Ready Implementation Plan (Final)

Branch

git checkout -b feat/235-threshold-aggregation-mode

Reference files to read first

1. packages/core/src/evaluation/evaluators/composite.ts — Primary file to modify. Contains current aggregation modes: weighted_average, code_judge, llm_judge
2. packages/core/src/evaluation/evaluators/types.ts — EvaluationScore, ChildEvaluatorResult, EvaluationVerdict ('pass' | 'fail' | 'borderline')
3. packages/core/src/evaluation/evaluators/scoring.ts — scoreToVerdict() thresholds: >= 0.8 pass, >= 0.6 borderline, < 0.6 fail
4. packages/core/test/evaluation/evaluators/composite.test.ts — Existing composite evaluator tests
5. examples/features/composite/evals/dataset.yaml — Current composite usage example

Step-by-step implementation

Step 1: Update aggregator type definition

In the types file where AggregatorConfig is defined:

Add 'threshold' to the aggregator type union:

type AggregatorType = 'weighted_average' | 'code_judge' | 'llm_judge' | 'threshold';

interface ThresholdAggregatorConfig {
  type: 'threshold';
  threshold: number;  // 0.0 to 1.0
}

Step 2: Implement threshold aggregation in composite.ts

Lenient with warning: borderline counts as passing, but warn when it contributes.

case 'threshold': {
  const threshold = config.aggregator.threshold;

  // Lenient: both 'pass' and 'borderline' count as passing
  const passingCount = childResults.filter(
    r => r.verdict === 'pass' || r.verdict === 'borderline'
  ).length;
  const borderlineCount = childResults.filter(r => r.verdict === 'borderline').length;
  const totalCount = childResults.length;
  const proportion = totalCount > 0 ? passingCount / totalCount : 0;
  const pass = proportion >= threshold;

  const borderlineWarning = pass && borderlineCount > 0
    ? ` (${borderlineCount} borderline)`
    : '';

  return {
    score: proportion,
    verdict: pass ? 'pass' : 'fail',
    reasoning: `${passingCount}/${totalCount} evaluators passed${borderlineWarning}. Threshold: ${(threshold * 100).toFixed(0)}%.`,
    hits: childResults
      .filter(r => r.verdict === 'pass' || r.verdict === 'borderline')
      .map(r => `${r.name}: ${r.verdict}`),
    misses: childResults
      .filter(r => r.verdict === 'fail')
      .map(r => `${r.name}: failed`),
    evaluatorResults: childResults,
  };
}

Step 3: Validate threshold config

Add Zod validation for the threshold aggregator:

const thresholdAggregatorSchema = z.object({
  type: z.literal('threshold'),
  threshold: z.number().min(0).max(1),
});

Merge into existing aggregator config schema.

Step 4: Add unit tests

In packages/core/test/evaluation/evaluators/composite.test.ts:

• Test: all children pass, threshold 0.5 → pass, score = 1.0
• Test: 2/4 children pass, threshold 0.5 → pass, score = 0.5
• Test: 1/4 children pass (verdict='pass'), threshold 0.5 → fail, score = 0.25
• Test: borderline child counts as passing (lenient)
• Test: warning message includes borderline count when borderline contributes to pass
• Test: no children pass, threshold 0.0 → pass (0 >= 0)
• Test: 3/4 children pass, threshold 1.0 → fail
• Test: evaluatorResults array included in output

Step 5: Add example

Create examples/features/threshold-evaluator/evals/dataset.yaml:

name: threshold-demo
version: "1.0"

cases:
  - id: quality-gate
    input: "Test input"
    evaluators:
      - name: quality_gate
        type: composite
        aggregator:
          type: threshold
          threshold: 0.5
        evaluators:
          - name: latency
            type: latency
            max_ms: 5000
          - name: quality
            type: llm_judge
            prompt: "Is this response helpful?"

Verification checklist

bun run typecheck
bun run lint
bun run test
bun run build