feat: JUnit XML and JSON output formats

[christso]

Problem

AgentV outputs JSONL for results. CI/CD tools (GitHub Actions, Jenkins, GitLab CI) natively render JUnit XML. A single JSON file with aggregate stats enables programmatic quality gates.

Proposal

Add -o flag with format inference from file extension:

agentv run --target my-agent evals/ \
  -o results.jsonl \
  -o results.json \
  -o results.xml

Formats

• .jsonl — current default, unchanged
• .json — single object with aggregate stats + per-case results
• .xml — JUnit XML for CI tools

JSON Schema

{
  "evalId": "run-2026-02-19-001",
  "stats": {
    "total": 25,
    "passed": 22,
    "failed": 3,
    "pass_rate": 0.88
  },
  "results": [...]
}

JUnit XML

<testsuites name="agentv-eval" tests="25" failures="3">
  <testsuite name="eval-name">
    <testcase name="case-id" classname="eval-file"/>
  </testsuite>
</testsuites>

Why Not HTML?

Per design principle #1: "CLI wrappers that consume AgentV JSON/JSONL output for post-processing." HTML reports are a presentation concern. Build a separate viewer that reads JSONL/JSON output.

Design Principles Alignment

• ✅ Lightweight Core — two additional serialization formats, no UI
• ✅ Industry Standard — JUnit XML is the universal CI format
• ✅ Non-Breaking Extension — existing output unchanged, -o is additive

Acceptance Criteria

• -o flag with multiple outputs per run
• JSON output with stats
• JUnit XML compatible with GitHub Actions / Jenkins / GitLab
• Format inferred from extension
• Documentation

[christso]

Agent-Ready Implementation Plan

Branch

git checkout -b feat/232-junit-xml-json-output

Reference files to read first

1. apps/cli/src/commands/eval/jsonl-writer.ts — Current output writer (pattern to follow)
2. apps/cli/src/commands/eval/run-eval.ts — Main eval command, handles output file args
3. apps/cli/src/commands/eval/ — All files in eval command directory
4. packages/core/src/evaluation/types.ts — EvaluationResult shape
5. apps/cli/src/cli.ts — CLI argument parsing (cmd-ts)

Step-by-step implementation

Step 1: Add -o flag to CLI

In apps/cli/src/commands/eval/run-eval.ts (or wherever CLI args are defined):

• Add -o / --output flag that accepts multiple values
• Each value is a file path; format inferred from extension
• If no -o specified, default to current JSONL behavior (stdout or default file)

Using cmd-ts:

output: option({
  type: optional(multioption(string)),
  long: 'output',
  short: 'o',
  description: 'Output file(s). Format inferred from extension: .jsonl, .json, .xml',
}),

Step 2: Create JSON output writer

Create apps/cli/src/commands/eval/json-writer.ts:

import type { EvaluationResult } from '@agentv/core';

export class JsonWriter {
  private results: EvaluationResult[] = [];
  private evalId: string;

  constructor(evalId: string) {
    this.evalId = evalId;
  }

  addResult(result: EvaluationResult): void {
    this.results.push(result);
  }

  async write(filePath: string): Promise<void> {
    const stats = {
      total: this.results.length,
      passed: this.results.filter(r => r.score >= (r.threshold ?? 0.5)).length,
      failed: this.results.filter(r => r.score < (r.threshold ?? 0.5)).length,
      pass_rate: 0,
    };
    stats.pass_rate = stats.total > 0 ? stats.passed / stats.total : 0;

    const output = {
      evalId: this.evalId,
      stats,
      results: this.results.map(toSnakeCaseDeep),
    };

    await writeFile(filePath, JSON.stringify(output, null, 2));
  }
}

Step 3: Create JUnit XML output writer

Create apps/cli/src/commands/eval/junit-writer.ts:

import type { EvaluationResult } from '@agentv/core';

export class JUnitWriter {
  private results: EvaluationResult[] = [];

  addResult(result: EvaluationResult): void {
    this.results.push(result);
  }

  async write(filePath: string): Promise<void> {
    const total = this.results.length;
    const failures = this.results.filter(r => r.score < (r.threshold ?? 0.5)).length;

    // Group results by eval/dataset
    const suites = this.groupBySuite();

    let xml = `<?xml version="1.0" encoding="UTF-8"?>\n`;
    xml += `<testsuites name="agentv-eval" tests="${total}" failures="${failures}">\n`;

    for (const [suiteName, cases] of Object.entries(suites)) {
      const suiteFailures = cases.filter(c => c.score < (c.threshold ?? 0.5)).length;
      xml += `  <testsuite name="${escapeXml(suiteName)}" tests="${cases.length}" failures="${suiteFailures}">\n`;

      for (const c of cases) {
        const pass = c.score >= (c.threshold ?? 0.5);
        const time = c.durationMs ? (c.durationMs / 1000).toFixed(3) : undefined;
        xml += `    <testcase name="${escapeXml(c.caseId)}" classname="${escapeXml(suiteName)}"${time ? ` time="${time}"` : ''}>\n`;
        if (!pass) {
          xml += `      <failure message="${escapeXml(c.reasoning ?? 'Score below threshold')}"`;
          xml += ` type="EvalFailure">Score: ${c.score}</failure>\n`;
        }
        xml += `    </testcase>\n`;
      }

      xml += `  </testsuite>\n`;
    }

    xml += `</testsuites>\n`;
    await writeFile(filePath, xml);
  }
}

Critical: Properly escape XML entities (&, <, >, ", ').

Step 4: Create output dispatcher

Create apps/cli/src/commands/eval/output-writers.ts:

export function createOutputWriters(outputPaths: string[]): OutputWriter[] {
  return outputPaths.map(path => {
    if (path.endsWith('.jsonl')) return new JsonlWriter(path);
    if (path.endsWith('.json')) return new JsonWriter(path);
    if (path.endsWith('.xml')) return new JUnitWriter(path);
    throw new Error(`Unknown output format for: ${path}. Use .jsonl, .json, or .xml`);
  });
}

Step 5: Wire into eval runner

In apps/cli/src/commands/eval/run-eval.ts:

• Create writers from -o paths
• On each result: call writer.addResult(result) for all writers
• After eval completes: call writer.write() for aggregate writers (JSON, JUnit)
• JSONL writer writes per-result as before

Step 6: Add tests

• Test JSON output shape matches spec (evalId, stats, results)
• Test JUnit XML validates against JUnit schema
• Test multiple -o flags produce all formats
• Test format inference from extension
• Test unknown extension throws error

Verification checklist

bun run typecheck
bun run lint
bun run test
bun run build

Notes

• No external XML library needed — generate XML via string templating (it's a simple schema)
• Use toSnakeCaseDeep() from existing code for JSON output consistency
• JUnit XML spec reference: https://github.com/testmoapp/junitxml — test with GitHub Actions' test reporter