feat(sdk): defineConfig() with agentv.config.ts — typed configuration file

[christso]

Summary

Add support for an agentv.config.ts configuration file with a defineConfig() helper that provides type-safe project-level configuration with IDE autocomplete.

Motivation

AgentV currently uses environment variables and CLI flags for configuration. A typed config file provides:

• IDE autocomplete for all options
• Project-level defaults (provider keys, worker count, output format)
• Shareable team configuration
• Custom assertion/provider registration at startup

Evidence from research:

• nem035/agentevals: agentevals.config.ts with defineConfig() + Zod validation — proven ergonomic
• opencode: opencode.config.ts with typed config
• Mastra: TypeScript config objects, Zod-validated
• Vitest/Vite: defineConfig() pattern — industry standard for TS tooling

Proposed Design

Config File

// agentv.config.ts
import { defineConfig } from "@agentv/core";
import { defineAssertion } from "@agentv/eval";

const sentiment = defineAssertion({
  type: "sentiment",
  evaluate: async ({ input, output }) => ({
    score: analyzeSentiment(output),
    reason: "Sentiment analysis",
  }),
});

export default defineConfig({
  // Default provider settings
  providers: {
    default: "anthropic",
    anthropic: { model: "claude-sonnet-4-20250514" },
    openai: { model: "gpt-4.1" },
  },

  // Register custom assertions
  assertions: {
    sentiment,
  },

  // Default execution settings
  execution: {
    workers: 5,
    retries: 2,
  },

  // Output format
  output: {
    format: "jsonl",
    dir: "./results",
  },

  // Cost limits
  limits: {
    maxCostUsd: 10.0,
    maxDurationMs: 300_000,
  },
});

Discovery

AgentV looks for config in this order:

1. agentv.config.ts (project root)
2. agentv.config.js
3. .agentv/config.ts
4. CLI flags override config file values

defineConfig() Helper

import { z } from "zod";

const configSchema = z.object({
  providers: z.object({...}).optional(),
  assertions: z.record(z.any()).optional(),
  execution: z.object({...}).optional(),
  output: z.object({...}).optional(),
  limits: z.object({...}).optional(),
});

export function defineConfig(config: z.input<typeof configSchema>) {
  return configSchema.parse(config);
}

Acceptance Criteria

• defineConfig() exported from @agentv/core
• Auto-discovery of agentv.config.ts / agentv.config.js
• Zod validation of config with helpful error messages
• CLI flags override config file values
• Custom assertions registered in config are available in EVAL.yaml under assert:
• TypeScript type inference provides full autocomplete
• agentv init creates a starter config file
• Documentation with config file guide

Related

• #328 SDK naming and package strategy — naming decisions
• feat(sdk): assertion plugin registry — replace hardcoded switch/case with extensible registration #320 — Assertion plugin registry (assertions registered via config)

Research References

• nem035/agentevals — defineConfig() pattern
• opencode SDK patterns — typed config

[christso]

Implemented and merged via PRs #338, #339, #341. All acceptance criteria met (some nice-to-have items deferred — see #328 for details).