Summary
Follow-up improvements to the TypeScript SDK (#328) that improve discoverability and documentation for users extending AgentV. These were identified during acceptance criteria audits but deferred as non-blocking.
Items
1. Unified "Custom Assertions" guide
Source: #320 acceptance criteria
Current state: Documentation for custom assertions is spread across README.md, packages/eval/README.md, SKILL.md, and docs/evaluation/sdk.mdx. Each source covers part of the story.
Proposed: Create a dedicated docs page (docs/evaluators/custom-assertions.mdx) that consolidates:
- When to use
defineAssertion() vs defineCodeJudge()
- Convention-based discovery (
.agentv/assertions/)
- Pass/fail vs score patterns
- Testing custom assertions
- Full working example
2. Custom provider documentation
Source: #321 acceptance criteria
Current state: CLI provider (provider: cli + command_template) is documented, but there is no guide for users who want to implement native TypeScript providers via ProviderRegistry.register().
Proposed: Add a section to the SDK docs page covering:
- Provider interface contract
- How to register via
createBuiltinProviderRegistry().register()
- Example: wrapping an HTTP API as a provider
3. Convention-based provider discovery (.agentv/providers/)
Source: #321 acceptance criteria
Current state: Assertions support convention-based discovery from .agentv/assertions/*.ts, but providers do not have equivalent auto-discovery from .agentv/providers/. The agentv create provider scaffold creates files there, but they must be manually referenced.
Proposed: Mirror the discoverAssertions() pattern for providers — scan .agentv/providers/*.ts and auto-register them in the provider registry. This would complete the symmetry between assertion and provider extensibility.
Priority
Low — all items are documentation and DX polish. The SDK is fully functional without these.
Related
Summary
Follow-up improvements to the TypeScript SDK (#328) that improve discoverability and documentation for users extending AgentV. These were identified during acceptance criteria audits but deferred as non-blocking.
Items
1. Unified "Custom Assertions" guide
Source: #320 acceptance criteria
Current state: Documentation for custom assertions is spread across README.md,
packages/eval/README.md, SKILL.md, anddocs/evaluation/sdk.mdx. Each source covers part of the story.Proposed: Create a dedicated docs page (
docs/evaluators/custom-assertions.mdx) that consolidates:defineAssertion()vsdefineCodeJudge().agentv/assertions/)2. Custom provider documentation
Source: #321 acceptance criteria
Current state: CLI provider (
provider: cli+command_template) is documented, but there is no guide for users who want to implement native TypeScript providers viaProviderRegistry.register().Proposed: Add a section to the SDK docs page covering:
createBuiltinProviderRegistry().register()3. Convention-based provider discovery (
.agentv/providers/)Source: #321 acceptance criteria
Current state: Assertions support convention-based discovery from
.agentv/assertions/*.ts, but providers do not have equivalent auto-discovery from.agentv/providers/. Theagentv create providerscaffold creates files there, but they must be manually referenced.Proposed: Mirror the
discoverAssertions()pattern for providers — scan.agentv/providers/*.tsand auto-register them in the provider registry. This would complete the symmetry between assertion and provider extensibility.Priority
Low — all items are documentation and DX polish. The SDK is fully functional without these.
Related