Summary
On agentv@4.27.0, agentv eval <file> ... still works as an alias for agentv eval run <file> ..., but agentv eval --help only documents subcommands (run, assert, aggregate).
That mismatch is easy to misread as “run is mandatory”, which caused review confusion downstream when evaluating whether docs/examples should use:
agentv eval <file> ...
agentv eval run <file> ...
Repro
agentv --version
# 4.27.0
agentv eval --help
Current help shows:
agentv eval <subcommand>
> Evaluation commands
where <subcommand> can be one of:
- run
- assert
- aggregate
But this still executes the eval path directly:
agentv eval ./evals/some.eval.yaml --target llm
Expected
One of these should be made true and explicit:
-
If agentv eval <file> ... is supported intentionally
- document it in help as a supported shorthand / alias for
run
- ideally show both forms or mention that bare
eval dispatches to run
-
If run is meant to be the only supported form
- reject
agentv eval <file> ... with a clear error
- update any compatibility behavior accordingly
Why this matters
Right now the CLI behavior and the help text disagree. Humans reviewing docs or automation tend to trust --help, so they conclude that run is required even though the binary still accepts the no-subcommand form.
That ambiguity leaks into downstream docs, workflows, and PR reviews.
Summary
On
agentv@4.27.0,agentv eval <file> ...still works as an alias foragentv eval run <file> ..., butagentv eval --helponly documents subcommands (run,assert,aggregate).That mismatch is easy to misread as “
runis mandatory”, which caused review confusion downstream when evaluating whether docs/examples should use:agentv eval <file> ...agentv eval run <file> ...Repro
Current help shows:
But this still executes the eval path directly:
agentv eval ./evals/some.eval.yaml --target llmExpected
One of these should be made true and explicit:
If
agentv eval <file> ...is supported intentionallyrunevaldispatches torunIf
runis meant to be the only supported formagentv eval <file> ...with a clear errorWhy this matters
Right now the CLI behavior and the help text disagree. Humans reviewing docs or automation tend to trust
--help, so they conclude thatrunis required even though the binary still accepts the no-subcommand form.That ambiguity leaks into downstream docs, workflows, and PR reviews.