Skip to content

Clarify whether 'agentv eval <file>' is a supported alias for 'agentv eval run' #1233

@christso

Description

@christso

Summary

On agentv@4.27.0, agentv eval <file> ... still works as an alias for agentv eval run <file> ..., but agentv eval --help only documents subcommands (run, assert, aggregate).

That mismatch is easy to misread as “run is mandatory”, which caused review confusion downstream when evaluating whether docs/examples should use:

  • agentv eval <file> ...
  • agentv eval run <file> ...

Repro

agentv --version
# 4.27.0

agentv eval --help

Current help shows:

agentv eval <subcommand>
> Evaluation commands

where <subcommand> can be one of:
- run
- assert
- aggregate

But this still executes the eval path directly:

agentv eval ./evals/some.eval.yaml --target llm

Expected

One of these should be made true and explicit:

  1. If agentv eval <file> ... is supported intentionally

    • document it in help as a supported shorthand / alias for run
    • ideally show both forms or mention that bare eval dispatches to run
  2. If run is meant to be the only supported form

    • reject agentv eval <file> ... with a clear error
    • update any compatibility behavior accordingly

Why this matters

Right now the CLI behavior and the help text disagree. Humans reviewing docs or automation tend to trust --help, so they conclude that run is required even though the binary still accepts the no-subcommand form.

That ambiguity leaks into downstream docs, workflows, and PR reviews.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions