Skip to content

docs: sync CLI documentation with actual command surface#1296

Merged
notgitika merged 2 commits into
aws:mainfrom
arepageek:feature/docs
May 19, 2026
Merged

docs: sync CLI documentation with actual command surface#1296
notgitika merged 2 commits into
aws:mainfrom
arepageek:feature/docs

Conversation

@arepageek
Copy link
Copy Markdown
Contributor

Description

This PR brings the CLI documentation in line with the actual CLI surface. It does three things:

  1. Documents 5 stable commands that were not in docs/commands.md at allimport (with all 5 subcommands), add policy-engine, add policy, add runtime-endpoint, and telemetry (with enable/disable/status).
  2. Adds missing flags to commands that were already documented but out of datecreate, add agent, add evaluator, add online-eval, dev, and run eval.
  3. Fixes incorrect command names in docs/PERMISSIONS.mdrun batch-eval and run recommend were used where the actual CLI commands are run batch-evaluation and run recommendation.

After these changes, 100% of stable CLI commands are documented or mentioned in docs/ or README.md (preview commands are intentionally not yet documented).

Files changed

  • docs/commands.md — most of the work (new sections + flag updates)
  • docs/evals.md — kept in sync with commands.md for the add evaluator, run eval, and add online-eval tables
  • docs/PERMISSIONS.md — command name corrections

docs/commands.md — new sections

  • ### import — top-level flags + 5 subcommands (runtime, memory, evaluator, online-eval, gateway) with examples and per-subcommand flag tables
  • ### add policy-engine — Cedar policy engine creation, attach modes
  • ### add policy — Cedar policy authoring (file, inline statement, --generate from natural language)
  • ### add runtime-endpoint — named endpoints (version aliases) for runtimes
  • ### telemetry — anonymous usage analytics: enable, disable, status

docs/commands.md — flag updates to existing sections

  • create: clarified --name (it's the agent name, also used as project dir if --project-name is omitted), added --project-name, --session-storage-mount-path, --with-config-bundle, added AGUI to --protocol
  • add agent: added AGUI to --protocol, --request-header-allowlist, --session-storage-mount-path, --with-config-bundle
  • add evaluator: refactored to support both llm-as-a-judge (default) and code-based evaluator types via --type, --lambda-arn, --timeout, --kms-key-arn
  • add online-eval: added --endpoint <name>
  • dev: added -b, --no-browser and --no-traces
  • run eval: added --endpoint, -A, --assertion, --expected-trajectory, --expected-response
  • remove: added policy-engine, policy, and runtime-endpoint to the examples and the consolidated flag table
  • update: documented that update cli is the explicit form of update

docs/evals.md

Same add evaluator / run eval / add online-eval table updates as in commands.md, kept consistent across both.

docs/PERMISSIONS.md

  • Replace `run batch-eval` with `run batch-evaluation` (9 occurrences) — matches the real CLI command
  • Replace `run recommend` with `run recommendation` (5 occurrences) — matches the real CLI command
  • Table column alignment in the logging section is regenerated automatically by Prettier after the replacements

How I verified this

  • Built the CLI and parsed --help output for every command and subcommand to extract the real flag set.
  • Ran a script that scans every code example (agentcore <cmd> ...) across all docs and asserts every flag exists in the CLI — 0 unknown flags.
  • Ran a separate script that lists every CLI command and verifies it has either a dedicated section in docs/commands.md or a clear mention in another doc — 0 stable commands missing.
  • Preview commands (ab-test, config-bundle, archive, recommendations, promote, run batch-evaluation, run recommendation, stop batch-evaluation) were intentionally left out of this PR.

Related Issue

Closes #

Documentation PR

N/A — documentation changes are included in this PR.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Other (please describe):

Testing

How have you tested the change?

  • I ran npm run test:unit and npm run test:integ (unit tests: 3912/3916 — the 4 failures (DeployStatus, LogPanel, SecretInput) are preexisting on main and unrelated to this PR; verified they fail identically on upstream/main without my changes)
  • I ran npm run typecheck
  • I ran npm run lint (0 errors; 29 preexisting warnings, none in files touched by this PR)
  • If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots (no asset changes)

Checklist

  • I have read the CONTRIBUTING document
  • I have added any necessary tests that prove my fix is effective or my feature works (N/A — documentation-only change; verified via the audit scripts described above)
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed (every new section includes runnable examples)
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

- Replace `batch-eval` with `batch-evaluation` in PERMISSIONS.md for consistency
- Replace `recommend` with `recommendation` in PERMISSIONS.md for consistency
- Update table column alignment in PERMISSIONS.md logging section
- Standardize table formatting across commands.md and evals.md documentation
- Align CLI command naming conventions throughout documentation
Comment thread docs/commands.md Outdated
notgitika
notgitika previously approved these changes May 18, 2026
Copy link
Copy Markdown
Contributor

@notgitika notgitika left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thank you!

Per review feedback (aws#1296), the auto-prefix for bare header names
without 'X-' is the legacy 'X-Amzn-Bedrock-AgentCore-Runtime-Custom-',
not 'X-Custom-' as the table previously stated. The X-* pass-through
half was already correct.

Confirmed against normalizeHeaderName in src/cli/commands/shared/header-utils.ts
and the HEADER_ALLOWLIST_PREFIX constant in src/schema/schemas/agent-env.ts.
Copy link
Copy Markdown
Contributor

@notgitika notgitika left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@notgitika notgitika merged commit 5ae559a into aws:main May 19, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants