feat(coverage): wire 11 behavioral orphans (closes behavioral coverage) - #57
Merged
Conversation
Adds 11 behavioral checks covering the remaining behavioral-doable orphans. After this batch, the only orphans left are the 3 source-layer ones that are out of scope by current policy. Groups: - A. Error-battery (5 checks via shared bad-flag probe): `p2-must-exit-codes`, `p2-must-json-errors`, `p2-should-consistent-envelope`, `p4-must-actionable-errors`, `p4-should-json-error-output`. Probe is the unknown long flag `--this-flag-does-not-exist-anc`; JSON-specific MUSTs gate on the binary advertising `--output json` in `--help`. - B. Subcommand-help recursion (3 checks via shared helper): `p3-must-subcommand-examples`, `p3-should-paired-examples`, `p6-should-subcommand-operations`. Recursion caps at one level and skips built-in `help`/`completions` subcommands. - C. Destructive-op detection (2 checks via shared verb list): `p5-must-force-yes`, `p5-must-read-write-distinction`. Destructive verbs match substring; read/write verbs match exact name. - D. TTY-vs-pipe (1 check): `p7-may-auto-verbosity`. Uses the defensible cheap path: scan `--help` for TTY-aware language. Pty allocation is out of scope. Drops two `CheckResult` constructors out of `fn run` per the project's Source Check Convention. Dogfood test `tests/dogfood.rs` adds `p2-json-errors` to its temporary allowlist with a justification: `anc` itself uses clap's `parse()` rather than `try_parse()`, so it emits plain-text errors regardless of `--output json`. Honoring JSON mode for parse errors is separate work. Audit confirms: orphan count drops from 14 to 3, MUST orphans drop to zero, behavioral coverage is complete.
brettdavies
added a commit
that referenced
this pull request
Jun 1, 2026
## Summary `v0.5.0` of `anc`. Vendored `agentnative-spec` advances `0.4.0` → `0.5.0`. Scorecard JSON schema advances `0.5` → `0.7` in two steps (7-status taxonomy at `0.6`; `check_id` → `audit_id` rename at `0.7`). Coverage matrix JSON `schema_version` rebases from `"1.0"` to `"0.1"` so it matches the project's pre-release `0.x` convention. Two new universal P3 requirements (`p3-must-version`, `p3-should-version-short`) plus 22 additional behavioral checks across P1-P8 close the remaining behavioral orphan coverage. CLI subcommand surface renames `check` → `audit` and `generate` → `emit` (breaking). Badge eligibility floor lowers from 80% to 70%, and the leaderboard score becomes credit-weighted on shipped-binary behavior only. The Vale prose-check stack (`styles/`, `.vale.ini`, `scripts/prose-check.sh`) moves to dev-only contributor tooling; the workflow guard keeps it out of `main` going forward. ## Changelog ### Added - 22 new behavioral checks across P1-P8 close the remaining behavioral orphan coverage (verbose, color, raw, examples, defaults-in-help, rich-tui, about/long-about, stdin-input, consistent-naming, timeout-behavioral, structured exit codes, JSON error envelopes, actionable errors, force/yes, read/write distinction, TTY-aware verbosity, and more). - Two universal P3 requirements (`p3-must-version`, `p3-should-version-short`) verify that every CLI ships a `--version` flag plus a short alias from the `-V` / `-v` / `-version` family. - New `opt_out` and `n_a` scorecard statuses surface explicit non-adoption and unmet-antecedent rows; each `results[]` entry now carries `tier` plus `audit_id`. Pre-`0.6` consumers feature-detect. - Top-level `--verbose` / `-v` (env `AGENTNATIVE_VERBOSE`), `--examples`, `--color`, and `--raw` flags surface diagnostic detail, curated invocation blocks, ANSI styling control, and pipe-friendly `id<TAB>status` rows. - `anc skill install --all` and `anc skill update [host|--all]` iterate every known host in one invocation. ### Changed - **BREAKING**: `anc check` renamed to `anc audit`; `anc generate` renamed to `anc emit`; `anc schema` folded under `anc emit schema`. The implicit-default-subcommand injection now writes `audit`. Update scripts and CI invocations. - **BREAKING**: scorecard `schema_version` advances `"0.5"` → `"0.7"` (two-step), the `check_id` field renames to `audit_id`, and `results[]` switches to one entry per requirement-row rather than per probe. Consumers pinning the prior shape must feature-detect or update. - Leaderboard `score_pct` is now credit-weighted on shipped-binary behavior only: `warn` earns half credit, `opt_out` counts against, source/project audits no longer affect the score. Badge eligibility floor lowers from 80% to 70%. - Vendored `agentnative-spec` advances `0.4.0` → `0.5.0`; scorecard `spec_version` reports `"0.5.0"`. - `coverage/matrix.json` `schema_version` bumps `"1.0"` → `"0.1"` so the matrix follows the project's pre-release `0.x` convention used by the CLI crate, scorecard schema, and vendored spec. The lone consumer (`agentnative-site/src/build/coverage.mjs`) validates the field's existence but does not pin a specific value, so this is safe. - Vale prose-check stack (`styles/`, `.vale.ini`, `scripts/prose-check.sh`) repositions as dev-only contributor tooling. `guard-main-docs.yml` blocks those paths from `main` going forward. ### Fixed - Error output under `--output json` / `--json` now emits a JSON envelope with `error`, `kind`, `message` instead of clap's plain-text rendering, so agents pinned to JSON can parse failures with one shape. - Text mode now renders `[N/A]` (with antecedent evidence) for conditional requirements whose prerequisite was opted out, instead of a misleading `[FAIL]` on the probe id. Text row count and badge score match `--output json`. - The spec parser rejects three malformed conditional-applicability inputs that previously fell through silently (whitespace-only `antecedent.audit_id`, mixed legacy `if:` + new `kind:`, unknown keys inside `antecedent`). ### Documentation - `RELEASES-PREFLIGHT.md` lands as the operational pre-flight checklist gating each release cut. `RELEASES.md` gains a `### Cherry-pick conflicts on guarded paths` subsection and a `### Dev-direct exception` subsection covering engineering-docs paths and the prose-check stack. - Repo-wide prose pass: every tracked markdown doc plus the historical sections of `CHANGELOG.md` carry the same per-occurrence punctuation tightening applied to all 64 PR bodies in the v0.4.0 → v0.5.0 window. ## Type of Change - [x] `BREAKING CHANGE`: Breaking API change (requires major version bump) ## Related Issues/Stories - Story: n/a - Issue: n/a - Architecture: `agentnative-spec` `v0.5.0` (vendored at upstream commit `d5d4086`). Consumer-facing JSON shape moves to schema `0.7` with the `audit_id` rename. - Related PRs: cherry-picks of #55, #56, #57, #58, #59, #60, #61, #62, #63, #64, #65, #66, #68, #69 from `dev`. ## Testing - [x] All tests passing **Test Summary:** - `cargo test`: 793 passed, 2 ignored. - `anc emit coverage-matrix --check`: exit 0. - `cargo deny check advisories`: ok. - Release-mechanics preflight per `RELEASES-PREFLIGHT.md`: green except the cross-repo `agentnative-site` schema gap (tracked separately). ## Files Modified **Modified:** - Source: `src/audits/`, `src/scorecard/`, `src/principles/`, `src/main.rs`, `src/cli.rs`, `src/audit.rs`, `src/types.rs`, `src/runner/`, `src/source.rs`, `src/skill_install.rs`, and supporting modules. - Schema: `schema/scorecard.schema.json` regenerated to the `0.7` contract. - Vendored spec: `src/principles/spec/*` advances to `v0.5.0`. - Coverage artifacts: `docs/coverage-matrix.md`, `coverage/matrix.json` reflect 59 requirement rows. - Workflows: `.github/workflows/guard-main-docs.yml` (extra_paths). - Tracked docs: `README.md`, `AGENTS.md`, `CLAUDE.md`, `PRODUCT.md`, `BRAND.md`, `RELEASES.md`, `RELEASES-RATIONALE.md`, `scripts/SYNCS.md`, `.github/pull_request_template.md`, `CHANGELOG.md`. - Scripts: `scripts/generate-changelog.sh` (duplicate-section guard), `scripts/sync-spec.sh` (`--ref` flag and `gh api` transport), `scripts/hooks/pre-push` (shellcheck step). - Build identity: `Cargo.toml`, `Cargo.lock` (`0.5.0`). **Created:** - `RELEASES-PREFLIGHT.md` (operational pre-flight checklist). **Renamed:** - `anc check` → `anc audit` (subcommand surface). - `anc generate` → `anc emit` (subcommand surface). - Scorecard `check_id` → `audit_id` (JSON field). **Deleted:** - `.vale.ini`, `styles/`, `scripts/prose-check.sh`, plus the orphaned helpers `scripts/sync-prose-tooling.sh`, `scripts/test-prose-check.mjs`, `scripts/generate-pack-readme.mjs`. Replaced by the `guard-main-docs.yml` `extra_paths` guard that keeps the stack off `main` going forward. ## Breaking Changes - [x] Breaking changes described below: 1. **Scorecard JSON shape**: `schema_version` `"0.5"` → `"0.7"`. `check_id` renamed to `audit_id` (per-row and in `coverage/matrix.json`). `results[]` switches to one entry per requirement-row instead of per probe; each entry carries new `tier` and `audit_id` fields. Status enum expands from 3 to 7 values (`pass`, `warn`, `fail`, `opt_out`, `n_a`, `skip`, `error`). Consumers pinned to `0.5` or the prior `check_id` field must update. 2. **Subcommand surface**: `anc check` renamed to `anc audit`; `anc generate` renamed to `anc emit`; `anc schema` folded under `anc emit schema`. Scripts and CI invocations pinned to the old names must update. ## Deployment Notes - `agentnative-site` must add `'0.7'` to its `SUPPORTED_SCHEMA_VERSIONS` allowlist before this release is tagged. Without that, the site's regen pipeline rejects every `v0.5.0` scorecard upload. - After tag publish: run `./scripts/sync-dev-after-release.sh v0.5.0 && git push origin dev` per `RELEASES.md` § After publish to backport `Cargo.toml`, `Cargo.lock`, and `CHANGELOG.md` to `dev`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires 11 behavioral checks against the 11 remaining behavioral-doable orphan requirements. After this batch lands, the only orphans left will be the 3 source-layer ones declared out of scope by current policy (
p4-should-gating-before-network,p5-should-idempotency,p6-should-tier-gating).The work is grouped by probe shape so checks within a group share helpers and runner cache:
--this-flag-does-not-exist-anc) feeds five P2/P4 checks that observe exit codes, error messages, and JSON envelopes.--help(capped at one level, skipping built-inhelp/completions) and feeds three P3/P6 example/operation checks.--helpfor TTY-aware language since pty allocation is out of scope.Audit confirms orphan count drops from 14 to 3, MUST orphans drop to zero, behavioral coverage is complete.
Changelog
Added
Type of Change
feat: New feature (non-breaking change which adds functionality)Related Issues/Stories
Testing
Test Summary:
-Dwarningsclean, cargo-deny clean, Windows compat cleanCoverage matrix regenerated and drift-check clean (
anc generate coverage-matrix --checkexits 0).Files Modified
Modified:
src/checks/behavioral/mod.rs: register 11 new check structs plus 3 shared helper modules.tests/dogfood.rs: addp2-json-errorsto the temporary allowlist with a justification (anc itself uses clap'sparse()rather thantry_parse(), so it emits plain-text errors regardless of--output json; honoring JSON mode for parse errors is separate work).coverage/matrix.json: regenerated from the registry +covers()declarations.docs/coverage-matrix.md: regenerated table now showing the 11 newly-covered IDs.Created:
src/checks/behavioral/error_probe.rs: shared bad-flag probe + JSON applicability gate (Group A).src/checks/behavioral/structured_exit_codes.rs:p2-must-exit-codes.src/checks/behavioral/json_errors.rs:p2-must-json-errors(strict 3-key envelope).src/checks/behavioral/consistent_envelope.rs:p2-should-consistent-envelope.src/checks/behavioral/actionable_errors.rs:p4-must-actionable-errors.src/checks/behavioral/json_error_output.rs:p4-should-json-error-output.src/checks/behavioral/subcommand_help.rs: shared subcommand--helprecursion (Group B).src/checks/behavioral/subcommand_examples.rs:p3-must-subcommand-examples.src/checks/behavioral/paired_examples.rs:p3-should-paired-examples.src/checks/behavioral/subcommand_operations.rs:p6-should-subcommand-operations.src/checks/behavioral/destructive_ops.rs: shared destructive/read/write verb classification (Group C).src/checks/behavioral/force_yes.rs:p5-must-force-yes.src/checks/behavioral/read_write_distinction.rs:p5-must-read-write-distinction.src/checks/behavioral/auto_verbosity.rs:p7-may-auto-verbosity.Renamed:
Deleted:
Key Features
The dogfood pass against
ancitself surfaces real, defensible gaps that future PRs should close:p2-json-errors→ Fail (allowlisted as pending; clapparse()vstry_parse()).p3-subcommand-examples→ Fail (subcommands lack example blocks; clapafter_helpwould satisfy).p3-paired-examples,p4-json-error-output,p7-auto-verbosity→ Warn (room for improvement).These verdicts are the dogfooding loop working as designed: checks find real issues in the linter itself.