feat(scorecard): 7-status taxonomy, per-row emission, antecedent propagation (schema 0.6) - #62
Merged
Merged
Conversation
Pin-vendors agentnative-spec at commit b4f4d02 (PR brettdavies/agentnative#34 squash-merge on dev) as the basis for U2 of the scorecard fairness taxonomy plan (docs/plans/2026-05-21-001-feat-scorecard-fairness-taxonomy-plan.md in the site repo). The vendored tree carries U1: `applicability.kind: conditional` with `antecedent.check_id` shape on five requirement rows: `p2-must-schema-print` and `p2-should-schema-file` (antecedent `p2-json-output`), plus `p8-must-bundle-install`, `p8-may-install-all`, `p8-may-bundle-update` (antecedent `p8-bundle-exists`). Replaces the legacy `applicability.if: <prose>` shape on those rows. p8 prose re-expressed in if-X-then-Y construction. VERSION stays at 0.4.0 by design because the spec-side VERSION bump is deferred to spec's release PR per U1's status notes. The build pipeline already reads the vendored tree at `src/principles/spec/` via `build.rs`; downstream U2 work consumes the new shape from there. Resolved short SHA: b4f4d02 (printed by sync-spec.sh during vendoring; recorded here so the pin is traceable post-merge).
Extends the build-time spec parser, runtime registry, and matrix renderer to accept the new conditional applicability shape introduced by agentnative-spec PR #34 (vendored at b4f4d02 in the prior commit) while keeping the legacy `{ if: "<prose>" }` shape for the 18 spec rows that have not yet migrated. Schema change (parsed YAML, runtime enum, matrix.json output): - `Applicability::Conditional { condition: Option<String>, antecedent: Option<Antecedent> }` replaces the previous `Conditional(String)` tuple variant. At least one of `condition` or `antecedent` must be set; both is permitted (the new shape supplements the legacy prose with a machine-readable check id). - `Antecedent { check_id }` is the v1 machine-readable form. Compound antecedents (`all_of` / `any_of`) are deferred to a future schema bump per plan Sub-decision 2b. - The matrix.json row's `applicability` block uses `#[serde(skip_serializing_if = "Option::is_none")]` so legacy rows continue to emit `{ "kind": "conditional", "condition": "<prose>" }` and new rows emit `{ "kind": "conditional", "antecedent": { "check_id": "<id>" } }`. No site renderer changes required for the legacy path. The parser rejects four error modes loudly: `kind:` values other than `conditional`; `antecedent:` mappings missing `check_id`; `kind: conditional` with neither `condition:` nor `antecedent:` set; `condition:` values that are not non-empty strings. Each error names the offending file, requirement id, and the malformed field so build failures are actionable. This commit is the foundation for U2 of the scorecard fairness taxonomy plan (docs/plans/2026-05-21-001-feat-scorecard-fairness-taxonomy-plan.md in the site repo). Per-row emission, antecedent status propagation, and the schema 0.6 bump that consume this new shape land in the following commits on this branch.
Implements the bulk of U2 from the scorecard fairness taxonomy plan (docs/plans/2026-05-21-001-feat-scorecard-fairness-taxonomy-plan.md). Three changes ship together because they share the same pattern-match exhaustiveness contract on `CheckStatus`: the new variants, the per-row pipeline that emits them, and the test fixtures that pin the JSON shape. **7-status taxonomy (Decision 1).** `CheckStatus` gains `OptOut(String)` for deliberate non-adoption and `NotApplicable(String)` for conditional rows whose antecedent is unmet. JSON serializes as `opt_out` and `n_a` respectively. Every match site (`build_summary`, `score_pct`, `exit_code`, `format_text`, `format_text_raw`, `CheckResultView::from_row`) now handles all seven variants exhaustively. The plan's transitional `score_pct` formula keeps the historic `pass / (pass + warn + fail)` shape and excludes opt_out from the denominator and n_a from both sides (per U2 work item: "minimal change that respects the new semantics without committing to a new formula"); the final formula is the U3 spec issue. **Per-row emission (Decision 2c).** New `fan_out_per_row(raw, catalog)` walks each probe-level `CheckResult` and emits one row per requirement in `Check::covers()`, replacing the row `id` with the requirement id and carrying the probe's check id forward as provenance. A probe like `p3-version` (covers `p3-must-version` + `p3-should-version-short`) now produces two rows in `results[]`; a probe whose check is not yet wired into the registry passes through as a single row keyed by check id. `CheckResultView` gains `tier` (registry-looked-up `must` / `should` / `may`, or `null` for unknown ids) and `check_id` (probe provenance). The shape change drives the schema bump from 0.5 to 0.6; pre-0.6 consumers feature-detect. **Antecedent propagation (Decision 2a).** New `propagate_antecedents(rows, raw)` reads each row's registry `Applicability` and applies the propagation table to conditional rows: `opt_out` / `n_a` antecedent collapses the consequent to `n_a`; `skip` and `error` inherit; `pass` / `warn` / `fail` leave the consequent untouched (the probe's own status stands). Evidence text on the rewritten row cites the antecedent check id and its status so the provenance is legible in the scorecard JSON. `Summary` gains `opt_out` and `n_a` counters alongside the historic five so consumers can read the new buckets without scanning `results[]`. The badge derivation reads the per-row vector (post-propagation) so the embed URL the JSON emits and the post-summary text hint stay in lock-step. The audience classifier and coverage_summary continue to read raw probe results because signal classification keys on check ids and coverage counts requirements covered by the underlying probes. Test updates pin the new shape: 14 new scorecard-module tests cover fan-out for single-row and multi-row probes, the full propagation table (pass/warn/fail/opt_out/n_a/skip/error antecedents), score_pct exclusions, summary counting, and the `tier` / `check_id` round-trip; integration tests update the suppression test to look up the row id under per-row emission; the schema_v05 drift guard adds 0.6 keys (summary counters, per-row `tier`/`check_id`) so consumers' shape contract is checked end-to-end.
…rtifacts Wires the two probe-level cases where the 7-status taxonomy's `opt_out` carries clear semantics, then regenerates the committed schema artifacts so the JSON Schema, coverage matrix, and live scorecard agree on the shape introduced in the previous commits on this branch. **Probe-level opt_out emissions.** - `p8-bundle-exists` (project layer): the "no top-level `AGENTS.md` / `SKILL.md` found" branch now returns `OptOut` instead of `Warn`. A bundle that exists but is malformed (missing YAML frontmatter or `name:` field) is a real SHOULD violation and stays `Warn`. The opt_out signal feeds antecedent propagation: every conditional row whose antecedent is `p8-bundle-exists` (`p8-must-bundle-install`, `p8-may-install-all`, `p8-may-bundle-update`) collapses to `n_a` automatically when the project ships no bundle. - `p2-json-output` (behavioral layer): the two "no `--output` / `--format` flag detected" branches (top-level help and per-subcommand probe) now return `OptOut`. The conditional rows `p2-must-schema-print` and `p2-should-schema-file` are antecedent-gated on `p2-json-output`, so a tool without structured output sees those rows propagate to `n_a` rather than counting twice against the score. The probe-level changes are deliberately scoped to the two cases the plan calls out by name; other probes that currently emit `Skip` for "feature absent" are left as-is for follow-up work (the propagation table handles them correctly through whatever conditional antecedent they sit under, when the antecedent gets a machine-readable shape via `sync-spec`). **Regenerated artifacts.** - `schema/scorecard.schema.json` bumps `$id` to `https://anc.dev/scorecard-v0.6.schema.json`. The `CheckResultView` definition adds `tier` (enum of `must` / `should` / `may` / `null`) and `check_id`; the `Summary` definition adds `opt_out` and `n_a` counters; the `status` enum gains `opt_out` and `n_a` values; the example block updates to a per-row row with `tier` + `check_id` populated. The Summary's `required` list now names all seven counters. - `coverage/matrix.json` and `docs/coverage-matrix.md` regenerate against the conditional-applicability schema landed in the spec-scaffolding commit. The five conditional rows in p2 and p8 surface with `applicability.kind: conditional` and `applicability.antecedent.check_id` populated; legacy rows continue to emit `applicability.condition: "<prose>"` until each prerequisite grows a check id in the verifier catalog. Dogfood verification against `--command echo` (a CLI with no `--output` flag and no project context) produces a scorecard with `summary.opt_out: 1` for the `p2-must-output-flag` row (direct emission) and `summary.n_a: 1` for the `p2-must-schema-print` row (antecedent-propagated). Against the agentnative-cli repo itself (which has both `--output` and `AGENTS.md`), nothing routes through opt_out or n_a; the score stays at the v0.4.0 dogfood baseline.
Twenty-two adversarial tests across the four U2 surfaces, plus three parser strictness fixes the red team surfaced.
**Parser strictness fixes (`build_support/parser.rs`).** Three error modes that the original v1 parser accepted silently now hard-fail with a hint at the schema the author probably meant:
- `antecedent.check_id` is now trimmed before the non-empty check (whitespace-only strings were previously accepted as a valid id).
- Mixed legacy `if:` and new `kind:` in the same applicability block now errors. The legacy branch only fired for `map.len() == 1`, so the original parser silently dropped the `if:` prose when both keys were present, which would footgun any author mid-migration.
- Extra keys inside `antecedent` (beyond `check_id`) now error by name. Compound antecedents (`op: any_of | all_of`) are explicitly deferred to a future schema bump per plan Sub-decision 2b; silently ignoring v2 keys on a v1 row would let v2 syntax ship under v1 semantics.
**Parser red team (`tests/build_parser.rs`, +8 tests).** Whitespace-only `check_id`, `antecedent:` as a string, `antecedent:` as a list (the v2 compound shape), `kind: Conditional` with the wrong case, mixed legacy + new shape in one row, antecedent with `op:` smuggled alongside `check_id`, `antecedent: null`, empty antecedent mapping. Plus an `emit_rust` defense-in-depth test asserting that hostile characters in a `check_id` (quotes, backslashes) escape correctly in the generated Rust source.
**Registry consistency red team (`src/principles/registry.rs`, +2 tests).** Walks `REQUIREMENTS` and asserts every conditional row's `antecedent.check_id` resolves to a real check in `all_checks_catalog()`, because a typo or rename would silently mute propagation in production. A second test asserts no conditional row names its own covering check as its antecedent (the "self-gate" edge case where a row would gate its status against itself).
**Propagation red team (`src/scorecard/mod.rs`, +5 tests).** Idempotency (a second pass produces identical output), no-op when the antecedent didn't run (source-only or filtered run), `--audit-profile` suppression of the antecedent propagates as Skip with the suppression reason preserved, full pipeline end-to-end (fan-out → propagation → summary → score with opt_out + n_a counted separately).
**Score formula red team (`src/scorecard/mod.rs`, +4 tests).** Score is 0 with no panic when every row is opt_out or n_a (division-by-zero guard). One pass amid 999 n_a rows is 100% (n_a excluded from both numerator and denominator). Skip + Error continue to be excluded under the new taxonomy. `summary.total` equals the sum of all seven per-status counters (catches a future status variant added without updating `build_summary`).
**Serialization red team (`src/scorecard/mod.rs`, +2 tests).** Evidence strings containing quotes, backslashes, control chars (`\n`, `\t`, `\u{0007}`) and Unicode (zero-width joiner, RTL override) round-trip through serde → parse → assert without loss or escape corruption. Sanitization belongs at the render layer, not here; the scorecard's job is faithful pass-through.
**Schema drift red team (`tests/scorecard_schema_v05.rs`, +6 tests).** The committed `schema/scorecard.schema.json` is the consumer contract for the site renderer and third-party leaderboards. Tests assert: `$id` pins to the current `SCHEMA_VERSION` (0.6); the `status` enum lists all seven taxonomy values; `Summary.required` includes `opt_out` + `n_a`; `CheckResultView` includes `tier` (with the three RFC 2119 levels plus null) + `check_id`; the schema's own `examples[0]` block satisfies its own `required` lists (catches doc-vs-schema drift); a live scorecard from a real `anc audit` run carries every key in the top-level and per-row `required` lists.
The single supporting production change beyond the parser strictness is `#[derive(Debug)]` on `Summary` so red-team assertions can surface counter mismatches inline rather than panicking with a useless message.
…st-revised dates Pulls the date-discipline fix from agentnative-spec PR #38 (`fix(principles): enforce last-revised discipline + backfill stale dates`). Seven principle files refresh their `last-revised` frontmatter to match the date their content actually changed; p3 was already correct and stays as-is. The applicability semantics already vendored at `b4f4d02` (U1 of the scorecard fairness taxonomy plan) carry through unchanged; this is a frontmatter-only update. VERSION still reads `0.4.0` because the upstream spec defers the cut to a release PR; the resync is faithful. No source changes here; the regenerated `REQUIREMENTS` slice via `build.rs` is byte-identical to the prior build because the parser does not consume `last-revised`. Resolved short SHA: 1fb49fe (printed by sync-spec.sh during vendoring).
13 tasks
brettdavies
added a commit
that referenced
this pull request
May 28, 2026
## Summary The `--output text` view, its badge, and the process exit code were computed from raw probe results, while `--output json` already ran the schema-0.6 per-row + antecedent-propagation pipeline. So the terminal printed probe ids (`p2-schema-print`) instead of requirement-row ids (`p2-must-schema-print`), could never render `[N/A ]` (only propagation produces `n_a`), and its row count, badge score, and exit code could disagree with the JSON the site and scorer consume. A human auditing `bat` saw `[FAIL] … (p2-schema-print)` for a MUST the tool is actually exempt from. This routes all three text-side surfaces through the same pipeline. The fan-out + propagation pair that `build_scorecard` inlined is extracted into `build_row_results`, and `main::run` derives one per-row set that feeds `format_text`, `compute_badge`, and `exit_code`. Text and JSON now agree on the row set, counts, badge score, and exit code. The JSON path is byte-identical and the schema stays `0.6`. ## Changelog ### Changed - `anc audit` (text mode) now reports the same requirement rows as `--output json`: requirement-row ids (`p2-must-schema-print`) instead of probe ids, with the requirement tier (`must`/`should`/`may`) shown on each row. - The process exit code now reflects the per-row result set in both output modes. A requirement that collapses to `n_a` because its conditional prerequisite is `opt_out` no longer forces a non-zero exit; only a live `Fail`/`Error` exits `2` and a live `Warn` exits `1`. ### Fixed - Text mode now renders `[N/A ]` (with antecedent evidence) for conditional requirements whose prerequisite was opted out, instead of showing a misleading `[FAIL]` on the probe id. Text row count and badge score now match `--output json`. ## Type of Change - [x] `fix`: Bug fix (non-breaking change which fixes an issue) ## Related Issues/Stories - Story: n/a - Issue: n/a - Architecture: `docs/plans/2026-05-27-002-fix-text-renderer-per-row-parity-plan.md` - Related PRs: #62 (per-row pipeline + schema 0.6, JSON path only) ## Testing - [x] Unit tests added/updated - [x] Integration tests added/updated - [x] Manual testing completed - [x] All tests passing **Test Summary:** - New unit tests: `build_row_results_fans_out_then_propagates`, `exit_code_drops_to_zero_when_consequent_propagates_to_n_a`, `text_and_json_agree_on_a_propagated_conditional_row` (bat-shaped fixture asserting consequent shows `n_a` not `fail`, row counts match, badge scores match). - New integration test: `test_text_and_json_agree_on_row_count_and_exit_code` audits `ls` in both modes and pins text row count == JSON `results.len()` and exit-code parity, catching any revert that feeds the text renderer raw probe results. - Existing `format_text_*` / `format_text_raw` / R4 audit-profile masking tests pass unchanged. ## Files Modified **Modified:** - `src/main.rs`: derive one per-row projection after the principle filter; feed it to `format_text`, `compute_badge`, and `exit_code`. - `src/scorecard/mod.rs`: add `build_row_results`; `build_scorecard` calls it; `format_text` renders the tier suffix; consolidate the `Level`→token mapping behind `Level::as_str`. - `src/principles/registry.rs`: add `Level::as_str` as the single source of truth for the `must`/`should`/`may` token. - `tests/integration.rs`: text/JSON row-count + exit-code parity guard. - `docs/plans/2026-05-27-002-fix-text-renderer-per-row-parity-plan.md`: status `proposed` → `completed`. **Created:** - None. **Renamed:** - None. **Deleted:** - None. ## Breaking Changes - [x] No breaking changes The CLI surface and JSON schema are unchanged. The exit-code semantics shift for one edge: a tool that raw-Fails a check whose requirement row propagates to `n_a` now exits `0`/`1` instead of `2`. That is the corrected behavior (the requirement does not apply), it matches what the JSON `summary` already reported, and `anc` is `0.x` pre-launch. The `--audit-profile` masking path (plan R4) is independent and unchanged. ## Deployment Notes - [x] No special deployment steps required ## Post-Deploy Monitoring & Validation No additional operational monitoring is required because `anc` is a local CLI with no runtime service. The only downstream consumer is `agentnative-site`, which reads the JSON scorecard; the JSON path is byte-identical (schema stays `0.6`), so there is no consumer impact. Spot-check after merge: `anc audit --command bat` (text) shows `[N/A ] … (p2-must-schema-print) (must)` and no `[FAIL] … (p2-schema-print)`, and `anc audit --command bat` / `--output json` agree on exit code. ## Checklist - [x] Code follows project conventions and style guidelines - [x] Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/) - [x] Self-review of code completed - [x] Tests added/updated and passing - [x] No new warnings or errors introduced - [x] Changes are backward compatible (or breaking changes documented)
5 tasks
brettdavies
added a commit
that referenced
this pull request
May 29, 2026
#64) ## Summary Implements U3b of the scorecard fairness taxonomy plan: the CLI now computes the final leaderboard score defined in `agentnative-spec` `principles/scoring.md` (authority: `agentnative-spec@972c9d3`), replacing the transitional formula shipped in U2. The score now reflects shipped-binary behavior only. `score_pct` is restricted to behavioral-layer rows; source- and project-layer checks still emit in the scorecard but no longer move the score or badge eligibility. Over the denominator set `D = {pass, warn, fail, opt_out}`, the score is a credit-weighted ratio (`pass` 1.0, `warn` 0.5, `fail` and `opt_out` 0.0) with `n_a` / `skip` / `error` excluded from `D`. `opt_out` now counts in the denominator so deliberate non-adoption is reflected. Tier weights are published flat (MUST = SHOULD = MAY = 1) but encoded as named constants in the general `Σ w·credit / Σ w` form, so a future non-flat re-tune is a one-line change. The badge eligibility floor drops from 80% to 70%. A consequence of the behavioral-only scope: `anc audit . --source` now scores 0% and ineligible by design, because a source-only run has no behavioral rows to score. anc dogfoods to 100% in `--binary` and full mode (34 of 34 behavioral checks pass; 9 behavioral skips are excluded from the denominator). `Cargo.toml` version, `CHANGELOG.md`, and the tag publish are deferred to the release PR that closes out this unit, per the plan. ## Changelog ### Changed - The leaderboard score now reflects shipped-binary behavior only: source- and project-layer checks no longer affect `score_pct` or badge eligibility. - `score_pct` is now credit-weighted: `warn` earns half credit and `opt_out` counts against the score, replacing the prior pass-only ratio. - Lower the agent-native badge eligibility floor from 80% to 70%. ## Type of Change - [x] `feat`: New feature (non-breaking change which adds functionality) ## Related Issues/Stories - Story: U3b in `agentnative-site/docs/plans/2026-05-21-001-feat-scorecard-fairness-taxonomy-plan.md` - Issue: None. - Architecture: Formula authority is `agentnative-spec@972c9d3` `principles/scoring.md`. - Related PRs: brettdavies/agentnative#39 (spec formula, U3); #62 (7-status taxonomy, U2). ## Testing - [x] Unit tests added/updated - [x] All tests passing **Test Summary:** - Unit + integration tests: 793 passing, 2 ignored, across 7 suites. - Added: warn half-credit, behavioral-only exclusion, and the `scoring.md` worked example (20 pass / 7 warn / 0 fail / 1 opt_out / 1 n_a / 14 skip → 84). Reworked the at-floor (now 70%), opt_out-in-denominator, and full-pipeline propagation cases. - `cargo fmt --check`, `cargo clippy --all-targets --all-features -D warnings`, and `cargo test` all green. ## Files Modified **Modified:** - `src/scorecard/mod.rs`: rewrote `score_pct` (behavioral-only, credit-weighted, tier-weighted); added `tier_weight` and `status_credit` helpers; lowered `BADGE_ELIGIBILITY_FLOOR_PCT` to 70; updated badge docstrings and tests. - `README.md`: rewrote the Scoring section (7 statuses, credit formula, behavioral-only scope, 70% floor, cohort bands), the dogfood table, and the badge field reference. - `CLAUDE.md`: updated the scorecard `badge` reference to the new formula. **Created:** - None. **Renamed:** - None. **Deleted:** - None. ## Breaking Changes - [x] No breaking changes Scorecard schema stays at `0.6`; the change is in how `score_pct` is computed, not in the JSON shape. Consumers reading `badge.score_pct` see different values after rescore but no shape change. ## Deployment Notes - [x] No special deployment steps required The public registry rescore against this formula is U6 in the site repo, gated on a published CLI release, not part of this PR.
Merged
13 tasks
brettdavies
added a commit
to brettdavies/agentnative-site
that referenced
this pull request
May 29, 2026
) ## Summary Site-side renderer support for the scorecard fairness taxonomy (plan unit U4). The site can now render the schema 0.6 seven-status taxonomy and load 0.6 scorecards alongside the existing 0.5 corpus during the migration window. The live corpus stays 0.5 until the registry rescores against the 0.6 CLI (plan unit U6), so this change is additive: 0.5 scorecards render exactly as before, and 0.6 scorecards gain the two new statuses. The two new statuses are `opt_out` (deliberate non-adoption, counts against the score) and `n_a` (conditional antecedent unmet, excluded from the score). Both now render distinct from `skip` on the per-tool scorecard page and in the markdown twin. ## Changelog ### Added - Render the schema 0.6 seven-status taxonomy on scorecard pages: `opt_out` shows as `OPT-OUT` and `n_a` as `N/A`, each visually distinct from `skip`. The site now loads schema 0.5 and 0.6 scorecards side by side. ## Type of Change - [x] `feat`: New feature (non-breaking change which adds functionality) ## Related Issues/Stories - Story: n/a - Issue: n/a - Architecture: docs/plans/2026-05-21-001-feat-scorecard-fairness-taxonomy-plan.md (unit U4) - Related PRs: brettdavies/agentnative-cli#62 (emits the schema 0.6 shape this consumes) ## Testing - [x] Unit tests added/updated - [x] Integration tests added/updated - [x] Manual testing completed - [x] All tests passing **Test Summary:** - Unit tests: 765 passing (full `bun test` suite) - Integration tests: covered within the same suite (build + worker) - Coverage: n/a (no coverage gate configured) Notes: the 0.6 test fixture was validated against the published scorecard 0.6 JSON Schema (ajv, draft 2020-12) before being committed. `bun run build` is clean apart from three pre-existing registry-index owner/repo warnings unrelated to this change. ## Files Modified **Modified:** - `src/shared/scorecard-format.mjs` (new `statusLabel()` source of truth; markdown row formatter routes through it) - `src/build/scorecards-render.mjs` (HTML check rows route through `statusLabel()`) - `src/build/scorecards.mjs` (load-time schema gate widened from `0.5` to the set `{0.5, 0.6}`) - `src/styles/site.css` (`check--opt_out` and `check--n_a` status styles) - `src/worker/score/summary-render.ts` (status union widened to the full seven for type accuracy) - `tests/build.test.ts` (0.6 fixture; schema acceptance and 7-status render assertions) - `tests/scorecard-format-shared.test.ts` (`statusLabel` and new-status markdown rows) **Created:** - None. **Renamed:** - None. **Deleted:** - None. ## Key Features - One status-to-label map shared by the HTML rows and the markdown twin, so the two surfaces cannot drift. - Schema gate is a set, not a single version, so the migration window holds both 0.5 and 0.6 without a flag day. ## Benefits - Unblocks the U6 registry rescore: the renderer is ready for 0.6 data before that data is committed. - Backward compatible: unknown statuses fall back to uppercase rather than rendering blank. ## Breaking Changes - [x] No breaking changes ## Deployment Notes - [x] No special deployment steps required ## Checklist - [x] Code follows project conventions and style guidelines - [x] Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/) - [x] Self-review of code completed - [x] Tests added/updated and passing - [x] No new warnings or errors introduced - [x] Changes are backward compatible (or breaking changes documented) ## Additional Context The plan's U4 line item about splitting a leaderboard "skip + error" bucket describes code that does not exist: the leaderboard table renders score and principles only, with no per-status bucket column. Per-status rendering happens in the per-tool "All Checks" table, which this PR updates. The badge floor-display constant and the leaderboard methodology copy still describe the 0.5 formula and floor; rewriting those is unit U5 (methodology), coordinated with the U6 rescore, so they are intentionally untouched here.
4 tasks
brettdavies
added a commit
that referenced
this pull request
May 31, 2026
## Summary Vendors agentnative-spec v0.5.0 (upstream commit `d5d4086`) into `src/principles/spec/`. The diff is six files: `VERSION` bump 0.4.0 → 0.5.0, the upstream `CHANGELOG.md`, and prose-only edits on p3/p4/p5/p7 retargeting deferred-work footnotes from "v0.4.0 PR" to "future PR" now that those PRs have landed. No new requirements arrive via this sync. Every substantive item the upstream v0.5.0 release notes advertise already shipped here ahead of the spec release (PRs #55, #62, #65, and the scoring work): - `p3-must-version` and `p3-should-version-short` - The 7-status taxonomy with antecedent propagation (scorecard schema 0.6) - The `check` → `audit` rename - Badge floor lowered 80 → 70, with four cohort bands This vendoring closes the loop so the in-tree registry and the upstream tag point at the same content. The only consumer-visible behavior change is the scorecard's `spec_version` field, which starts emitting `"0.5.0"` instead of `"0.4.0"`. ## Changelog ### Changed - Scorecard `spec_version` now reports `"0.5.0"` (vendored agentnative-spec bumped 0.4.0 → 0.5.0). ## Type of Change - [x] `feat`: New feature (non-breaking change which adds functionality) ## Related Issues/Stories - Upstream tag: https://github.com/brettdavies/agentnative/releases/tag/v0.5.0 - Vendored commit: `d5d4086` - Pre-shipped here: #55, #62, #65 ## Testing - [x] All tests passing **Test Summary:** - `cargo build`: clean - `cargo test`: 793 passed, 2 ignored - `anc emit coverage-matrix --check`: exit 0 (no registry drift) - Pre-push hook (fmt, clippy `-Dwarnings`, test, cargo-deny, Windows check): passed ## Files Modified **Modified:** - `src/principles/spec/VERSION` - `src/principles/spec/CHANGELOG.md` - `src/principles/spec/principles/p3-progressive-help-discovery.md` - `src/principles/spec/principles/p4-fail-fast-actionable-errors.md` - `src/principles/spec/principles/p5-safe-retries-mutation-boundaries.md` - `src/principles/spec/principles/p7-bounded-high-signal-responses.md` ## Breaking Changes - [x] No breaking changes ## Deployment Notes - [x] No special deployment steps required
brettdavies
added a commit
that referenced
this pull request
Jun 1, 2026
## Summary `v0.5.0` of `anc`. Vendored `agentnative-spec` advances `0.4.0` → `0.5.0`. Scorecard JSON schema advances `0.5` → `0.7` in two steps (7-status taxonomy at `0.6`; `check_id` → `audit_id` rename at `0.7`). Coverage matrix JSON `schema_version` rebases from `"1.0"` to `"0.1"` so it matches the project's pre-release `0.x` convention. Two new universal P3 requirements (`p3-must-version`, `p3-should-version-short`) plus 22 additional behavioral checks across P1-P8 close the remaining behavioral orphan coverage. CLI subcommand surface renames `check` → `audit` and `generate` → `emit` (breaking). Badge eligibility floor lowers from 80% to 70%, and the leaderboard score becomes credit-weighted on shipped-binary behavior only. The Vale prose-check stack (`styles/`, `.vale.ini`, `scripts/prose-check.sh`) moves to dev-only contributor tooling; the workflow guard keeps it out of `main` going forward. ## Changelog ### Added - 22 new behavioral checks across P1-P8 close the remaining behavioral orphan coverage (verbose, color, raw, examples, defaults-in-help, rich-tui, about/long-about, stdin-input, consistent-naming, timeout-behavioral, structured exit codes, JSON error envelopes, actionable errors, force/yes, read/write distinction, TTY-aware verbosity, and more). - Two universal P3 requirements (`p3-must-version`, `p3-should-version-short`) verify that every CLI ships a `--version` flag plus a short alias from the `-V` / `-v` / `-version` family. - New `opt_out` and `n_a` scorecard statuses surface explicit non-adoption and unmet-antecedent rows; each `results[]` entry now carries `tier` plus `audit_id`. Pre-`0.6` consumers feature-detect. - Top-level `--verbose` / `-v` (env `AGENTNATIVE_VERBOSE`), `--examples`, `--color`, and `--raw` flags surface diagnostic detail, curated invocation blocks, ANSI styling control, and pipe-friendly `id<TAB>status` rows. - `anc skill install --all` and `anc skill update [host|--all]` iterate every known host in one invocation. ### Changed - **BREAKING**: `anc check` renamed to `anc audit`; `anc generate` renamed to `anc emit`; `anc schema` folded under `anc emit schema`. The implicit-default-subcommand injection now writes `audit`. Update scripts and CI invocations. - **BREAKING**: scorecard `schema_version` advances `"0.5"` → `"0.7"` (two-step), the `check_id` field renames to `audit_id`, and `results[]` switches to one entry per requirement-row rather than per probe. Consumers pinning the prior shape must feature-detect or update. - Leaderboard `score_pct` is now credit-weighted on shipped-binary behavior only: `warn` earns half credit, `opt_out` counts against, source/project audits no longer affect the score. Badge eligibility floor lowers from 80% to 70%. - Vendored `agentnative-spec` advances `0.4.0` → `0.5.0`; scorecard `spec_version` reports `"0.5.0"`. - `coverage/matrix.json` `schema_version` bumps `"1.0"` → `"0.1"` so the matrix follows the project's pre-release `0.x` convention used by the CLI crate, scorecard schema, and vendored spec. The lone consumer (`agentnative-site/src/build/coverage.mjs`) validates the field's existence but does not pin a specific value, so this is safe. - Vale prose-check stack (`styles/`, `.vale.ini`, `scripts/prose-check.sh`) repositions as dev-only contributor tooling. `guard-main-docs.yml` blocks those paths from `main` going forward. ### Fixed - Error output under `--output json` / `--json` now emits a JSON envelope with `error`, `kind`, `message` instead of clap's plain-text rendering, so agents pinned to JSON can parse failures with one shape. - Text mode now renders `[N/A]` (with antecedent evidence) for conditional requirements whose prerequisite was opted out, instead of a misleading `[FAIL]` on the probe id. Text row count and badge score match `--output json`. - The spec parser rejects three malformed conditional-applicability inputs that previously fell through silently (whitespace-only `antecedent.audit_id`, mixed legacy `if:` + new `kind:`, unknown keys inside `antecedent`). ### Documentation - `RELEASES-PREFLIGHT.md` lands as the operational pre-flight checklist gating each release cut. `RELEASES.md` gains a `### Cherry-pick conflicts on guarded paths` subsection and a `### Dev-direct exception` subsection covering engineering-docs paths and the prose-check stack. - Repo-wide prose pass: every tracked markdown doc plus the historical sections of `CHANGELOG.md` carry the same per-occurrence punctuation tightening applied to all 64 PR bodies in the v0.4.0 → v0.5.0 window. ## Type of Change - [x] `BREAKING CHANGE`: Breaking API change (requires major version bump) ## Related Issues/Stories - Story: n/a - Issue: n/a - Architecture: `agentnative-spec` `v0.5.0` (vendored at upstream commit `d5d4086`). Consumer-facing JSON shape moves to schema `0.7` with the `audit_id` rename. - Related PRs: cherry-picks of #55, #56, #57, #58, #59, #60, #61, #62, #63, #64, #65, #66, #68, #69 from `dev`. ## Testing - [x] All tests passing **Test Summary:** - `cargo test`: 793 passed, 2 ignored. - `anc emit coverage-matrix --check`: exit 0. - `cargo deny check advisories`: ok. - Release-mechanics preflight per `RELEASES-PREFLIGHT.md`: green except the cross-repo `agentnative-site` schema gap (tracked separately). ## Files Modified **Modified:** - Source: `src/audits/`, `src/scorecard/`, `src/principles/`, `src/main.rs`, `src/cli.rs`, `src/audit.rs`, `src/types.rs`, `src/runner/`, `src/source.rs`, `src/skill_install.rs`, and supporting modules. - Schema: `schema/scorecard.schema.json` regenerated to the `0.7` contract. - Vendored spec: `src/principles/spec/*` advances to `v0.5.0`. - Coverage artifacts: `docs/coverage-matrix.md`, `coverage/matrix.json` reflect 59 requirement rows. - Workflows: `.github/workflows/guard-main-docs.yml` (extra_paths). - Tracked docs: `README.md`, `AGENTS.md`, `CLAUDE.md`, `PRODUCT.md`, `BRAND.md`, `RELEASES.md`, `RELEASES-RATIONALE.md`, `scripts/SYNCS.md`, `.github/pull_request_template.md`, `CHANGELOG.md`. - Scripts: `scripts/generate-changelog.sh` (duplicate-section guard), `scripts/sync-spec.sh` (`--ref` flag and `gh api` transport), `scripts/hooks/pre-push` (shellcheck step). - Build identity: `Cargo.toml`, `Cargo.lock` (`0.5.0`). **Created:** - `RELEASES-PREFLIGHT.md` (operational pre-flight checklist). **Renamed:** - `anc check` → `anc audit` (subcommand surface). - `anc generate` → `anc emit` (subcommand surface). - Scorecard `check_id` → `audit_id` (JSON field). **Deleted:** - `.vale.ini`, `styles/`, `scripts/prose-check.sh`, plus the orphaned helpers `scripts/sync-prose-tooling.sh`, `scripts/test-prose-check.mjs`, `scripts/generate-pack-readme.mjs`. Replaced by the `guard-main-docs.yml` `extra_paths` guard that keeps the stack off `main` going forward. ## Breaking Changes - [x] Breaking changes described below: 1. **Scorecard JSON shape**: `schema_version` `"0.5"` → `"0.7"`. `check_id` renamed to `audit_id` (per-row and in `coverage/matrix.json`). `results[]` switches to one entry per requirement-row instead of per probe; each entry carries new `tier` and `audit_id` fields. Status enum expands from 3 to 7 values (`pass`, `warn`, `fail`, `opt_out`, `n_a`, `skip`, `error`). Consumers pinned to `0.5` or the prior `check_id` field must update. 2. **Subcommand surface**: `anc check` renamed to `anc audit`; `anc generate` renamed to `anc emit`; `anc schema` folded under `anc emit schema`. Scripts and CI invocations pinned to the old names must update. ## Deployment Notes - `agentnative-site` must add `'0.7'` to its `SUPPORTED_SCHEMA_VERSIONS` allowlist before this release is tagged. Without that, the site's regen pipeline rejects every `v0.5.0` scorecard upload. - After tag publish: run `./scripts/sync-dev-after-release.sh v0.5.0 && git push origin dev` per `RELEASES.md` § After publish to backport `Cargo.toml`, `Cargo.lock`, and `CHANGELOG.md` to `dev`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ships U2 of the scorecard fairness taxonomy plan (
docs/plans/2026-05-21-001-feat-scorecard-fairness-taxonomy-plan.mdin the agentnative-site repo): the CLI now emits a 7-status taxonomy with per-row results, antecedent propagation for conditional requirements, and a bumped scorecard schema. The vendored spec snaps toagentnative-specdev at commitb4f4d02(PR brettdavies/agentnative#34, U1) so the newapplicability.kind: conditional/antecedent.check_idshape parses against five conditional rows in p2 and p8.The status enum gains
opt_out(deliberate non-adoption) andn_a(conditional antecedent unmet); the existing five values are preserved.Summarygains matching counters. Every entry inresults[]now represents one requirement row instead of one probe, carries the row's RFC 2119tier(must/should/may), and names the originating probe incheck_id. A probe whoseCheck::covers()lists multiple rows (e.g.,p3-version→p3-must-version+p3-should-version-short) now produces multiple result entries. Two probes (p8-bundle-existsandp2-json-output) emitopt_outdirectly when their feature is absent; antecedent propagation rewrites every conditional row whose antecedent collapses toopt_out/n_aso downstream MUSTs and MAYs stop double-counting against tools that deliberately do not ship the prerequisite feature.The badge score formula keeps its
pass / (pass + warn + fail)shape but extends the exclusion set to cover the new statuses:opt_outis excluded from the denominator andn_ais excluded from both sides, per the plan's transitional posture (the final tier-weighted formula is the U3 spec issue, after the disambiguated input has been rescored). Text mode renders the new statuses asOPTandN/Abadges; the summary line shows all seven counters.Three parser strictness fixes the red team surfaced harden the build-time spec gate: whitespace-only
check_idis rejected, mixed legacyif:plus newkind:in one applicability block now errors (instead of silently dropping the legacy prose), and any key insideantecedentbeyondcheck_iderrors by name so v2 compound-antecedent syntax (op: any_of) cannot ship under v1 semantics.The
Cargo.tomlversion andCHANGELOG.mdare unchanged; both are release-PR artifacts perRELEASES.md.Changelog
Added
opt_outandn_ascorecard statuses surface inanc audit --output json(statusfield on each row and matching counters insummary).opt_outmarks deliberate non-adoption (tool ships no--outputflag, noAGENTS.mdbundle);n_amarks a conditional requirement whose antecedent is unmet. Pre-0.6 consumers treat both as unknown and feature-detect.results[]now carries atierfield (must/should/may, ornullfor rows not in the registry) and acheck_idfield naming the probe that produced the row.anc emit schemareturns the schema 0.6 contract ($id: https://anc.dev/scorecard-v0.6.schema.json) with the new status enum values, summary counters, and per-row fields.Changed
anc audit --output jsonemits one result entry per requirement-row instead of one percheck_id. A probe likep3-version(coversp3-must-versionandp3-should-version-short) now produces two distinct entries, each tier-stamped, so downstream scoring layers no longer need a coverage-matrix join to attribute a probe's outcome to a specific RFC 2119 level.opt_outorn_aare propagated ton_ainresults[]; rows whose antecedent isskiporerrorinherit the indeterminacy. The propagated evidence string names the antecedent check id so the chain is legible from the JSON alone.opt_out(transitional) and excludesn_afrom both sides, matching the plan's posture that no formula is provably fair until the input shape is disambiguated.anc audittext mode rendersOPTandN/Astatus badges alongside the existing five, and the summary line reports all seven counters.p8-bundle-existsemitsopt_outwhen no top-levelAGENTS.mdorSKILL.mdis found (a malformed bundle still emitswarn);p2-json-outputemitsopt_outwhen no--outputor--formatflag is detected at top level or in any subcommand.agentnative-spectree updates todevcommitb4f4d02(PR feat(applicability): add machine-checkable conditional shape agentnative#34). Five rows inp2andp8migrate to the newapplicability.kind: conditional/antecedent.check_idshape; the remaining 18 legacyapplicability.if: <prose>rows stay as-is until each prerequisite grows a machine-readable check id.Fixed
antecedent.check_idcontaining only whitespace, an applicability block carrying both legacyif:and newkind:(the legacy branch only fired for single-key maps, so the prose was being dropped on the floor), and any key insideantecedentother thancheck_id(compound antecedents are deferred to v2 of the schema per the plan's Sub-decision 2b).Documentation
schema/scorecard.schema.jsonregenerates against the 0.6 contract: new enum values, new required counters,tierandcheck_idonCheckResultView, and a refreshedexamples[0]block.coverage/matrix.jsonanddocs/coverage-matrix.mdregenerate against the new conditional applicability shape. Conditional rows surface withapplicability.antecedent.check_idpopulated; legacy rows continue to emitapplicability.condition: "<prose>".Type of Change
feat: New feature (non-breaking change which adds functionality)Related Issues/Stories
docs/plans/2026-05-21-001-feat-scorecard-fairness-taxonomy-plan.md(agentnative-site)docs/plans/2026-05-21-001-feat-scorecard-fairness-taxonomy-plan.md(agentnative-site repo)dev, vendored here at SHAb4f4d02)Testing
Test Summary:
-Dwarnings, full test suite, cargo-deny, shellcheck, Windows compat. All green.anc audit . --output jsonproduces a schema 0.6 scorecard with the self-target at the v0.4.0 baseline (no opt_out / n_a, since the repo ships both--outputandAGENTS.md);anc audit --command echo --output jsonexercises the propagation chain end-to-end (summary.opt_out: 1forp2-must-output-flagfrom direct emission,summary.n_a: 1forp2-must-schema-printfrom antecedent propagation)anc emit coverage-matrix --checkclean (committed artifacts match the registry pluscovers()declarations)Files Modified
Modified:
build_support/parser.rs: New conditional applicability shape (kind: conditional+antecedent.check_id) parses alongside the legacyif:form; three strictness fixes for the malformed-input edge cases.src/principles/registry.rs:Applicability::Conditionalbecomes a struct variant withconditionandantecedentfields; newAntecedenttype carriescheck_id. Two registry-consistency guards added.src/principles/matrix.rs: Matrix renderer handles both the legacy condition and the new antecedent shape; matrix.json emits the new shape via additive serde fields.src/types.rs:CheckStatusgainsOptOut(String)andNotApplicable(String).src/scorecard/mod.rs:SCHEMA_VERSIONbumps to 0.6; newfan_out_per_rowandpropagate_antecedentshelpers run insidebuild_scorecard;CheckResultViewgainstier+check_id;Summarygainsopt_out+n_acounters;score_pct,build_summary,format_text,format_text_raw,exit_codeupdated for the new variants. Twenty-seven new tests covering fan-out, propagation, score formula, summary invariants, serialization roundtrip.src/checks/project/bundle_exists.rs: EmitsOptOutwhen no bundle file is found; malformed bundle still emitsWarn.src/checks/behavioral/json_output.rs: EmitsOptOutwhen no--output/--formatflag is detected at top level or in any subcommand.src/principles/spec/principles/p1-non-interactive-by-default.mdthroughp8-discoverable-skill-bundle.md: Vendored atagentnative-specdev SHAb4f4d02; five rows in p2 and p8 carry the new conditional shape.schema/scorecard.schema.json: Schema 0.6 contract ($id, enum values, required counters,tier+check_idon results, refreshed example).coverage/matrix.jsonanddocs/coverage-matrix.md: Regenerated against the new applicability shape.tests/integration.rs: Schema version assertion bumped to0.6; theaudit_profile diagnostic-onlytest looks up the per-row idp5-must-dry-run(withcheck_id: "p5-dry-run") under per-row emission.tests/scorecard_metadata_security.rs: Schema version assertion bumped to0.6.tests/scorecard_schema_v05.rs: Schema version assertion bumped to0.6;assert_v05_shapewalks the newsummary.opt_out/summary.n_akeys and every row'stier/check_id. Six schema-drift guards assert the committed schema agrees with the live scorecard.tests/build_parser.rs: Eight parser red-team tests covering malformedantecedentshapes, whitespace-onlycheck_id, mixed legacy plus new shape, smuggled v2op:key, and a defense-in-depthemit_rustquote-escape test.Created:
Renamed:
Deleted:
Breaking Changes
Schema 0.6 is an additive evolution of 0.5: every pre-0.6 key remains, the new keys are populated alongside the existing ones, and consumers that do not understand
opt_out/n_ashould treat them asskip(the conservative bucket, matching the contract in the schema's own description). The site renderer, leaderboard ingest, and any third-party agent consumer can ship their schema-0.6 support out-of-band without coordination.The one shape change that warrants a heads-up: each entry in
results[]is now keyed by requirement row id (e.g.,p3-must-version) instead of probe id (e.g.,p3-version). Consumers that pinned on the probe id should switch to the newcheck_idfield on each row, which carries the probe provenance forward.Deployment Notes
The release PR will bump
Cargo.tomlandCHANGELOG.md. Downstream consumers (site renderer, Homebrew tap, registry rescore) consume the published artifact after the tag pushes; no out-of-band coordination needed.Checklist