fix(scorecard): align text renderer with JSON via per-row pipeline - #63
Merged
Conversation
…pipeline Fold the fan_out_per_row + propagate_antecedents pair that build_scorecard inlined into one reusable function so the text path can derive the same per-row result set the JSON path uses. No behavior change: build_scorecard produces byte-identical output.
…eline The default --output text view and its badge were computed from raw probe results, so the terminal printed probe ids (p2-schema-print) instead of requirement-row ids (p2-must-schema-print), never rendered [N/A ], and could diverge from the JSON the site/scorer consume. Feed format_text and compute_badge the fanned-out + antecedent-propagated row set instead, and render the requirement tier (must/should/may) on each row. Text and JSON now agree on the row set, counts, and badge score.
exit_code read the raw probe results while summary, score_pct, and badge already keyed on the per-row set, so a tool that raw-Fails a check whose requirement row propagates to n_a (its antecedent opted out) exited 2 even though the requirement does not apply. Feed exit_code the same projected per-row set the text and JSON paths use, so a propagated-to-n_a row stops lifting the exit code. Only live Fail/Error drives exit 2 and live Warn drives exit 1 — mirroring the score_pct denominator. The --audit-profile R4 masking path is independent and unchanged.
Add two complementary guards for the data-flow gap. A unit test renders a bat-shaped fixture (opt_out antecedent + raw-Fail conditional consequent) through both the text projection and build_scorecard, asserting the consequent shows n_a not fail, row counts match, and badge scores match. An integration test audits ls in both modes and asserts the text row count equals JSON results.len() and the exit codes agree — catching any revert that feeds the text renderer raw probe results instead of the per-row set.
…vel::as_str Both CheckResultView::from_row (JSON tier field) and format_text (text tier suffix) open-coded the same match over Level, the exact metadata triplication that risks the text/JSON divergence this work closes. Add Level::as_str as the single source of truth, kept in lock-step with the serde lowercase spelling, and route both surfaces through it. Reword the build_row_results doc to describe a shared derivation rather than implying a single call site.
brettdavies
added a commit
that referenced
this pull request
Jun 1, 2026
## Summary `v0.5.0` of `anc`. Vendored `agentnative-spec` advances `0.4.0` → `0.5.0`. Scorecard JSON schema advances `0.5` → `0.7` in two steps (7-status taxonomy at `0.6`; `check_id` → `audit_id` rename at `0.7`). Coverage matrix JSON `schema_version` rebases from `"1.0"` to `"0.1"` so it matches the project's pre-release `0.x` convention. Two new universal P3 requirements (`p3-must-version`, `p3-should-version-short`) plus 22 additional behavioral checks across P1-P8 close the remaining behavioral orphan coverage. CLI subcommand surface renames `check` → `audit` and `generate` → `emit` (breaking). Badge eligibility floor lowers from 80% to 70%, and the leaderboard score becomes credit-weighted on shipped-binary behavior only. The Vale prose-check stack (`styles/`, `.vale.ini`, `scripts/prose-check.sh`) moves to dev-only contributor tooling; the workflow guard keeps it out of `main` going forward. ## Changelog ### Added - 22 new behavioral checks across P1-P8 close the remaining behavioral orphan coverage (verbose, color, raw, examples, defaults-in-help, rich-tui, about/long-about, stdin-input, consistent-naming, timeout-behavioral, structured exit codes, JSON error envelopes, actionable errors, force/yes, read/write distinction, TTY-aware verbosity, and more). - Two universal P3 requirements (`p3-must-version`, `p3-should-version-short`) verify that every CLI ships a `--version` flag plus a short alias from the `-V` / `-v` / `-version` family. - New `opt_out` and `n_a` scorecard statuses surface explicit non-adoption and unmet-antecedent rows; each `results[]` entry now carries `tier` plus `audit_id`. Pre-`0.6` consumers feature-detect. - Top-level `--verbose` / `-v` (env `AGENTNATIVE_VERBOSE`), `--examples`, `--color`, and `--raw` flags surface diagnostic detail, curated invocation blocks, ANSI styling control, and pipe-friendly `id<TAB>status` rows. - `anc skill install --all` and `anc skill update [host|--all]` iterate every known host in one invocation. ### Changed - **BREAKING**: `anc check` renamed to `anc audit`; `anc generate` renamed to `anc emit`; `anc schema` folded under `anc emit schema`. The implicit-default-subcommand injection now writes `audit`. Update scripts and CI invocations. - **BREAKING**: scorecard `schema_version` advances `"0.5"` → `"0.7"` (two-step), the `check_id` field renames to `audit_id`, and `results[]` switches to one entry per requirement-row rather than per probe. Consumers pinning the prior shape must feature-detect or update. - Leaderboard `score_pct` is now credit-weighted on shipped-binary behavior only: `warn` earns half credit, `opt_out` counts against, source/project audits no longer affect the score. Badge eligibility floor lowers from 80% to 70%. - Vendored `agentnative-spec` advances `0.4.0` → `0.5.0`; scorecard `spec_version` reports `"0.5.0"`. - `coverage/matrix.json` `schema_version` bumps `"1.0"` → `"0.1"` so the matrix follows the project's pre-release `0.x` convention used by the CLI crate, scorecard schema, and vendored spec. The lone consumer (`agentnative-site/src/build/coverage.mjs`) validates the field's existence but does not pin a specific value, so this is safe. - Vale prose-check stack (`styles/`, `.vale.ini`, `scripts/prose-check.sh`) repositions as dev-only contributor tooling. `guard-main-docs.yml` blocks those paths from `main` going forward. ### Fixed - Error output under `--output json` / `--json` now emits a JSON envelope with `error`, `kind`, `message` instead of clap's plain-text rendering, so agents pinned to JSON can parse failures with one shape. - Text mode now renders `[N/A]` (with antecedent evidence) for conditional requirements whose prerequisite was opted out, instead of a misleading `[FAIL]` on the probe id. Text row count and badge score match `--output json`. - The spec parser rejects three malformed conditional-applicability inputs that previously fell through silently (whitespace-only `antecedent.audit_id`, mixed legacy `if:` + new `kind:`, unknown keys inside `antecedent`). ### Documentation - `RELEASES-PREFLIGHT.md` lands as the operational pre-flight checklist gating each release cut. `RELEASES.md` gains a `### Cherry-pick conflicts on guarded paths` subsection and a `### Dev-direct exception` subsection covering engineering-docs paths and the prose-check stack. - Repo-wide prose pass: every tracked markdown doc plus the historical sections of `CHANGELOG.md` carry the same per-occurrence punctuation tightening applied to all 64 PR bodies in the v0.4.0 → v0.5.0 window. ## Type of Change - [x] `BREAKING CHANGE`: Breaking API change (requires major version bump) ## Related Issues/Stories - Story: n/a - Issue: n/a - Architecture: `agentnative-spec` `v0.5.0` (vendored at upstream commit `d5d4086`). Consumer-facing JSON shape moves to schema `0.7` with the `audit_id` rename. - Related PRs: cherry-picks of #55, #56, #57, #58, #59, #60, #61, #62, #63, #64, #65, #66, #68, #69 from `dev`. ## Testing - [x] All tests passing **Test Summary:** - `cargo test`: 793 passed, 2 ignored. - `anc emit coverage-matrix --check`: exit 0. - `cargo deny check advisories`: ok. - Release-mechanics preflight per `RELEASES-PREFLIGHT.md`: green except the cross-repo `agentnative-site` schema gap (tracked separately). ## Files Modified **Modified:** - Source: `src/audits/`, `src/scorecard/`, `src/principles/`, `src/main.rs`, `src/cli.rs`, `src/audit.rs`, `src/types.rs`, `src/runner/`, `src/source.rs`, `src/skill_install.rs`, and supporting modules. - Schema: `schema/scorecard.schema.json` regenerated to the `0.7` contract. - Vendored spec: `src/principles/spec/*` advances to `v0.5.0`. - Coverage artifacts: `docs/coverage-matrix.md`, `coverage/matrix.json` reflect 59 requirement rows. - Workflows: `.github/workflows/guard-main-docs.yml` (extra_paths). - Tracked docs: `README.md`, `AGENTS.md`, `CLAUDE.md`, `PRODUCT.md`, `BRAND.md`, `RELEASES.md`, `RELEASES-RATIONALE.md`, `scripts/SYNCS.md`, `.github/pull_request_template.md`, `CHANGELOG.md`. - Scripts: `scripts/generate-changelog.sh` (duplicate-section guard), `scripts/sync-spec.sh` (`--ref` flag and `gh api` transport), `scripts/hooks/pre-push` (shellcheck step). - Build identity: `Cargo.toml`, `Cargo.lock` (`0.5.0`). **Created:** - `RELEASES-PREFLIGHT.md` (operational pre-flight checklist). **Renamed:** - `anc check` → `anc audit` (subcommand surface). - `anc generate` → `anc emit` (subcommand surface). - Scorecard `check_id` → `audit_id` (JSON field). **Deleted:** - `.vale.ini`, `styles/`, `scripts/prose-check.sh`, plus the orphaned helpers `scripts/sync-prose-tooling.sh`, `scripts/test-prose-check.mjs`, `scripts/generate-pack-readme.mjs`. Replaced by the `guard-main-docs.yml` `extra_paths` guard that keeps the stack off `main` going forward. ## Breaking Changes - [x] Breaking changes described below: 1. **Scorecard JSON shape**: `schema_version` `"0.5"` → `"0.7"`. `check_id` renamed to `audit_id` (per-row and in `coverage/matrix.json`). `results[]` switches to one entry per requirement-row instead of per probe; each entry carries new `tier` and `audit_id` fields. Status enum expands from 3 to 7 values (`pass`, `warn`, `fail`, `opt_out`, `n_a`, `skip`, `error`). Consumers pinned to `0.5` or the prior `check_id` field must update. 2. **Subcommand surface**: `anc check` renamed to `anc audit`; `anc generate` renamed to `anc emit`; `anc schema` folded under `anc emit schema`. Scripts and CI invocations pinned to the old names must update. ## Deployment Notes - `agentnative-site` must add `'0.7'` to its `SUPPORTED_SCHEMA_VERSIONS` allowlist before this release is tagged. Without that, the site's regen pipeline rejects every `v0.5.0` scorecard upload. - After tag publish: run `./scripts/sync-dev-after-release.sh v0.5.0 && git push origin dev` per `RELEASES.md` § After publish to backport `Cargo.toml`, `Cargo.lock`, and `CHANGELOG.md` to `dev`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
--output textview, its badge, and the process exit code were computed from raw probe results, while--output jsonalready ran the schema-0.6 per-row + antecedent-propagation pipeline. So the terminal printed probe ids (p2-schema-print) instead of requirement-row ids (p2-must-schema-print), could never render[N/A ](only propagation producesn_a), and its row count, badge score, and exit code could disagree with the JSON the site and scorer consume. A human auditingbatsaw[FAIL] … (p2-schema-print)for a MUST the tool is actually exempt from.This routes all three text-side surfaces through the same pipeline. The fan-out + propagation pair that
build_scorecardinlined is extracted intobuild_row_results, andmain::runderives one per-row set that feedsformat_text,compute_badge, andexit_code. Text and JSON now agree on the row set, counts, badge score, and exit code. The JSON path is byte-identical and the schema stays0.6.Changelog
Changed
anc audit(text mode) now reports the same requirement rows as--output json: requirement-row ids (p2-must-schema-print) instead of probe ids, with the requirement tier (must/should/may) shown on each row.n_abecause its conditional prerequisite isopt_outno longer forces a non-zero exit; only a liveFail/Errorexits2and a liveWarnexits1.Fixed
[N/A ](with antecedent evidence) for conditional requirements whose prerequisite was opted out, instead of showing a misleading[FAIL]on the probe id. Text row count and badge score now match--output json.Type of Change
fix: Bug fix (non-breaking change which fixes an issue)Related Issues/Stories
docs/plans/2026-05-27-002-fix-text-renderer-per-row-parity-plan.mdTesting
Test Summary:
build_row_results_fans_out_then_propagates,exit_code_drops_to_zero_when_consequent_propagates_to_n_a,text_and_json_agree_on_a_propagated_conditional_row(bat-shaped fixture asserting consequent showsn_anotfail, row counts match, badge scores match).test_text_and_json_agree_on_row_count_and_exit_codeauditslsin both modes and pins text row count == JSONresults.len()and exit-code parity, catching any revert that feeds the text renderer raw probe results.format_text_*/format_text_raw/ R4 audit-profile masking tests pass unchanged.Files Modified
Modified:
src/main.rs: derive one per-row projection after the principle filter; feed it toformat_text,compute_badge, andexit_code.src/scorecard/mod.rs: addbuild_row_results;build_scorecardcalls it;format_textrenders the tier suffix; consolidate theLevel→token mapping behindLevel::as_str.src/principles/registry.rs: addLevel::as_stras the single source of truth for themust/should/maytoken.tests/integration.rs: text/JSON row-count + exit-code parity guard.docs/plans/2026-05-27-002-fix-text-renderer-per-row-parity-plan.md: statusproposed→completed.Created:
Renamed:
Deleted:
Breaking Changes
The CLI surface and JSON schema are unchanged. The exit-code semantics shift for one edge: a tool that raw-Fails a check whose requirement row propagates to
n_anow exits0/1instead of2. That is the corrected behavior (the requirement does not apply), it matches what the JSONsummaryalready reported, andancis0.xpre-launch. The--audit-profilemasking path (plan R4) is independent and unchanged.Deployment Notes
Post-Deploy Monitoring & Validation
No additional operational monitoring is required because
ancis a local CLI with no runtime service. The only downstream consumer isagentnative-site, which reads the JSON scorecard; the JSON path is byte-identical (schema stays0.6), so there is no consumer impact. Spot-check after merge:anc audit --command bat(text) shows[N/A ] … (p2-must-schema-print) (must)and no[FAIL] … (p2-schema-print), andanc audit --command bat/--output jsonagree on exit code.Checklist