Skip to content

fix(scorecard): align text renderer with JSON via per-row pipeline - #63

Merged
brettdavies merged 6 commits into
devfrom
fix/text-renderer-per-row-parity
May 28, 2026
Merged

fix(scorecard): align text renderer with JSON via per-row pipeline#63
brettdavies merged 6 commits into
devfrom
fix/text-renderer-per-row-parity

Conversation

@brettdavies

Copy link
Copy Markdown
Owner

Summary

The --output text view, its badge, and the process exit code were computed from raw probe results, while --output json already ran the schema-0.6 per-row + antecedent-propagation pipeline. So the terminal printed probe ids (p2-schema-print) instead of requirement-row ids (p2-must-schema-print), could never render [N/A ] (only propagation produces n_a), and its row count, badge score, and exit code could disagree with the JSON the site and scorer consume. A human auditing bat saw [FAIL] … (p2-schema-print) for a MUST the tool is actually exempt from.

This routes all three text-side surfaces through the same pipeline. The fan-out + propagation pair that build_scorecard inlined is extracted into build_row_results, and main::run derives one per-row set that feeds format_text, compute_badge, and exit_code. Text and JSON now agree on the row set, counts, badge score, and exit code. The JSON path is byte-identical and the schema stays 0.6.

Changelog

Changed

  • anc audit (text mode) now reports the same requirement rows as --output json: requirement-row ids (p2-must-schema-print) instead of probe ids, with the requirement tier (must/should/may) shown on each row.
  • The process exit code now reflects the per-row result set in both output modes. A requirement that collapses to n_a because its conditional prerequisite is opt_out no longer forces a non-zero exit; only a live Fail/Error exits 2 and a live Warn exits 1.

Fixed

  • Text mode now renders [N/A ] (with antecedent evidence) for conditional requirements whose prerequisite was opted out, instead of showing a misleading [FAIL] on the probe id. Text row count and badge score now match --output json.

Type of Change

  • fix: Bug fix (non-breaking change which fixes an issue)

Related Issues/Stories

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing completed
  • All tests passing

Test Summary:

  • New unit tests: build_row_results_fans_out_then_propagates, exit_code_drops_to_zero_when_consequent_propagates_to_n_a, text_and_json_agree_on_a_propagated_conditional_row (bat-shaped fixture asserting consequent shows n_a not fail, row counts match, badge scores match).
  • New integration test: test_text_and_json_agree_on_row_count_and_exit_code audits ls in both modes and pins text row count == JSON results.len() and exit-code parity, catching any revert that feeds the text renderer raw probe results.
  • Existing format_text_* / format_text_raw / R4 audit-profile masking tests pass unchanged.

Files Modified

Modified:

  • src/main.rs: derive one per-row projection after the principle filter; feed it to format_text, compute_badge, and exit_code.
  • src/scorecard/mod.rs: add build_row_results; build_scorecard calls it; format_text renders the tier suffix; consolidate the Level→token mapping behind Level::as_str.
  • src/principles/registry.rs: add Level::as_str as the single source of truth for the must/should/may token.
  • tests/integration.rs: text/JSON row-count + exit-code parity guard.
  • docs/plans/2026-05-27-002-fix-text-renderer-per-row-parity-plan.md: status proposedcompleted.

Created:

  • None.

Renamed:

  • None.

Deleted:

  • None.

Breaking Changes

  • No breaking changes

The CLI surface and JSON schema are unchanged. The exit-code semantics shift for one edge: a tool that raw-Fails a check whose requirement row propagates to n_a now exits 0/1 instead of 2. That is the corrected behavior (the requirement does not apply), it matches what the JSON summary already reported, and anc is 0.x pre-launch. The --audit-profile masking path (plan R4) is independent and unchanged.

Deployment Notes

  • No special deployment steps required

Post-Deploy Monitoring & Validation

No additional operational monitoring is required because anc is a local CLI with no runtime service. The only downstream consumer is agentnative-site, which reads the JSON scorecard; the JSON path is byte-identical (schema stays 0.6), so there is no consumer impact. Spot-check after merge: anc audit --command bat (text) shows [N/A ] … (p2-must-schema-print) (must) and no [FAIL] … (p2-schema-print), and anc audit --command bat / --output json agree on exit code.

Checklist

  • Code follows project conventions and style guidelines
  • Commit messages follow Conventional Commits
  • Self-review of code completed
  • Tests added/updated and passing
  • No new warnings or errors introduced
  • Changes are backward compatible (or breaking changes documented)

…pipeline

Fold the fan_out_per_row + propagate_antecedents pair that build_scorecard
inlined into one reusable function so the text path can derive the same
per-row result set the JSON path uses. No behavior change: build_scorecard
produces byte-identical output.
…eline

The default --output text view and its badge were computed from raw probe
results, so the terminal printed probe ids (p2-schema-print) instead of
requirement-row ids (p2-must-schema-print), never rendered [N/A ], and
could diverge from the JSON the site/scorer consume. Feed format_text and
compute_badge the fanned-out + antecedent-propagated row set instead, and
render the requirement tier (must/should/may) on each row. Text and JSON
now agree on the row set, counts, and badge score.
exit_code read the raw probe results while summary, score_pct, and badge
already keyed on the per-row set, so a tool that raw-Fails a check whose
requirement row propagates to n_a (its antecedent opted out) exited 2 even
though the requirement does not apply. Feed exit_code the same projected
per-row set the text and JSON paths use, so a propagated-to-n_a row stops
lifting the exit code. Only live Fail/Error drives exit 2 and live Warn
drives exit 1 — mirroring the score_pct denominator. The --audit-profile
R4 masking path is independent and unchanged.
Add two complementary guards for the data-flow gap. A unit test renders a
bat-shaped fixture (opt_out antecedent + raw-Fail conditional consequent)
through both the text projection and build_scorecard, asserting the
consequent shows n_a not fail, row counts match, and badge scores match.
An integration test audits ls in both modes and asserts the text row count
equals JSON results.len() and the exit codes agree — catching any revert
that feeds the text renderer raw probe results instead of the per-row set.
…vel::as_str

Both CheckResultView::from_row (JSON tier field) and format_text (text tier
suffix) open-coded the same match over Level, the exact metadata
triplication that risks the text/JSON divergence this work closes. Add
Level::as_str as the single source of truth, kept in lock-step with the
serde lowercase spelling, and route both surfaces through it. Reword the
build_row_results doc to describe a shared derivation rather than implying
a single call site.
@brettdavies
brettdavies merged commit b146bb7 into dev May 28, 2026
8 checks passed
@brettdavies
brettdavies deleted the fix/text-renderer-per-row-parity branch May 28, 2026 20:17
This was referenced May 31, 2026
brettdavies added a commit that referenced this pull request Jun 1, 2026
## Summary

`v0.5.0` of `anc`. Vendored `agentnative-spec` advances `0.4.0` →
`0.5.0`. Scorecard JSON schema advances `0.5` → `0.7` in two steps
(7-status taxonomy at `0.6`; `check_id` → `audit_id` rename at `0.7`).
Coverage matrix JSON `schema_version` rebases from `"1.0"` to `"0.1"` so
it matches the project's pre-release `0.x` convention. Two new universal
P3 requirements (`p3-must-version`, `p3-should-version-short`) plus 22
additional behavioral checks across P1-P8 close the remaining behavioral
orphan coverage. CLI subcommand surface renames `check` → `audit` and
`generate` → `emit` (breaking). Badge eligibility floor lowers from 80%
to 70%, and the leaderboard score becomes credit-weighted on
shipped-binary behavior only. The Vale prose-check stack (`styles/`,
`.vale.ini`, `scripts/prose-check.sh`) moves to dev-only contributor
tooling; the workflow guard keeps it out of `main` going forward.

## Changelog

### Added

- 22 new behavioral checks across P1-P8 close the remaining behavioral
orphan coverage (verbose, color, raw, examples, defaults-in-help,
rich-tui, about/long-about, stdin-input, consistent-naming,
timeout-behavioral, structured exit codes, JSON error envelopes,
actionable errors, force/yes, read/write distinction, TTY-aware
verbosity, and more).
- Two universal P3 requirements (`p3-must-version`,
`p3-should-version-short`) verify that every CLI ships a `--version`
flag plus a short alias from the `-V` / `-v` / `-version` family.
- New `opt_out` and `n_a` scorecard statuses surface explicit
non-adoption and unmet-antecedent rows; each `results[]` entry now
carries `tier` plus `audit_id`. Pre-`0.6` consumers feature-detect.
- Top-level `--verbose` / `-v` (env `AGENTNATIVE_VERBOSE`),
`--examples`, `--color`, and `--raw` flags surface diagnostic detail,
curated invocation blocks, ANSI styling control, and pipe-friendly
`id<TAB>status` rows.
- `anc skill install --all` and `anc skill update [host|--all]` iterate
every known host in one invocation.

### Changed

- **BREAKING**: `anc check` renamed to `anc audit`; `anc generate`
renamed to `anc emit`; `anc schema` folded under `anc emit schema`. The
implicit-default-subcommand injection now writes `audit`. Update scripts
and CI invocations.
- **BREAKING**: scorecard `schema_version` advances `"0.5"` → `"0.7"`
(two-step), the `check_id` field renames to `audit_id`, and `results[]`
switches to one entry per requirement-row rather than per probe.
Consumers pinning the prior shape must feature-detect or update.
- Leaderboard `score_pct` is now credit-weighted on shipped-binary
behavior only: `warn` earns half credit, `opt_out` counts against,
source/project audits no longer affect the score. Badge eligibility
floor lowers from 80% to 70%.
- Vendored `agentnative-spec` advances `0.4.0` → `0.5.0`; scorecard
`spec_version` reports `"0.5.0"`.
- `coverage/matrix.json` `schema_version` bumps `"1.0"` → `"0.1"` so the
matrix follows the project's pre-release `0.x` convention used by the
CLI crate, scorecard schema, and vendored spec. The lone consumer
(`agentnative-site/src/build/coverage.mjs`) validates the field's
existence but does not pin a specific value, so this is safe.
- Vale prose-check stack (`styles/`, `.vale.ini`,
`scripts/prose-check.sh`) repositions as dev-only contributor tooling.
`guard-main-docs.yml` blocks those paths from `main` going forward.

### Fixed

- Error output under `--output json` / `--json` now emits a JSON
envelope with `error`, `kind`, `message` instead of clap's plain-text
rendering, so agents pinned to JSON can parse failures with one shape.
- Text mode now renders `[N/A]` (with antecedent evidence) for
conditional requirements whose prerequisite was opted out, instead of a
misleading `[FAIL]` on the probe id. Text row count and badge score
match `--output json`.
- The spec parser rejects three malformed conditional-applicability
inputs that previously fell through silently (whitespace-only
`antecedent.audit_id`, mixed legacy `if:` + new `kind:`, unknown keys
inside `antecedent`).

### Documentation

- `RELEASES-PREFLIGHT.md` lands as the operational pre-flight checklist
gating each release cut. `RELEASES.md` gains a `### Cherry-pick
conflicts on guarded paths` subsection and a `### Dev-direct exception`
subsection covering engineering-docs paths and the prose-check stack.
- Repo-wide prose pass: every tracked markdown doc plus the historical
sections of `CHANGELOG.md` carry the same per-occurrence punctuation
tightening applied to all 64 PR bodies in the v0.4.0 → v0.5.0 window.

## Type of Change

- [x] `BREAKING CHANGE`: Breaking API change (requires major version
bump)

## Related Issues/Stories

- Story: n/a
- Issue: n/a
- Architecture: `agentnative-spec` `v0.5.0` (vendored at upstream commit
`d5d4086`). Consumer-facing JSON shape moves to schema `0.7` with the
`audit_id` rename.
- Related PRs: cherry-picks of #55, #56, #57, #58, #59, #60, #61, #62,
#63, #64, #65, #66, #68, #69 from `dev`.

## Testing

- [x] All tests passing

**Test Summary:**

- `cargo test`: 793 passed, 2 ignored.
- `anc emit coverage-matrix --check`: exit 0.
- `cargo deny check advisories`: ok.
- Release-mechanics preflight per `RELEASES-PREFLIGHT.md`: green except
the cross-repo `agentnative-site` schema gap (tracked separately).

## Files Modified

**Modified:**

- Source: `src/audits/`, `src/scorecard/`, `src/principles/`,
`src/main.rs`, `src/cli.rs`, `src/audit.rs`, `src/types.rs`,
`src/runner/`, `src/source.rs`, `src/skill_install.rs`, and supporting
modules.
- Schema: `schema/scorecard.schema.json` regenerated to the `0.7`
contract.
- Vendored spec: `src/principles/spec/*` advances to `v0.5.0`.
- Coverage artifacts: `docs/coverage-matrix.md`, `coverage/matrix.json`
reflect 59 requirement rows.
- Workflows: `.github/workflows/guard-main-docs.yml` (extra_paths).
- Tracked docs: `README.md`, `AGENTS.md`, `CLAUDE.md`, `PRODUCT.md`,
`BRAND.md`, `RELEASES.md`, `RELEASES-RATIONALE.md`, `scripts/SYNCS.md`,
`.github/pull_request_template.md`, `CHANGELOG.md`.
- Scripts: `scripts/generate-changelog.sh` (duplicate-section guard),
`scripts/sync-spec.sh` (`--ref` flag and `gh api` transport),
`scripts/hooks/pre-push` (shellcheck step).
- Build identity: `Cargo.toml`, `Cargo.lock` (`0.5.0`).

**Created:**

- `RELEASES-PREFLIGHT.md` (operational pre-flight checklist).

**Renamed:**

- `anc check` → `anc audit` (subcommand surface).
- `anc generate` → `anc emit` (subcommand surface).
- Scorecard `check_id` → `audit_id` (JSON field).

**Deleted:**

- `.vale.ini`, `styles/`, `scripts/prose-check.sh`, plus the orphaned
helpers `scripts/sync-prose-tooling.sh`, `scripts/test-prose-check.mjs`,
`scripts/generate-pack-readme.mjs`. Replaced by the
`guard-main-docs.yml` `extra_paths` guard that keeps the stack off
`main` going forward.

## Breaking Changes

- [x] Breaking changes described below:

1. **Scorecard JSON shape**: `schema_version` `"0.5"` → `"0.7"`.
`check_id` renamed to `audit_id` (per-row and in
`coverage/matrix.json`). `results[]` switches to one entry per
requirement-row instead of per probe; each entry carries new `tier` and
`audit_id` fields. Status enum expands from 3 to 7 values (`pass`,
`warn`, `fail`, `opt_out`, `n_a`, `skip`, `error`). Consumers pinned to
`0.5` or the prior `check_id` field must update.
2. **Subcommand surface**: `anc check` renamed to `anc audit`; `anc
generate` renamed to `anc emit`; `anc schema` folded under `anc emit
schema`. Scripts and CI invocations pinned to the old names must update.

## Deployment Notes

- `agentnative-site` must add `'0.7'` to its `SUPPORTED_SCHEMA_VERSIONS`
allowlist before this release is tagged. Without that, the site's regen
pipeline rejects every `v0.5.0` scorecard upload.
- After tag publish: run `./scripts/sync-dev-after-release.sh v0.5.0 &&
git push origin dev` per `RELEASES.md` § After publish to backport
`Cargo.toml`, `Cargo.lock`, and `CHANGELOG.md` to `dev`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant