feat(scorecard): implement final behavioral-only scoring formula (U3b) - #64
Merged
Conversation
Replace the transitional U2 score with the formula chosen in agentnative-spec principles/scoring.md (PR #39, 972c9d3): - Restrict score_pct to behavioral-layer rows. Source- and project-layer checks still emit in the scorecard but no longer move the score, so a source-only run now reports 0% and ineligible by design. - Score over the denominator set D = {pass, warn, fail, opt_out} with credit-weighted numerator: pass 1.0, warn 0.5, fail and opt_out 0.0. opt_out now counts in the denominator; n_a / skip / error stay excluded. - Encode tier weights as named consts (W_MUST = W_SHOULD = W_MAY = 1.0) applied in the general Σ w·credit / Σ w form, so a future non-flat re-tune is a one-line change. - Lower the badge eligibility floor from 80% to 70%. Update the README Scoring section, dogfood table, and badge field docs, plus the CLAUDE.md scorecard reference, to the new formula and the behavioral-only scope. anc dogfoods to 100% (34/34 behavioral pass). Cargo.toml version, CHANGELOG, and tag publish are deferred to the release PR per the plan. ## Changelog ### Changed - The leaderboard score now reflects shipped-binary behavior only: source- and project-layer checks no longer affect score_pct or badge eligibility. - score_pct is now credit-weighted: warn earns half credit and opt_out counts against the score, replacing the prior pass-only ratio. - Lower the agent-native badge eligibility floor from 80% to 70%.
brettdavies
added a commit
that referenced
this pull request
Jun 1, 2026
## Summary `v0.5.0` of `anc`. Vendored `agentnative-spec` advances `0.4.0` → `0.5.0`. Scorecard JSON schema advances `0.5` → `0.7` in two steps (7-status taxonomy at `0.6`; `check_id` → `audit_id` rename at `0.7`). Coverage matrix JSON `schema_version` rebases from `"1.0"` to `"0.1"` so it matches the project's pre-release `0.x` convention. Two new universal P3 requirements (`p3-must-version`, `p3-should-version-short`) plus 22 additional behavioral checks across P1-P8 close the remaining behavioral orphan coverage. CLI subcommand surface renames `check` → `audit` and `generate` → `emit` (breaking). Badge eligibility floor lowers from 80% to 70%, and the leaderboard score becomes credit-weighted on shipped-binary behavior only. The Vale prose-check stack (`styles/`, `.vale.ini`, `scripts/prose-check.sh`) moves to dev-only contributor tooling; the workflow guard keeps it out of `main` going forward. ## Changelog ### Added - 22 new behavioral checks across P1-P8 close the remaining behavioral orphan coverage (verbose, color, raw, examples, defaults-in-help, rich-tui, about/long-about, stdin-input, consistent-naming, timeout-behavioral, structured exit codes, JSON error envelopes, actionable errors, force/yes, read/write distinction, TTY-aware verbosity, and more). - Two universal P3 requirements (`p3-must-version`, `p3-should-version-short`) verify that every CLI ships a `--version` flag plus a short alias from the `-V` / `-v` / `-version` family. - New `opt_out` and `n_a` scorecard statuses surface explicit non-adoption and unmet-antecedent rows; each `results[]` entry now carries `tier` plus `audit_id`. Pre-`0.6` consumers feature-detect. - Top-level `--verbose` / `-v` (env `AGENTNATIVE_VERBOSE`), `--examples`, `--color`, and `--raw` flags surface diagnostic detail, curated invocation blocks, ANSI styling control, and pipe-friendly `id<TAB>status` rows. - `anc skill install --all` and `anc skill update [host|--all]` iterate every known host in one invocation. ### Changed - **BREAKING**: `anc check` renamed to `anc audit`; `anc generate` renamed to `anc emit`; `anc schema` folded under `anc emit schema`. The implicit-default-subcommand injection now writes `audit`. Update scripts and CI invocations. - **BREAKING**: scorecard `schema_version` advances `"0.5"` → `"0.7"` (two-step), the `check_id` field renames to `audit_id`, and `results[]` switches to one entry per requirement-row rather than per probe. Consumers pinning the prior shape must feature-detect or update. - Leaderboard `score_pct` is now credit-weighted on shipped-binary behavior only: `warn` earns half credit, `opt_out` counts against, source/project audits no longer affect the score. Badge eligibility floor lowers from 80% to 70%. - Vendored `agentnative-spec` advances `0.4.0` → `0.5.0`; scorecard `spec_version` reports `"0.5.0"`. - `coverage/matrix.json` `schema_version` bumps `"1.0"` → `"0.1"` so the matrix follows the project's pre-release `0.x` convention used by the CLI crate, scorecard schema, and vendored spec. The lone consumer (`agentnative-site/src/build/coverage.mjs`) validates the field's existence but does not pin a specific value, so this is safe. - Vale prose-check stack (`styles/`, `.vale.ini`, `scripts/prose-check.sh`) repositions as dev-only contributor tooling. `guard-main-docs.yml` blocks those paths from `main` going forward. ### Fixed - Error output under `--output json` / `--json` now emits a JSON envelope with `error`, `kind`, `message` instead of clap's plain-text rendering, so agents pinned to JSON can parse failures with one shape. - Text mode now renders `[N/A]` (with antecedent evidence) for conditional requirements whose prerequisite was opted out, instead of a misleading `[FAIL]` on the probe id. Text row count and badge score match `--output json`. - The spec parser rejects three malformed conditional-applicability inputs that previously fell through silently (whitespace-only `antecedent.audit_id`, mixed legacy `if:` + new `kind:`, unknown keys inside `antecedent`). ### Documentation - `RELEASES-PREFLIGHT.md` lands as the operational pre-flight checklist gating each release cut. `RELEASES.md` gains a `### Cherry-pick conflicts on guarded paths` subsection and a `### Dev-direct exception` subsection covering engineering-docs paths and the prose-check stack. - Repo-wide prose pass: every tracked markdown doc plus the historical sections of `CHANGELOG.md` carry the same per-occurrence punctuation tightening applied to all 64 PR bodies in the v0.4.0 → v0.5.0 window. ## Type of Change - [x] `BREAKING CHANGE`: Breaking API change (requires major version bump) ## Related Issues/Stories - Story: n/a - Issue: n/a - Architecture: `agentnative-spec` `v0.5.0` (vendored at upstream commit `d5d4086`). Consumer-facing JSON shape moves to schema `0.7` with the `audit_id` rename. - Related PRs: cherry-picks of #55, #56, #57, #58, #59, #60, #61, #62, #63, #64, #65, #66, #68, #69 from `dev`. ## Testing - [x] All tests passing **Test Summary:** - `cargo test`: 793 passed, 2 ignored. - `anc emit coverage-matrix --check`: exit 0. - `cargo deny check advisories`: ok. - Release-mechanics preflight per `RELEASES-PREFLIGHT.md`: green except the cross-repo `agentnative-site` schema gap (tracked separately). ## Files Modified **Modified:** - Source: `src/audits/`, `src/scorecard/`, `src/principles/`, `src/main.rs`, `src/cli.rs`, `src/audit.rs`, `src/types.rs`, `src/runner/`, `src/source.rs`, `src/skill_install.rs`, and supporting modules. - Schema: `schema/scorecard.schema.json` regenerated to the `0.7` contract. - Vendored spec: `src/principles/spec/*` advances to `v0.5.0`. - Coverage artifacts: `docs/coverage-matrix.md`, `coverage/matrix.json` reflect 59 requirement rows. - Workflows: `.github/workflows/guard-main-docs.yml` (extra_paths). - Tracked docs: `README.md`, `AGENTS.md`, `CLAUDE.md`, `PRODUCT.md`, `BRAND.md`, `RELEASES.md`, `RELEASES-RATIONALE.md`, `scripts/SYNCS.md`, `.github/pull_request_template.md`, `CHANGELOG.md`. - Scripts: `scripts/generate-changelog.sh` (duplicate-section guard), `scripts/sync-spec.sh` (`--ref` flag and `gh api` transport), `scripts/hooks/pre-push` (shellcheck step). - Build identity: `Cargo.toml`, `Cargo.lock` (`0.5.0`). **Created:** - `RELEASES-PREFLIGHT.md` (operational pre-flight checklist). **Renamed:** - `anc check` → `anc audit` (subcommand surface). - `anc generate` → `anc emit` (subcommand surface). - Scorecard `check_id` → `audit_id` (JSON field). **Deleted:** - `.vale.ini`, `styles/`, `scripts/prose-check.sh`, plus the orphaned helpers `scripts/sync-prose-tooling.sh`, `scripts/test-prose-check.mjs`, `scripts/generate-pack-readme.mjs`. Replaced by the `guard-main-docs.yml` `extra_paths` guard that keeps the stack off `main` going forward. ## Breaking Changes - [x] Breaking changes described below: 1. **Scorecard JSON shape**: `schema_version` `"0.5"` → `"0.7"`. `check_id` renamed to `audit_id` (per-row and in `coverage/matrix.json`). `results[]` switches to one entry per requirement-row instead of per probe; each entry carries new `tier` and `audit_id` fields. Status enum expands from 3 to 7 values (`pass`, `warn`, `fail`, `opt_out`, `n_a`, `skip`, `error`). Consumers pinned to `0.5` or the prior `check_id` field must update. 2. **Subcommand surface**: `anc check` renamed to `anc audit`; `anc generate` renamed to `anc emit`; `anc schema` folded under `anc emit schema`. Scripts and CI invocations pinned to the old names must update. ## Deployment Notes - `agentnative-site` must add `'0.7'` to its `SUPPORTED_SCHEMA_VERSIONS` allowlist before this release is tagged. Without that, the site's regen pipeline rejects every `v0.5.0` scorecard upload. - After tag publish: run `./scripts/sync-dev-after-release.sh v0.5.0 && git push origin dev` per `RELEASES.md` § After publish to backport `Cargo.toml`, `Cargo.lock`, and `CHANGELOG.md` to `dev`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements U3b of the scorecard fairness taxonomy plan: the CLI now computes the final leaderboard score defined in
agentnative-specprinciples/scoring.md(authority:agentnative-spec@972c9d3), replacing the transitional formula shipped in U2.The score now reflects shipped-binary behavior only.
score_pctis restricted to behavioral-layer rows; source- and project-layer checks still emit in the scorecard but no longer move the score or badge eligibility. Over the denominator setD = {pass, warn, fail, opt_out}, the score is a credit-weighted ratio (pass1.0,warn0.5,failandopt_out0.0) withn_a/skip/errorexcluded fromD.opt_outnow counts in the denominator so deliberate non-adoption is reflected. Tier weights are published flat (MUST = SHOULD = MAY = 1) but encoded as named constants in the generalΣ w·credit / Σ wform, so a future non-flat re-tune is a one-line change. The badge eligibility floor drops from 80% to 70%.A consequence of the behavioral-only scope:
anc audit . --sourcenow scores 0% and ineligible by design, because a source-only run has no behavioral rows to score. anc dogfoods to 100% in--binaryand full mode (34 of 34 behavioral checks pass; 9 behavioral skips are excluded from the denominator).Cargo.tomlversion,CHANGELOG.md, and the tag publish are deferred to the release PR that closes out this unit, per the plan.Changelog
Changed
score_pctor badge eligibility.score_pctis now credit-weighted:warnearns half credit andopt_outcounts against the score, replacing the prior pass-only ratio.Type of Change
feat: New feature (non-breaking change which adds functionality)Related Issues/Stories
agentnative-site/docs/plans/2026-05-21-001-feat-scorecard-fairness-taxonomy-plan.mdagentnative-spec@972c9d3principles/scoring.md.Testing
Test Summary:
scoring.mdworked example (20 pass / 7 warn / 0 fail / 1 opt_out / 1 n_a / 14 skip → 84). Reworked the at-floor (now 70%), opt_out-in-denominator, and full-pipeline propagation cases.cargo fmt --check,cargo clippy --all-targets --all-features -D warnings, andcargo testall green.Files Modified
Modified:
src/scorecard/mod.rs: rewrotescore_pct(behavioral-only, credit-weighted, tier-weighted); addedtier_weightandstatus_credithelpers; loweredBADGE_ELIGIBILITY_FLOOR_PCTto 70; updated badge docstrings and tests.README.md: rewrote the Scoring section (7 statuses, credit formula, behavioral-only scope, 70% floor, cohort bands), the dogfood table, and the badge field reference.CLAUDE.md: updated the scorecardbadgereference to the new formula.Created:
Renamed:
Deleted:
Breaking Changes
Scorecard schema stays at
0.6; the change is in howscore_pctis computed, not in the JSON shape. Consumers readingbadge.score_pctsee different values after rescore but no shape change.Deployment Notes
The public registry rescore against this formula is U6 in the site repo, gated on a published CLI release, not part of this PR.