Skip to content

Document the grading-schema override in the release-prepare eval README #951

Description

@justinmclean

Summary

tools/skill-evals/evals/release-prepare/README.md documents assertions.json
as the only way to grade prose steps, but a per-step grading-schema.json
override also exists and is used in this repo.

Background

The suite's "Grading prose steps" section is where a contributor editing
fixtures looks first. It never mentions grading-schema.json, so someone
hitting a brittle exact-match on a prose field will not know the override exists
and will work around it instead, usually by loosening the expected value until
it passes.

Where to look

  • evals/release-prepare/README.md - the "Grading prose steps" section.
    grep -c grading-schema on it currently returns 0.
  • evals/release-verify-rc/*/fixtures/grading-schema.json - working examples.
  • tools/skill-evals/src/skill_evals/runner.py - how the override is resolved.

Acceptance criteria

  • The README mentions grading-schema.json, what it does, and where it goes.
  • It points at a real example in the tree.
  • It states the relationship to assertions.json.

Estimated effort

~1 hour for someone new to the codebase.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions