Summary
tools/skill-evals/evals/release-prepare/README.md documents assertions.json
as the only way to grade prose steps, but a per-step grading-schema.json
override also exists and is used in this repo.
Background
The suite's "Grading prose steps" section is where a contributor editing
fixtures looks first. It never mentions grading-schema.json, so someone
hitting a brittle exact-match on a prose field will not know the override exists
and will work around it instead, usually by loosening the expected value until
it passes.
Where to look
evals/release-prepare/README.md - the "Grading prose steps" section.
grep -c grading-schema on it currently returns 0.
evals/release-verify-rc/*/fixtures/grading-schema.json - working examples.
tools/skill-evals/src/skill_evals/runner.py - how the override is resolved.
Acceptance criteria
Estimated effort
~1 hour for someone new to the codebase.
Summary
tools/skill-evals/evals/release-prepare/README.mddocumentsassertions.jsonas the only way to grade prose steps, but a per-step
grading-schema.jsonoverride also exists and is used in this repo.
Background
The suite's "Grading prose steps" section is where a contributor editing
fixtures looks first. It never mentions
grading-schema.json, so someonehitting a brittle exact-match on a prose field will not know the override exists
and will work around it instead, usually by loosening the expected value until
it passes.
Where to look
evals/release-prepare/README.md- the "Grading prose steps" section.grep -c grading-schemaon it currently returns 0.evals/release-verify-rc/*/fixtures/grading-schema.json- working examples.tools/skill-evals/src/skill_evals/runner.py- how the override is resolved.Acceptance criteria
grading-schema.json, what it does, and where it goes.assertions.json.Estimated effort
~1 hour for someone new to the codebase.