data: add anti-saturation benchmark schema and readiness diagnostics (#419)#425
Merged
Merged
Conversation
…419) Add the RFC-0016 `anti_saturation_semantic_v1` profile and the `codelewm.downstream_anti_saturation_report.v1` diagnostic so a hard downstream reranking pack exposes anti-saturation eligibility before any CodeLeWM scoring. New `codelewm/eval/downstream_anti_saturation.py`: - `build_anti_saturation_report` computes eligibility from the RFC-0016 gates: problem_count >= 100, 6-12 candidates per pool, no-action and lexical pass@1 < 0.85, LLM-order pass@1 < 0.90, >= 70% of problems with two or more distinct failing hard-negative classes, plus the source/license, split-leakage, manifest, and secret-scan gates. Saturated, under-covered, or missing-baseline slices are preserved with `eligible=False` and a typed `blocked_reasons` entry; nothing is dropped. - `build_anti_saturation_claim_gate` opens only when CodeLeWM beats no-action, lexical, AND LLM-order on pass@1 and MRR (with lift CIs excluding zero when supplied) on an eligible >=100-problem slice. - `compute_model_independent_baselines` derives the no-action / lexical / LLM-order / random pass@1, pool sizes, class coverage, dual-coverage fraction, and parser/apply failure rate from a materialized pack. It is text-only: patch apply is lazy-imported and parseability uses ast.parse. - `validate_hard_negative_class` enforces the RFC-0016 class enumeration. `downstream_pack.py` gains optional `profile` (pack config) and `hard_negative_class` (candidate) fields. When the profile is set, the pack build writes `reports/anti_saturation_report.json`, records it in the manifest, feeds eligibility into the readiness report, and reports the path + eligibility on the result. Plain fixture configs are unaffected. Adds a public-safe fixture pack (one task, six classed candidates) that is non-saturated on every baseline and blocked only by the 1<100 problem count, plus focused tests for eligible/saturated/too-small/missing-baseline cases, the claim gate, and the pack-build + CLI paths. Closes #419. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the RFC-0016
anti_saturation_semantic_v1profile and thecodelewm.downstream_anti_saturation_report.v1diagnostic so a hard downstream reranking pack exposes anti-saturation eligibility before any CodeLeWM scoring. The change is additive: a new schema-onlydownstream_anti_saturationmodule plus optionalprofile/hard_negative_classconfig fields on the existing pack builder. Plain (non-profile) fixture configs are byte-for-byte unaffected.Linked Issue
Closes #419.
Spec / RFC Reference
docs/spec/11-llm-world-model-harness.mddocs/rfcs/RFC-0016-hard-downstream-reranking-benchmark.mdPublic Surface Impact
New Python API (exported from
codelewm.eval):build_anti_saturation_report,build_anti_saturation_claim_gate,compute_model_independent_baselines,validate_hard_negative_class,validate_anti_saturation_report,validate_anti_saturation_claim_gate,anti_saturation_report_json_schema,lexical_similarity,stable_random_key,DownstreamAntiSaturationError, and theANTI_SATURATION_*/HARD_NEGATIVE_CLASSES/ ceiling constants.New schema versions (additive):
codelewm.downstream_anti_saturation_report.v1,codelewm.downstream_anti_saturation_claim_gate.v1.New config keys (both optional):
profile(pack config, onlyanti_saturation_semantic_v1),hard_negative_class(candidate).DownstreamBenchmarkPackResultgains optionalanti_saturation_report_path/anti_saturation_eligible. No existing field, baseline tuple, or schema version changed.No new CLI command — the existing
eval downstream-packemits the report when the config sets the profile.Validation
Artifact Impact
A pack built with
profile: anti_saturation_semantic_v1now writesreports/anti_saturation_report.json(schemacodelewm.downstream_anti_saturation_report.v1), records it inmanifest.jsonmetadata (profile,anti_saturation_report,anti_saturation_eligible), and addsanti_saturation_eligible/anti_saturation_blocked_reasonsto the readiness report. Packs without a profile are unchanged.Deprecations
noneCaveats / Follow-ups
eligible=false(blocked only byproblem_count_below_minimum:1<100), preserving the v1.0 claim boundary.