Skip to content

harness: ingest LLM candidate packs into the hard benchmark #421

Description

@AbdelStark

Parent

#417

What to build

Allow the hard downstream benchmark to consume manifest-backed LLM candidate packs while preserving the existing OpenRouter/BYOK, redaction, non-execution, and secret-scan boundaries.

Acceptance criteria

  • Benchmark configs can reference one or more codelewm.llm_candidate_pack.v1 artifacts as candidate sources.
  • Candidate-pack ingestion records provider/model/SDK metadata, prompt redaction status, candidate checksums, parser/apply status, and source manifest lineage.
  • Missing, malformed, unscanned, or secret-positive candidate packs become typed blockers.
  • Live provider use is not required for fixture tests; deterministic checked-in or generated fixtures cover the ingestion path.
  • LLM candidates participate in anti-saturation diagnostics and baseline rows without being executed during model scoring.

Blocked by

#419

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions