feat: HumanEval coding benchmark for TRINITY training + retry/timeout reliability fixes

[dangeReis]

Summary

• GSM8K saturates at one worker hitting 100% solve rate, giving sep-CMA-ES nothing meaningful to learn from. Adds train/train_trinity_coding.py, using HumanEval (auto-graded via real subprocess execution of candidate code) as a more discriminating benchmark.
• Fixes two reliability bugs found while validating train/train_trinity_real.py end-to-end against a real NIM worker pool:
  • Unretried litellm.RateLimitError (429) was silently scored as a wrong answer, corrupting the reward signal under normal rate limiting.
  • litellm.completion() had no timeout, so a stalled connection could hang the whole run indefinitely.
  • Follow-up fix (caught by an independent review pass): a call that exhausted retries or hit a non-rate-limit exception (e.g. a timeout) was still being permanently cached as a graded "unsolved" answer for that (worker, task) pair — corrupting the cache for the rest of the run even though nothing was ever actually graded. Fixed so only real graded results are cached; failed calls are retried on a later iteration instead.
• Scrubs API-key env vars from the subprocess environment used to execute untrusted candidate code in train_trinity_coding.py.

Both scripts now retry 429s with exponential backoff and use timeout=45 on the LLM call.

Test plan

• Live GSM8K run (train_trinity_real.py) against a real NIM worker pool: PASS, coordinator (1.00) matched best single worker.
• Local smoke test of the HumanEval grading/assembly logic against a real openai_humaneval problem + canonical solution (including round-trip through the fenced-code extractor) — confirmed valid, executable Python.
• Live HumanEval run (4-worker pool) against a real NIM worker pool: baseline per-worker solve rates 0.75–0.95, coordinator converged to 1.000 — PASS.
• Independent code review flagged the cache-poisoning bug described above; fixed and re-verified live: PASS again after the fix (0.925 coordinator vs 0.925 best single worker, run against a 3-worker subset after one model hit a transient provider-side outage during that pass).

🤖 Generated with Claude Code

Summary by Sourcery

Introduce a HumanEval-based TRINITY training script for coding tasks, add support for loading materialized coordinator weights in the router, and harden LLM worker calls against rate limits, timeouts, and cache corruption.

New Features:

• Add a TRINITY self-training script that uses the HumanEval coding benchmark with real worker pools and executable code grading.
• Support loading materialized coordinator weights and router head from a checkpoint directory as an alternative to SVF vector reconstruction.

Bug Fixes:

• Ensure LLM worker calls are retried on rate limiting, respect a timeout, and avoid caching failed or ungraded calls as permanent unsolved results.

Enhancements:

• Improve logging and robustness of worker call handling and routing, including safer subprocess execution that scrubs API-key environment variables before running untrusted code.

Documentation:

• Add a HANDOFF document describing the TRINITY materialized-weights loader workflow and verification steps.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[sourcery-ai]

Reviewer's Guide

Adds a new TRINITY HumanEval-based coding training script, introduces a materialized-weights loader path for FuguRouter, and hardens LLM worker calls with retries, timeouts, and safer caching/environment handling.

File-Level Changes

Change	Details	Files
Add HumanEval-based TRINITY coding training script using real LLM workers and subprocess-graded code execution.	Introduce Backbone helper to compute penultimate Qwen3-0.6B hidden-state features for HumanEval prompts with caching. Implement routing and CMA-ES optimization over a bias-free linear head matching the existing TRINITY setup. Define worker_solves that calls litellm with retry/backoff and timeout, executes candidate code in a scrubbed subprocess environment, and caches graded results by (worker, task_id). Wire up CLI arguments, dataset loading for openai/openai_humaneval, baseline per-worker evaluation, training loop, and result reporting.	train/train_trinity_coding.py
Harden GSM8K TRINITY training script LLM calls with retries, timeout, and safer caching/logging semantics.	Refactor worker_solves to build a litellm.completion kwargs dict including timeout=45 and optional api_key/base, and initialize ok/call_succeeded flags. Wrap LLM call in up-to-5-attempt loop with exponential backoff on litellm.RateLimitError and early exit on non-rate-limit exceptions. Avoid caching failed or never-succeeded calls, logging a SKIPPED message instead; only cache actually graded answers and log cache size for visibility. Import time module required for backoff sleeps.	train/train_trinity_real.py
Add support for loading materialized TRINITY checkpoints (weights + router head) as an alternative to vector-based SVF reconstruction in FuguRouter.	Relax FuguRouter constructor to accept an optional vector_path and remove unconditional vector load at init. Add optional FUGU_CHECKPOINT_DIR path: when set, call load_materialized to retrieve materialized weights and router head, copy weights into the Hugging Face model state dict under no_grad, remember svf_keys, and move head tensor to device. Fallback to existing vector_path-based npy loading and SVF application when FUGU_CHECKPOINT_DIR is unset, validating vector shape and reshaping head accordingly.	openfugu/mini.py
Implement materialized checkpoint loader that maps Bumblebee-exported safetensors into the Qwen3-0.6B state dict and router head tensor for TRINITY.	Read manifest.json from a checkpoint directory to discover selected_tensors entries describing the 9 adapted weights. Load each referenced safetensors file with safetensors.torch.load_file, select the appropriate key (with a substring fallback), and ensure shapes match expected Qwen3-0.6B modules, applying transposes where needed (including special cases for k_proj and v_proj). Load router_head.safetensors, pick a known or first tensor entry, enforce shape (10, 1024) with transpose fallback, and return both the weights dict and router head tensor. Use descriptive errors when checkpoint files, keys, or shapes are missing or incompatible.	openfugu/materialized.py
Add handoff documentation describing the TRINITY materialized-weights loader task and verification steps.	Document repository context, problem statement about missing CMA-ES vector and new HF dataset with materialized weights. Specify steps to implement load_materialized, download artifacts via huggingface_hub, and minimally hook the loader into FuguRouter via FUGU_CHECKPOINT_DIR. Describe verification procedure using FUGU_MODEL, FUGU_CHECKPOINT_DIR, and FUGU_FIXTURE with the mini.py self-test, plus expected accuracy targets and guidance if results look wrong.	HANDOFF.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 1 security issue, and 3 other issues

Security issues:

• Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'. (link)

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="train/train_trinity_coding.py" line_range="125" />
<code_context>
+        call_succeeded = False
+        # Candidate code runs via subprocess in the parent's env; scrub API keys so
+        # hallucinated/adversarial completions can't read or exfiltrate them.
+        run_env = {k: v for k, v in os.environ.items() if "KEY" not in k.upper()}
+        for attempt in range(5):
+            try:
</code_context>
<issue_to_address>
**🚨 issue (security):** Environment scrubbing for untrusted code is too narrow and may leave other secrets accessible.

Filtering only env vars containing "KEY" still leaves many sensitive values (tokens, passwords, org/tenant IDs, etc.) visible to untrusted code. Given this runs arbitrary completions, consider starting from an empty env and explicitly allowlisting only what’s required (e.g. PATH, PYTHONPATH if needed), rather than copying almost everything and subtracting keys with "KEY" in the name.
</issue_to_address>

### Comment 2
<location path="openfugu/materialized.py" line_range="41-48" />
<code_context>
+        if not os.path.exists(abs_chk_path):
+            raise FileNotFoundError(f"Checkpoint file not found: {abs_chk_path}")
+            
+        tensors_dict = load_file(abs_chk_path)
+        if path_key not in tensors_dict:
+            # fallback to checking if any key in tensors_dict contains the name
+            matching_keys = [k for k in tensors_dict if path_key in k]
+            if matching_keys:
+                tensor = tensors_dict[matching_keys[0]]
+            else:
+                tensor = list(tensors_dict.values())[0]
+        else:
+            tensor = tensors_dict[path_key]
</code_context>
<issue_to_address>
**issue (bug_risk):** Fallback to the first tensor when `path_key` is missing is risky and can silently load wrong weights.

When `path_key` is missing and no partial match exists, this falls back to `list(tensors_dict.values())[0]`, which can bind an unrelated tensor to `source_name` and corrupt the model in a non-obvious way. Instead of guessing, consider raising an error that includes `path_key` and the available keys.
</issue_to_address>

### Comment 3
<location path="openfugu/materialized.py" line_range="49-53" />
<code_context>
+        else:
+            tensor = tensors_dict[path_key]
+            
+        expected_shape = expected_shapes[source_name]
+        if source_name in [
+            "model.layers.26.self_attn.k_proj.weight",
</code_context>
<issue_to_address>
**suggestion:** Indexing `expected_shapes[source_name]` without validation can throw on manifest/key drift.

This assumes every `source_name` in the manifest exists in `expected_shapes`, so any new/renamed tensor will raise a bare `KeyError`. Consider checking membership first and raising a clearer error that explains the unexpected `source_name` and the mismatch between the checkpoint and the hardcoded expectations.

```suggestion
        else:
            tensor = tensors_dict[path_key]

        if source_name not in expected_shapes:
            available_expected = ", ".join(sorted(expected_shapes.keys()))
            available_tensors = ", ".join(sorted(tensors_dict.keys()))
            raise KeyError(
                f"Unexpected tensor source_name {source_name!r} when looking up expected_shapes. "
                f"This usually indicates drift between the checkpoint/manifest and the "
                f"hardcoded expected_shapes. "
                f"Available expected_shapes keys: {available_expected}. "
                f"Available checkpoint tensor keys: {available_tensors}."
            )

        expected_shape = expected_shapes[source_name]
        if source_name in [
```
</issue_to_address>

### Comment 4
<location path="train/train_trinity_coding.py" line_range="139" />
<code_context>
                    res = subprocess.run([sys.executable, tmp_path], capture_output=True, timeout=10, env=run_env)
</code_context>
<issue_to_address>
**security (python.lang.security.audit.dangerous-subprocess-use-audit):** Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'.

*Source: opengrep*
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[sourcery-ai]

security (python.lang.security.audit.dangerous-subprocess-use-audit): Detected subprocess function 'run' without a static string. If this data can be controlled by a malicious actor, it may be an instance of command injection. Audit the use of this call to ensure it is not controllable by an external resource. You may consider using 'shlex.escape()'.

Source: opengrep

[dangeReis]

Re: the remaining opengrep finding on `train/train_trinity_coding.py:141` (`subprocess.run` "without a static string" / possible command injection):

This one's a static-analysis pattern match rather than an actual injection vector in this call shape — there's no `shell=True`, arguments are passed as a list (`[sys.executable, tmp_path]`), so there's no shell to escape into. Running the candidate completion as a real Python process is the intended behavior — that's how HumanEval-style grading works (same pattern as the reference HumanEval eval harness): the model's candidate solution has to actually execute against the test cases to produce a pass/fail reward.

The genuine risk here isn't shell injection, it's "this executes untrusted model output" in general — which is inherent to any code-benchmark grader, not something `shlex.escape()` addresses (there's nothing to escape, no shell is invoked). What is mitigated in this PR: the subprocess environment is allowlisted (PATH/PYTHONPATH only, verified live) rather than inheriting the full parent env, so a malicious/hallucinated completion can't read secrets from the environment. Happy to add a sandbox (container/seccomp) if that's wanted for this repo, but wanted to flag that as a separate, bigger scope-of-work decision rather than block this PR on a semgrep pattern that doesn't apply to this call shape.