fix(agent-challenge): stop forwarding operator OpenRouter key after cutoff - #67
Open
echobt wants to merge 1 commit into
Open
fix(agent-challenge): stop forwarding operator OpenRouter key after cutoff#67echobt wants to merge 1 commit into
echobt wants to merge 1 commit into
Conversation
…utoff The operator was paying for miner inference by injecting OPENROUTER_API_KEY from the validator process into every evaluation container via harbor_forward_env_vars, and that forward also shadowed any miner-supplied key. Miners must supply their own key for new submissions. Grandfather submissions created before operator_env_forward_cutoff_at so in-flight work keeps the previous behavior. Comparison uses submission.created_at (not evaluation start) so a later re-eval of an old submission stays grandfathered. The boundary instant is post-cutoff. Post-cutoff missing keys raise a ValueError that names the var and is surfaced through the existing terminal-bench failure path. Env: CHALLENGE_OPERATOR_ENV_FORWARD_CUTOFF_AT (ISO-8601 UTC). Unset keeps legacy always-forward until ops configures a cutoff.
📝 WalkthroughWalkthroughChangesThe challenge configuration now accepts an optional UTC cutoff timestamp. Terminal-Bench environment construction uses submission creation time to disable operator environment forwarding at or after the cutoff, validates required miner variables, and propagates timestamps through all execution paths. Operator environment forwarding cutoff
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Submission
participant TerminalBenchRunner
participant ChallengeSettings
participant TerminalBenchEnv
Submission->>TerminalBenchRunner: provide created_at
TerminalBenchRunner->>ChallengeSettings: read operator_env_forward_cutoff_at
TerminalBenchRunner->>TerminalBenchEnv: build environment with forwarding decision
TerminalBenchEnv-->>TerminalBenchRunner: return validated environment
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/challenges/agent-challenge/tests/test_operator_env_forward_cutoff.py`:
- Around line 21-22: Replace the API-key-shaped literals assigned to
OPERATOR_KEY and MINER_KEY with clearly non-secret sentinel fixture strings that
do not resemble real OpenRouter tokens, while preserving their distinct roles in
the tests.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 35f1af22-7d44-4e50-86d3-dca3b2e0ab7c
📒 Files selected for processing (5)
packages/challenges/agent-challenge/config.example.yamlpackages/challenges/agent-challenge/src/agent_challenge/evaluation/runner.pypackages/challenges/agent-challenge/src/agent_challenge/sdk/config.pypackages/challenges/agent-challenge/tests/test_config.pypackages/challenges/agent-challenge/tests/test_operator_env_forward_cutoff.py
Comment on lines
+21
to
+22
| OPERATOR_KEY = "sk-or-v1-operator-forward-test-value" | ||
| MINER_KEY = "sk-or-v1-miner-supplied-test-value" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
Replace API-key-shaped test literals with non-secret sentinels.
These literals mimic full OpenRouter API tokens, which violates the repository rule even in test evidence. Use clearly non-secret fixture values instead.
Proposed fix
-OPERATOR_KEY = "<redacted API-key-shaped value>"
-MINER_KEY = "<redacted API-key-shaped value>"
+OPERATOR_KEY = "operator-test-sentinel"
+MINER_KEY = "miner-test-sentinel"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| OPERATOR_KEY = "sk-or-v1-operator-forward-test-value" | |
| MINER_KEY = "sk-or-v1-miner-supplied-test-value" | |
| OPERATOR_KEY = "operator-test-sentinel" | |
| MINER_KEY = "miner-test-sentinel" |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@packages/challenges/agent-challenge/tests/test_operator_env_forward_cutoff.py`
around lines 21 - 22, Replace the API-key-shaped literals assigned to
OPERATOR_KEY and MINER_KEY with clearly non-secret sentinel fixture strings that
do not resemble real OpenRouter tokens, while preserving their distinct roles in
the tests.
Source: Coding guidelines
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OPENROUTER_API_KEYinto miner evaluation containers for new submissions so miners pay for their own inference.Why
The operator was paying for miner inference:
harbor_forward_env_vars(prod:OPENROUTER_API_KEY) was read from the validator process env and injected into every terminal-bench job. That forward also shadowed any miner-supplied key of the same name. Product requirement: miners must put their API key in submission env for upcoming submissions only.Behavior
created_atsanitize_miner_env_for_job; not shadowedValueErrornaming the missing varCutoff keying:
submission.created_at, not evaluation start time. Re-evaluating an old submission later still grandfathers it. Boundary instant (created_at == cutoff) is post-cutoff.Config (prod):
operator_env_forward_cutoff_atCHALLENGE_OPERATOR_ENV_FORWARD_CUTOFF_AT(ISO-8601 UTC, e.g.2026-07-30T12:00:00+00:00)null= legacy always-forward (safe until ops sets the cutoff)Changes
sdk/config.py— new optional datetime settingevaluation/runner.py—_should_forward_operator_env, cutoff-aware_terminal_bench_env, plumbsubmission.created_atat all call sitestests/test_operator_env_forward_cutoff.py— pre/post/boundary/missing-key/env-overrideconfig.example.yaml,test_config.pydefault assertionTest plan
test_miner_env_lock,test_llm_model_reaches_agent,test_terminal_bench_env_keeps_controlled_writable_cache,test_normal_validator_defaultsruff check+ruff formaton touched filesCI
Notes
ac-keyrelease-ratls, emission shares, weights, or metagraph.CHALLENGE_OPERATOR_ENV_FORWARD_CUTOFF_ATat deploy time to the desired grandfather boundary (typically deploy time).Summary by CodeRabbit
New Features
Bug Fixes