Add save files for all 6 benchmark scenarios#17
Merged
Conversation
Phase A of scenario save creation: copies the crashlanded base save and patches the difficulty XML field for each scenario (Medium, Rough, or Hard). Adds create_scenario_saves.py script with Phase B checklist for manual dev-mode pawn/item spawning on scenarios needing 4-5 colonists. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase B: script now clones Thing_Human184 via XML manipulation to add colonists for 4-5 pop scenarios. Remaps thing IDs, hediff/gene loadIDs, clears social relations, and inserts scenario-specific item stacks. - Toxic Fallout: 4 colonists, wood/food/steel - Raid Defense: 5 colonists, weapons/armor/steel/wood/components - Plague Response: 5 colonists, medicine/food - Ship Launch: 5 colonists, steel/components/plasteel/gold/uranium Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The set-forbidden endpoint (PR #65) handles unforbidding at runtime in run_scenario.py. No need to mutate save XML. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The RIMAPI DDD refactor wraps /api/v1/map/things in
{success, data, errors}. Our client expected a raw list,
so unforbid_all_items silently found 0 items every time.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Member
Author
|
All test plan items verified: Live game (OpenRouter Nemotron 120B paid):
Docker:
|
7 tasks
jkbennitt
added a commit
that referenced
this pull request
Jun 6, 2026
…outs, cost source (#20) * Restore per-scenario *_deliberations.jsonl export in run_scenario Plan item A1. docker-benchmark (Apr 12) wrote *_deliberations.jsonl per scenario via run_benchmark.py; pr17-live (run via run_scenario.py) silently dropped the export, so post-PR-#17 runs lost the per-tick agent rationale (actions, reasons, summary). Re-add it via a public deliberation_log property on RLEGameLoop so both scripts use the same contract. - RLEGameLoop.deliberation_log: public property returning a shallow copy of the per-tick records (tick, agent, status, plus actions/summary/ confidence for success and raw/reason for failures). - run_scenario.py: writes <scenario>_deliberations.jsonl alongside the CSV when --output is given, off-thread via asyncio.to_thread. - run_benchmark.py: switched to the public property (was reading the private _deliberation_log attribute). - Integration test asserts record count and shallow-copy semantics. Tests: 383 pass (+1). ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Expand DELIBERATION event payload with actions + summary Plan item A2. The DELIBERATION events emitted to events.jsonl previously carried only latency_ms / confidence / num_actions; the rich payload (actions array with type/target/priority/reason, plus the plan summary) lived only in the in-memory _deliberation_log mirror. Now both writers emit the same data so events.jsonl is self-contained for downstream analysis without needing to cross-reference *_deliberations.jsonl. Also surfaces the raw LLM content (first 500 chars) on parse_failure ERROR events, matching what _deliberation_log already records. Extracted three truncation constants (_ACTION_REASON_CHARS=200, _PLAN_SUMMARY_CHARS=300, _PARSE_FAILURE_RAW_CHARS=500) so the two writers share the same shape and the limits are documented in one place. Tests: 384 pass (+1). ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Surface raw LLM output via PROVIDER_CALL events Plan item A3. _last_raw_output (the full LLM completion text) was stored in-memory on each agent but never exported, so post-hoc analysis of a weird deliberation required keeping the process alive. Now every PROVIDER_CALL event carries the agent's raw completion text (head- truncated to 4 KB) plus a raw_output_truncated bool so consumers know whether the JSONL has the full response or just a prefix. PROVIDER_CALL is the right semantic home — it's the provider's response to the call we already log token counts for. The truncation length is configurable via _RAW_OUTPUT_CHARS (matches the other event-log truncation constants from A2). 4 KB covers typical action-plan JSON plus a reasoning preamble. Tests: 385 pass (+1). ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add replay-grade metadata + --seed flag + per-scenario summary JSON Plan item A6 (commit 1 of 2). Surfaces the metadata a benchmark replay kit needs: a versioned scoring identifier, the deployed RIMAPI DLL hash, the RIMAPI fork commit, and an explicit seed for RLE-side stochasticity. Scenario save_sha256 + loader validation lands in a follow-up. metadata.py: * SCORING_VERSION constant ("1.0") — bumped when DEFAULT_WEIGHTS, metric implementations, or composite math change in a way that makes older runs not directly comparable. Surfaced in every run summary. * file_sha256(path): stdlib-only chunked SHA-256, returns None on missing/unreadable so the metadata dict stays serializable. * _rimapi_dll_path(): env override ($RIMAPI_DLL_PATH) → Workshop default (Steam path). Hashed via file_sha256. * _rimapi_fork_commit(): env override ($RIMAPI_FORK_PATH) → sibling ../RIMAPI checkout. Empty string when not reachable. * collect_metadata(random_seed=None): threads the seed through so the summary records what was actually set. Docstring notes that the seed only affects RLE-side randomness, not RimWorld's internal RNG. CLI: * Both run_benchmark.py and run_scenario.py gain --seed INT. When set, random.seed() is called before any work and the value is threaded through to collect_metadata() at every summary write. run_scenario.py: * Writes <scenario>_summary.json alongside the CSV/deliberations, including the metadata block, model/provider config, outcome, final composite, ticks_run, cost_snapshot, and event_summary. The new helper _build_run_summary keeps main() from getting longer. Tests: 391 pass (+6 new metadata tests). ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Pin canonical save SHA-256 into scenario YAMLs + loader validation Plan item A6 (commit 2 of 2). Live game runs can silently drift from the canonical docker/saves/ mirror (RimWorld saves into AppData on Windows; nothing forces it to match the repo). When the agent + scoring code are working off a different save than the one the leaderboard tracks, every result is uncomparable. Pin and verify. schema.py: * ScenarioConfig.save_sha256: str | None — optional pinned hash of the docker/saves/<save_name>.rws file. loader.py: * ScenarioSaveMismatchError — distinct exception so callers can handle this specifically (e.g., a CLI flag --allow-unpinned). * canonical_save_path(save_name) — resolves to <repo>/docker/saves/. * load_scenario(path, allow_unpinned=False) — if save_sha256 is set AND the canonical save exists on disk AND the actual SHA doesn't match, raises. allow_unpinned=True is the intentional escape hatch. Short-circuits silently when the canonical file is missing (CI without docker volumes) — that's a separate failure mode caught at game-load time. scripts/hash_saves.py: * New helper that hashes every canonical save and updates the matching scenario YAML's save_sha256 field. --print for dry-run. definitions/*.yaml: * All 6 scenarios now carry pinned save_sha256 fields (the actual hashes of the current docker/saves/*.rws files). Re-run scripts/hash_saves.py after rebuilding saves to re-pin. Tests: 394 pass (+3 new SHA validation tests). ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Per-task LLM timeout via asyncio.wait_for Plan item A7. A single hung LLM call previously blocked the entire tick because asyncio.gather has no per-task timeout. Now each deliberation runs under a wait_for(role_timeout_s) wrapper; on timeout the orchestrator emits a deliberation_timeout ERROR event, records the agent's empty contribution in the deliberation log, and proceeds. The hung thread keeps running in the background until the provider's own timeout cleans it up (Python threads can't be force-killed) but the tick advances regardless. Why it matters for the benchmark: cheap and premium models can be in the same matrix only if tail latency is bounded. Without this, a stuck 30B/120B request stalls every other agent's runtime measurement. config.py: * RLEConfig.role_timeout_s: float = 60.0 (~8x the 7s avg deliberation observed in docker-benchmark; tunable via .env or CLI override). game_loop.py: * _deliberate_agent_with_timeout(): new async wrapper, runs _deliberate_agent in a worker thread under asyncio.wait_for. Emits EventType.ERROR with error_type=deliberation_timeout + timeout_s on TimeoutError; returns (agent, None). * _deliberate_parallel(): collapsed the inline _run closure into the new wrapper. * _deliberate_sequential(): is now async and awaits the wrapper. Sequential-mode dispatch in run_tick() now awaited. * MapAnalyst dispatch also goes through the timeout wrapper. Test: TestMultiAgent.test_role_timeout_emits_event_and_other_agents_continue injects a 5s-sleep provider with role_timeout_s=0.05; asserts 7 deliberation_timeout events fire, merged plan has 0 actions, tick completes successfully. Tests: 395 pass (+1). ruff + mypy strict clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Populate benchmark_history.jsonl from run_scenario.py too Plan item A8. run_benchmark.py already calls append_history (line 711); run_scenario.py never did. Combined with the fact that the user has been doing live smoke testing via run_scenario.py (not run_benchmark.py), results/benchmark_history.jsonl stayed at 1 byte even though dozens of runs were persisted in per-run subdirectories. Now run_scenario.py appends one history entry per --output run, tagged run_type: "scenario" so consumers can distinguish single-scenario runs from full benchmark batteries (which would tag run_type: "benchmark" when run_benchmark.py is updated; that's a separate trivial follow-up). Skips the history append when cost_tracker.num_calls == 0 (smoke tests where the LLM was never reached, e.g. today's broken nano-4b run that generated parse failures across the board because LM Studio was down). Those rows would skew the leaderboard's avg-score-per-model otherwise. Tests: 395 pass. ruff + mypy src/ clean. (Script-level mypy errors at run_scenario.py:171, :175 are pre-existing in untouched code.) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Cost tracker: surface pricing source + manual override flags Plan item A9. The live smoke test reported \$0.044 internal vs \$0.168 actual OpenRouter charge. Internal estimates depend on the public /models pricing, which can diverge from billed cost due to provider routing surcharges, BYOK markups, or stale prices. The tracker would silently fall back to \$0/token if the model wasn't found or the API was unreachable. Now those failure modes are visible, and operators can plug in the actual price they're seeing on their bill. cost_tracker.py: * CostSnapshot gains prompt_price_per_token, completion_price_per_token, and pricing_source (one of: openrouter_api / override / unknown). "unknown" flags that estimated_cost_usd is \$0 because pricing resolution failed — consumers should not aggregate it. * create_cost_tracker() accepts prompt_price_override / completion_price_override (per-token USD); when both are passed, the live fetch is skipped entirely (source="override"). * Logs the resolved prices at INFO ("prompt=\$X.XX/MTok completion= \$Y.YY/MTok source=...") and a WARNING when the fetch resolved to \$0 for a model the user explicitly named. CLI: * --prompt-price-per-mtok / --completion-price-per-mtok on both run_scenario.py and run_benchmark.py. Per-MTok is the standard human-readable unit on OpenRouter's pricing page; the scripts convert to per-token before handing to create_cost_tracker. Tests: 398 pass (+3 new pricing-source / override tests; +1 nemotron-120B reconciliation test that pins the exact math from the docker-benchmark cost recording). ruff + mypy strict clean. Reconciliation against an actual billed amount is still a follow-up: needs an authenticated query to OpenRouter's /credits or per-generation endpoints. For now the override is the unblock. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Treat .rws saves as binary so scenario SHA pins are cross-platform CI failed every scenario-loader test on PR #20: pinned save_sha256 values matched the on-disk hash on Windows but not on Linux. Root cause was Git's autocrlf=true converting CRLF<->LF on commit/checkout for the .rws saves (which are XML text). The repo wound up storing LF bytes while my Windows working tree had CRLF, so the SHA-256 differed between OSes and the loader rejected every scenario. .gitattributes now declares *.rws as binary, which disables line-ending translation. Re-staged the 6 save files so the repo stores the same bytes as the working tree (CRLF, as RimWorld writes them on Windows). File sizes grew ~3% as expected. The pinned SHAs in the YAMLs were already the CRLF hashes, so no YAML changes are needed. Cross-platform contract going forward: the bytes in docker/saves/ are the canonical bytes the scenario loader hashes against, regardless of OS. RimWorld instances writing into AppData are still free to use any line endings — the loader compares against docker/saves/, not against the AppData copy. Tests: tests/unit/test_scenario_loader.py — 15 pass locally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: jkbennitt <jkbennitt@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.rwssave files todocker/saves/so all 6 benchmark scenarios can run end-to-end (local and Docker)scripts/create_scenario_saves.pygenerates saves programmatically from the crashlanded base via XML manipulation — no manual RimWorld dev mode neededTest plan
Automated (no game needed):
python scripts/run_benchmark.py --smoke-test --ticks 2completes all 6 scenariospython scripts/create_scenario_saves.py --phase bis idempotent (re-running produces identical files)Manual (requires RimWorld + RIMAPI):
curl -X POST http://localhost:8765/api/v1/game/load -H "Content-Type: application/json" -d '{"file_name":"rle_first_winter_v1"}'(repeat for all 5 new saves)Stretch (Docker):
docker runwith all 6 saves mounted viarimworld-linux+rimapi-modvolumes (port 8766)🤖 Generated with Claude Code