Claw Model Back-Tester

Automated LLM-based scoring of historical OpenClaw session transcripts. Sends each session's user request + assistant response to an evaluator LLM with a strict rubric, receives a 1-3 score and rationale, and saves results to scores.jsonl.

The goal: find out if a fast local model (or Claude/GPT) can reliably "score the scorer" — enabling bulk evaluation of session quality for later training/LoRA work.

Scoring Rubric

• 1 (Poor / Dangerous): Hallucinated, missed key details, ignored context, wrong action, dangerous outcome.
• 2 (Good Enough): Required steering or had minor mistakes, but progressed the task.
• 3 (Excellent): One-shot, clean solve on the first try.

Usage

# Score sessions using a local Ollama model (default: llama3 @ localhost:11434)
./score_sessions.py /path/to/sessions/

# Use a different local model
./score_sessions.py /path/to/sessions/ --model mistral

# Use Claude via OpenRouter / any OpenAI-compatible endpoint
./score_sessions.py /path/to/sessions/ \
  --api-base https://openrouter.ai/api/v1 \
  --model anthropic/claude-sonnet-4-20250514 \
  --api-key sk-or-...

# Use OpenAI directly
./score_sessions.py /path/to/sessions/ \
  --api-base https://api.openai.com/v1 \
  --model gpt-4o \
  --api-key sk-...

# Dry run — see what would be scored without calling the LLM
./score_sessions.py /path/to/sessions/ --dry-run

# Use env vars instead of CLI args
export EVAL_API_BASE=http://localhost:11434/v1
export EVAL_MODEL=llama3
export EVAL_API_KEY=
./score_sessions.py /path/to/sessions/

Configuration

Setting	CLI Flag	Env Var	Default
API base URL	--api-base	EVAL_API_BASE	http://localhost:11434/v1
Model name	--model	EVAL_MODEL	llama3
API key	--api-key	EVAL_API_KEY	(none)

CLI flags override env vars.

Output

Results append to scores.jsonl, one JSON object per line:

{"session_id": "abc123", "model": "claude-sonnet-4-20250514", "score": 3, "rationale": "Clean one-shot solve with correct tool use.", "evaluator": "llama3"}

Already-scored sessions are skipped on re-runs.

Score the Scorer

Compare two evaluator models head-to-head to see if a fast/cheap model can replace an expensive one as an automated judge.

The script picks N random sessions, scores each with both a reference ("best") evaluator and a candidate ("evaluated") evaluator, then reports exact-match accuracy.

# Compare local llama3 against Claude as reference (20 random sessions)
./score_the_scorer.py /path/to/sessions/ \
  --best-api-base https://openrouter.ai/api/v1 \
  --best-model anthropic/claude-sonnet-4-20250514 \
  --best-api-key sk-or-... \
  --eval-api-base http://localhost:11434/v1 \
  --eval-model llama3 \
  -n 20

# Use env vars for keys
export BEST_API_BASE=https://openrouter.ai/api/v1
export BEST_MODEL=anthropic/claude-sonnet-4-20250514
export BEST_API_KEY=sk-or-...
./score_the_scorer.py /path/to/sessions/ --eval-model mistral -n 30

# Reproducible selection with a seed
./score_the_scorer.py /path/to/sessions/ --seed 42 -n 15

# Dry run — see which sessions would be selected
./score_the_scorer.py /path/to/sessions/ -n 10 --dry-run

Scorer Comparison Configuration

Setting	CLI Flag	Env Var	Default
Reference API base	--best-api-base	BEST_API_BASE	https://openrouter.ai/api/v1
Reference model	--best-model	BEST_MODEL	anthropic/claude-sonnet-4-20250514
Reference API key	--best-api-key	BEST_API_KEY	(none)
Candidate API base	--eval-api-base	EVAL_API_BASE	http://localhost:11434/v1
Candidate model	--eval-model	EVAL_MODEL	llama3
Candidate API key	--eval-api-key	EVAL_API_KEY	(none)
Sample size	-n	—	10
Random seed	--seed	—	(random)

Scorer Comparison Output

Results append to scorer_comparison.jsonl:

{"session_id": "abc123", "session_model": "claude-sonnet-4-20250514", "best_evaluator": "anthropic/claude-sonnet-4-20250514", "best_score": 3, "best_rationale": "Clean solve.", "eval_evaluator": "llama3", "eval_score": 3, "eval_rationale": "Good result.", "match": true}

The summary includes exact-match accuracy (to 2 decimal places), score distributions, and mismatch breakdown.

Dependencies

pip install requests