Compaction fails under TPM pressure and compacted sessions can restore stale history

[ste-bah]

Summary

Large Archon sessions can become unusable even when the TUI reports they are below the context-window threshold, for example:

ctx main 470k/1000k (47% bundled-catalog) | cache 33/135k
body around 1.87MB
rate limited, retrying in 30s (attempt 1/3)
rate limited, retrying in 30s (attempt 2/3)
rate limited, retrying in 30s (attempt 3/3)

This is not a context-window overflow. It is token-per-minute/request-pressure failure. The provider retry loop resends the same huge request body, /compact refuses to run below 60%, and compaction can fail to survive restart because persisted stale message rows are not removed.

PRD: /home/unixdude/Archon-projects/archon/prds/finalisation/TASK-COMPACTION-TPM-PERSISTENCE-RECOVERY-v1.0.md

Source-verified contributing bugs

• crates/archon-core/src/agent/memory_integration.rs still sends speed: "fast", which Anthropic rejects for claude-sonnet-4-6.
• crates/archon-core/src/agent/autocompact.rs also sends speed: "fast" for compaction summaries.
• crates/archon-llm/src/anthropic.rs retries 429s with the same large request body up to three times.
• crates/archon-core/src/agent/compaction.rs makes bare /compact threshold-gated below 60%.
• src/session_loop/mod.rs persists current messages by upserting indexes but does not delete stale tail rows after compaction.
• crates/archon-session/src/storage.rs has truncate_messages_after, but normal persistence/manual compact do not use replace/truncate semantics.
• Slash-command permission events can write governed-learning relations through the project .archon/learning.db handle instead of the governed-learning DB handle, causing missing relation warnings.
• AgentTool::with_agent_listing injects all discovered agents into the tool description, so the tool schema grows with the local agent catalog and is sent every request.

Expected behavior

• Large request pressure should be able to trigger compaction even below context-window percentage thresholds.
• A huge request that receives 429 should not be retried unchanged three times.
• Bare /compact should force a useful compaction strategy; /compact auto can remain threshold-gated.
• Successful compaction should persist atomically and survive /resume/restart.
• Unsupported provider parameters should be capability-gated or omitted.
• Governed-learning event writes should use the governed-learning DB handle.
• Agent discovery should be lazy/paginated instead of embedded wholesale in every Agent tool description.
• /context/status should distinguish context-window source from fixed prompt/tool overhead and latest request-body pressure.

Acceptance criteria

• No speed parameter is sent to unsupported Anthropic model requests for memory extraction or compaction summaries.
• No permission-mode event logs Cannot find requested stored relation 'permission_runtime_events' due to DB handle mismatch.
• After compaction and restart, load_messages() returns only the compacted message set, with no stale tail rows.
• At ctx 470k/1000k, bare /compact attempts compaction rather than returning below-threshold.
• Large 429-limited requests do not retry the exact same body three times.
• Agent tool schema no longer embeds all discovered agent definitions.

[ste-bah]

PRD updated to rev 2 after audit feedback: provider-layer speed/effort gating is now mandatory, Anthropic wire-body tests are required, prompt-cache verification precedes any lazy-catalog refactor, replace_messages rejects empty inputs and has crash-safety semantics, /compact keeps backward-compatible auto behavior with new /compact force, rate-limit compaction scope/circuit-breaker interactions are specified, and quantitative first-turn floor targets were added. Local PRD: /home/unixdude/Archon-projects/archon/prds/finalisation/TASK-COMPACTION-TPM-PERSISTENCE-RECOVERY-v1.0.md

[ste-bah]

Rev 3 PRD update applied locally at:

/home/unixdude/Archon-projects/archon/prds/finalisation/TASK-COMPACTION-TPM-PERSISTENCE-RECOVERY-v1.0.md

Cold-read rev-2 findings folded in:

• sessions.message_count is now specified as an enforced logical-length invariant: replace_messages writes exact length, load_messages clamps to it, and tests must prove later saves cannot resurrect stale tail rows.
• TPM-005 hook points are pinned to real production paths: Agent::process_message main-session stream/error arms and SubagentRunner::run subagent stream/error arms. No unnamed provider wrapper assumption remains.
• TPM-006 Phase A now requires deterministic byte-stability testing of the serialized Agent/tool block before trusting prompt cache. Current AgentRegistry::list() is sorted, but the serialized request still needs a cache-hit regression.
• TPM-006 Phase A now requires cache-control breakpoint budget audit before adding any tool breakpoint, so identity/system prompt breakpoints are not evicted or over budget.
• Metrics now distinguish first-turn raw payload from cache-warm steady-state billable input; caching is not claimed to reduce first-turn cost.
• TPM-002 now requires a governed-relation writer census across all ensure_learning_schema relations, not just the original permission-event repro path.

No production code was changed in this PRD update.

[ste-bah]

Rev 4 PRD update applied locally at:

/home/unixdude/Archon-projects/archon/prds/finalisation/TASK-COMPACTION-TPM-PERSISTENCE-RECOVERY-v1.0.md

This folds in the missed DELTA-3 architectural directive:

• Long-tail agent catalog removal is now mandatory and unconditional in TASK-TPM-006.
• Prompt caching is complementary only: it optimizes residual fixed prompt/core-tool overhead after catalog removal, but it cannot be used as a reason to keep injecting all agent definitions.
• Phase B now always runs: keep a small curated/common inline agent set and move the long tail behind lazy AgentCatalog lookup.
• Acceptance now requires long-tail catalog removal unconditionally; it no longer says lazy catalog runs only if the cache-warm floor target is missed.
• The PRD keeps the measurement caveat: catalog removal is estimated around 20-25k tokens, not the entire observed 47% floor, so residual overhead still needs measurement and follow-up if high.

No production code was changed in this PRD update.