diff --git a/README.md b/README.md index 137b0698..84a885e0 100644 --- a/README.md +++ b/README.md @@ -380,18 +380,18 @@ LongMemEval (Wu et al., ICLR 2025): 500 human-curated questions embedded in ~40 | | Cortex | What it means | |---|---|---| | Recall@10 | **98.2%** | The right memory shows up in the top 10 for nearly every question | -| MRR | **0.915** | The correct *memory* is usually ranked first or second — retrieval rank only, no LLM reader | +| MRR | **0.9167** | The correct *memory* is usually ranked first or second — retrieval rank only, no LLM reader | -n=500, E1 v3 verification campaign — per-row JSONs with code SHAs in `benchmarks/results/ablation/longmemeval-s_v3/`. Re-verified on a clean DB 2026-06-10. +n=500, clean-DB run `benchmarks/results/repro/20260714-v4.14.1-pretag/longmemeval-s.json`, code SHA `28145f0b7a113fc06e22568de6feea7f8444eaf5`, dirty=false. | Category | MRR | R@10 | |---|---|---| | Single-session (assistant) | 1.000 | 100.0% | -| Multi-session reasoning | 0.962 | 100.0% | -| Knowledge updates | 0.925 | 100.0% | -| Temporal reasoning | 0.926 | 98.5% | -| Single-session (user) | 0.814 | 94.3% | -| Single-session (preference) | 0.668 | 93.3% | +| Multi-session reasoning | 0.964 | 100.0% | +| Knowledge updates | 0.932 | 100.0% | +| Temporal reasoning | 0.917 | 97.7% | +| Single-session (user) | 0.841 | 95.7% | +| Single-session (preference) | 0.685 | 90.0% | Knowledge updates score near-perfect because the retrieval stack's recency signal and update-intent routing push the newest version of a fact above older ones. @@ -401,10 +401,10 @@ LoCoMo (Maharana et al., ACL 2024): 1,986 questions across 10 conversations — | | Cortex | What it means | |---|---|---| -| Recall@10 | **91.5%** | Right memory in top 10 over 9 times out of 10 | -| MRR | **0.805** | The correct *memory* is typically ranked first — retrieval rank only, no LLM reader | +| Recall@10 | **94.35%** | Right memory in top 10 over 9 times out of 10 | +| MRR | **0.8279** | The correct *memory* is typically ranked first — retrieval rank only, no LLM reader | -n=1986, BASELINE_NO_CONSOLIDATION, post-plasticity-fix — `docs/benchmarks/e1-v3-locomo-results-post-fix.md`. +n=1986, BASELINE_NO_CONSOLIDATION, post-plasticity-fix run at code SHA `2f45bcb39dbe15fa0ef857cc8c8c3783175d05db` — `docs/benchmarks/e1-v3-locomo-results-post-fix.md`. | Category | MRR | R@10 | |---|---|---| diff --git a/benchmarks/README.md b/benchmarks/README.md index 577543b3..c4ec995b 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -84,7 +84,9 @@ session(s) appear in the top 10. - The comparable published baseline is the best retrieval configuration in the LongMemEval paper itself (Wu et al., ICLR 2025): **Recall@10 - 78.4%**. Cortex: **98.4%** (n=500). + 78.4%**. Cortex: **98.2%**, MRR **0.9167** (n=500; clean-DB run + `results/repro/20260714-v4.14.1-pretag/longmemeval-s.json`, code SHA + `28145f0b7a113fc06e22568de6feea7f8444eaf5`, dirty=false). - This is **not** the end-to-end QA accuracy that LLM-answering leaderboards report (an LLM answers from the retrieved context and a judge scores the answer). Retrieval recall and QA accuracy are diff --git a/docs/arxiv-context-assembly/main.pdf b/docs/arxiv-context-assembly/main.pdf index bf2b7340..01b932e0 100644 Binary files a/docs/arxiv-context-assembly/main.pdf and b/docs/arxiv-context-assembly/main.pdf differ diff --git a/docs/arxiv-context-assembly/main.tex b/docs/arxiv-context-assembly/main.tex index 2b777777..8fa75eae 100644 --- a/docs/arxiv-context-assembly/main.tex +++ b/docs/arxiv-context-assembly/main.tex @@ -356,8 +356,10 @@ \subsection{Reciprocal Rank Fusion and Hybrid Search} Client-side, FlashRank (ONNX cross-encoder) reranks the top-$3k$ candidates to produce the final ranking. -This pipeline is strong at moderate scale: 98.4\% R@10 on -LongMemEval, 94.2\% R@10 on LoCoMo (E1 v3, May 2026). The five-signal fusion +This pipeline is strong at moderate scale: 98.2\% R@10 on +LongMemEval (clean run, July 2026; code SHA \texttt{28145f0}) and +94.35\% R@10 on LoCoMo (E1 v3 post-fix run, May 2026; code SHA +\texttt{2f45bcb}). The five-signal fusion mitigates any single signal's weakness (\eg, vector similarity misses lexical matches that trigram catches; FTS misses paraphrases that vectors catch). But at BEAM-10M scale, all five signals suffer @@ -1291,8 +1293,11 @@ \subsection{Baselines} \paragraph{WRRF baseline.} Cortex's production pipeline without the assembler: 5-signal server-side fusion + FlashRank client-side reranking. This is a -strong baseline: 98.4\% R@10 on LongMemEval, 94.2\% R@10 on LoCoMo -(E1 v3, May 2026), and 0.591 MRR on BEAM-100K. +strong baseline: 98.2\% R@10 on LongMemEval (clean run, July 2026; +code SHA \texttt{28145f0}), 94.35\% R@10 on LoCoMo (E1 v3 post-fix +run, May 2026; code SHA \texttt{2f45bcb}), and 0.591 MRR on +BEAM-100K (five-conversation protocol, $n{=}100$; code SHA +\texttt{a071d89}). \paragraph{LIGHT} \citep{Tavakoli2026}. The strongest published system on BEAM, achieving 0.266 overall on diff --git a/docs/arxiv-thermodynamic/main.pdf b/docs/arxiv-thermodynamic/main.pdf index f0a3f7d0..484065e9 100644 Binary files a/docs/arxiv-thermodynamic/main.pdf and b/docs/arxiv-thermodynamic/main.pdf differ diff --git a/docs/arxiv-thermodynamic/main.tex b/docs/arxiv-thermodynamic/main.tex index cc55e78c..4df1b46d 100644 --- a/docs/arxiv-thermodynamic/main.tex +++ b/docs/arxiv-thermodynamic/main.tex @@ -47,8 +47,8 @@ \citep{Kandel2001,McClelland1995}, and (iv)~weighted score-fusion (WSF) retrieval with heat as a tie-breaker. On three independent long-term-memory -benchmarks, Cortex reaches LongMemEval R@10\,$=$\,98.4\% and -LoCoMo R@10\,$=$\,94.2\% (retrieval recall, directly comparable to +benchmarks, Cortex reaches LongMemEval R@10\,$=$\,98.2\% and +LoCoMo R@10\,$=$\,94.35\% (retrieval recall, directly comparable to the published retrieval baselines), and a retrieval-proxy MRR of 0.591 on BEAM-100K. We report the BEAM number as a retrieval-proxy (does the gold memory appear in the retrieved set), which is @@ -102,11 +102,11 @@ \section{Introduction} thermodynamic architecture, and reports empirical results on three published long-term-memory benchmarks. The headline numbers preview the argument: on LongMemEval \citep{Wu2025}, Cortex reaches -R@10\,$=$\,98.4\% versus 78.4\% paper-best; on BEAM +R@10\,$=$\,98.2\% versus 78.4\% paper-best; on BEAM \citep{Tavakoli2026}, a retrieval-proxy MRR of 0.591 on BEAM-100K (not comparable to BEAM's end-to-end metric; see \S\ref{sec:beam-proxy}); on -LoCoMo \citep{Maharana2024}, R@10\,$=$\,94.2\%. All numbers are +LoCoMo \citep{Maharana2024}, R@10\,$=$\,94.35\%. All numbers are produced by the production PL/pgSQL retrieval code path, single-process, on a clean database seeded only with each benchmark's data. @@ -131,8 +131,8 @@ \subsection{Contributions} decay prevents collapse: information-theoretic, operational (WSF tie-breaking), and curation-by-attrition. \item Empirical results on three long-term-memory benchmarks - (\S\ref{sec:empirical}): R@10\,$=$\,98.4\% on LongMemEval - ($+20.0$\,pp over paper-best), R@10\,$=$\,94.2\% on LoCoMo, + (\S\ref{sec:empirical}): R@10\,$=$\,98.2\% on LongMemEval + ($+19.8$\,pp over paper-best), R@10\,$=$\,94.35\% on LoCoMo, and a retrieval-proxy MRR of 0.591 on BEAM-100K. \end{enumerate} @@ -563,16 +563,17 @@ \section{Empirical Evidence} \begin{table}[t] \centering \caption{Cortex vs.\ paper-best on three long-term-memory benchmarks. -Cortex numbers: clean DB, single process, April 2026.} +Cortex numbers: clean DB, single process, protocol-specific runs; +sources and code SHAs are enumerated in the accompanying figure audit.} \label{tab:benchmarks} \begin{tabular}{llrrr} \toprule Benchmark & Venue & Metric & Cortex & Paper-best \\ \midrule -LongMemEval & ICLR 2025 & R@10 & \textbf{98.4\%} & 78.4\% \\ -LongMemEval & ICLR 2025 & MRR & \textbf{0.9124} & --- \\ -LoCoMo & ACL 2024 & R@10 & \textbf{94.2\%} & --- \\ -LoCoMo & ACL 2024 & MRR & \textbf{0.8278} & --- \\ +LongMemEval & ICLR 2025 & R@10 & \textbf{98.2\%} & 78.4\% \\ +LongMemEval & ICLR 2025 & MRR & \textbf{0.9167} & --- \\ +LoCoMo & ACL 2024 & R@10 & \textbf{94.35\%} & --- \\ +LoCoMo & ACL 2024 & MRR & \textbf{0.8279} & --- \\ BEAM-100K & ICLR 2026 & MRR (retrieval-proxy)$^\dagger$ & \textbf{0.591} & --- \\ \bottomrule \end{tabular} @@ -586,7 +587,7 @@ \section{Empirical Evidence} retrieval baselines.} \end{table} -The $+20.0$\,pp absolute gain on LongMemEval R@10 is the headline +The $+19.8$\,pp absolute gain on LongMemEval R@10 is the headline result against a published baseline; the BEAM-100K retrieval-proxy MRR of 0.591 is reported only for within-system comparison (\S\ref{sec:beam-proxy}), not against BEAM's incommensurable @@ -745,9 +746,10 @@ \subsection{Direct Dose--Response: Decay On vs.\ Off on BEAM \subsection{Per-mechanism evidence (LongMemEval-S, $n{=}500$)} \label{sec:per-mechanism} -The \S\ref{sec:empirical} table reports the integrated stack against -published baselines. This subsection decomposes the integrated number -into per-mechanism contributions on two benchmarks---LongMemEval-S +The \S\ref{sec:empirical} table reports the current integrated stack against +published baselines. This subsection reports a separate, earlier clean +E1~v3 ablation snapshot and decomposes that snapshot into per-mechanism +contributions on two benchmarks---LongMemEval-S at $n{=}500$, single seed (\S\S\ref{sec:per-mech-table}--\ref{sec:per-mech-architectural}) and LoCoMo at $n{=}1986$, single seed (\S\ref{sec:locomo-evidence})---at the calibrated equilibrium described in \S\ref{sec:calibration} below. @@ -763,12 +765,13 @@ \subsection{Per-mechanism evidence (LongMemEval-S, $n{=}500$)} Question-level bootstrap CIs for the headline rows are deferred to the camera-ready. -\paragraph{Headline against the established Cortex baseline.} At +\paragraph{Historical E1 v3 ablation snapshot.} At $n{=}500$ the calibrated integrated stack reaches \textbf{MRR $= 0.9124$} and \textbf{R@10 $= 0.984$} (artefact: \texttt{benchmarks/results/ablation/longmemeval-s\_v3/BASELINE.json}; manifest: \texttt{manifest.json}, code SHA \texttt{0e858e8}, dirty=false, -finished 2026-05-03). Against the previously established CLAUDE.md +finished 2026-05-03). These values belong to this named ablation +snapshot, not to the current headline run. Against the previously established CLAUDE.md reference (MRR $= 0.882$, R@10 $= 0.978$) this is \textbf{+3.0\% MRR and +0.6\% R@10}. The single-seed limitation of \S\ref{sec:empirical} applies; the per-row noise floor on $n{=}500$ is empirically @@ -975,16 +978,21 @@ \subsubsection{LoCoMo evidence: empirical resolution of the architectural-mismat \text{ablated}$. \paragraph{Headline.} \texttt{BASELINE\_NO\_CONSOLIDATION} reaches -MRR $= 0.8278$, R@10 $= 0.942$ on LoCoMo ($n = 1986$). Against the -established LoCoMo baseline (MRR $= 0.794$, R@10 $= 0.926$) this is -$+4.3\%$ MRR, $+1.6\%$ R@10. -\texttt{BASELINE\_WITH\_CONSOLIDATION} reaches MRR $= 0.8264$, -R@10 $= 0.940$ ($\Delta$ vs.\ NO $= +0.0014$, within the per-row +MRR $= 0.8279$, R@10 $= 0.9435$ on LoCoMo ($n = 1986$). Against the +historical April 2026 clean-DB Cortex comparator (MRR $= 0.794$, +R@10 $= 0.926$, $n = 1982$) this is $+4.3\%$ MRR and $+1.75$ +percentage points R@10.\footnote{The historical pair was first +published in repository commit \texttt{b4057a}; its original +per-query artefact was not committed, so it is retained only as a +historical comparator, not as the current baseline.} +\texttt{BASELINE\_WITH\_CONSOLIDATION} reaches MRR $= 0.8265$, +R@10 $= 0.9410$ ($\Delta$ vs.\ NO $= +0.0014$, within the per-row noise floor). The two anchors agreeing at full $n$ confirms that the consolidation cadence fix described in \S\ref{sec:cadence-fix} holds -on the full benchmark, not only on smoke. Artefacts at -\texttt{benchmarks/results/ablation/locomo\_v3/}; manifest code SHA -\texttt{ef178da}, dirty=false, finished 2026-05-03. +on the full benchmark, not only on smoke. The authoritative run +record is \texttt{docs/benchmarks/e1-v3-locomo-results-post-fix.md}; +launch code SHA \texttt{2f45bcb39dbe15fa0ef857cc8c8c3783175d05db}, +dirty=false, finished 2026-05-04. \begin{center} \small @@ -992,20 +1000,20 @@ \subsubsection{LoCoMo evidence: empirical resolution of the architectural-mismat \toprule Mechanism & MRR (abl.) & R@10 (abl.) & $\Delta$MRR & $\Delta$R@10 & Anchor \\ \midrule -BASELINE\_NO\_CONSOLIDATION & 0.8278 & 0.942 & 0 & 0 & self \\ -RECONSOLIDATION & 0.8202 & 0.931 & \textbf{$+$0.0076} & $+$0.011 & NO \\ -\emph{CO\_ACTIVATION} & 0.8268 & 0.940 & $\approx 0$ & $+$0.001 & NO \\ -ADAPTIVE\_DECAY & 0.8441 & 0.962 & \textbf{$-$0.0163} & $-$0.020 & NO \\ -BASELINE\_WITH\_CONSOLIDATION & 0.8264 & 0.940 & 0 & 0 & self \\ -\emph{CASCADE} & 0.8272 & 0.941 & $\approx 0$ & $-$0.001 & WITH \\ -\emph{INTERFERENCE} & 0.8260 & 0.939 & $\approx 0$ & $+$0.001 & WITH \\ -HOMEOSTATIC\_PLASTICITY & 0.8289 & 0.945 & \textbf{$-$0.0025} & $-$0.005 & WITH \\ -\emph{SYNAPTIC\_PLASTICITY} & 0.8264 & 0.940 & $\approx 0$ & 0 & WITH \\ -\emph{MICROGLIAL\_PRUNING} & 0.8253 & 0.939 & $\approx 0$ & $+$0.001 & WITH \\ -\emph{TWO\_STAGE\_MODEL} & 0.8276 & 0.941 & $\approx 0$ & $-$0.001 & WITH \\ -\emph{EMOTIONAL\_DECAY} & 0.8249 & 0.940 & $\approx 0$ & $\phantom{-}0.000$ & WITH \\ -\emph{TRIPARTITE\_SYNAPSE} & 0.8268 & 0.941 & $\approx 0$ & $-$0.001 & WITH \\ -\emph{SCHEMA\_ENGINE} & 0.8268 & 0.941 & $\approx 0$ & $-$0.001 & WITH \\ +BASELINE\_NO\_CONSOLIDATION & 0.8279 & 0.9435 & 0 & 0 & self \\ +RECONSOLIDATION & 0.8188 & 0.9289 & \textbf{$+$0.0091} & $+$0.0146 & NO \\ +\emph{CO\_ACTIVATION} & 0.8264 & 0.9400 & $+$0.0015 & $+$0.0035 & NO \\ +ADAPTIVE\_DECAY & 0.8442 & 0.9622 & \textbf{$-$0.0163} & $-$0.0187 & NO \\ +BASELINE\_WITH\_CONSOLIDATION & 0.8265 & 0.9410 & 0 & 0 & self \\ +\emph{CASCADE} & 0.8268 & 0.9425 & $-$0.0002 & $-$0.0015 & WITH \\ +\emph{INTERFERENCE} & 0.8271 & 0.9410 & $-$0.0005 & $\phantom{-}0.0000$ & WITH \\ +\emph{HOMEOSTATIC\_PLASTICITY} & 0.8248 & 0.9390 & $+$0.0017 & $+$0.0020 & WITH \\ +\emph{SYNAPTIC\_PLASTICITY} & 0.8269 & 0.9405 & $-$0.0003 & $+$0.0005 & WITH \\ +\emph{MICROGLIAL\_PRUNING} & 0.8269 & 0.9420 & $-$0.0004 & $-$0.0010 & WITH \\ +\emph{TWO\_STAGE\_MODEL} & 0.8267 & 0.9395 & $-$0.0002 & $+$0.0015 & WITH \\ +\emph{EMOTIONAL\_DECAY} & 0.8263 & 0.9415 & $+$0.0002 & $-$0.0005 & WITH \\ +\emph{TRIPARTITE\_SYNAPSE} & 0.8266 & 0.9415 & $-$0.0001 & $-$0.0005 & WITH \\ +\emph{SCHEMA\_ENGINE} & 0.8249 & 0.9395 & $+$0.0017 & $+$0.0015 & WITH \\ \bottomrule \end{tabular}\\[2pt] {\footnotesize Bold $=$ above noise floor ($|\Delta\text{MRR}| > @@ -1023,8 +1031,8 @@ \subsubsection{LoCoMo evidence: empirical resolution of the architectural-mismat \toprule Mechanism & LME-S $\Delta$MRR & LoCoMo $\Delta$MRR & Resolution \\ \midrule -RECONSOLIDATION & $+$0.0000 & $+$0.0076 & Confirmed: fires on multi-session recall \\ -CO\_ACTIVATION & $+$0.0000 & $+$0.0010 & Confirmed; smaller magnitude \\ +RECONSOLIDATION & $+$0.0000 & $+$0.0091 & Confirmed: fires on multi-session recall \\ +CO\_ACTIVATION & $+$0.0000 & $+$0.0015 & Confirmed; smaller magnitude \\ ADAPTIVE\_DECAY & $-$0.0014 & $-$0.0163 & Same sign, amplified $\sim 11\times$ \\ \bottomrule \end{tabular} @@ -1040,13 +1048,13 @@ \subsubsection{LoCoMo evidence: empirical resolution of the architectural-mismat \paragraph{Top contributors per anchor group.} In the longitudinal-read-path group, ADAPTIVE\_DECAY ($|\Delta\text{MRR}| = 0.0163$, counterproductive) and RECONSOLIDATION ($\Delta\text{MRR} = -+0.0076$, positive) dominate; CO\_ACTIVATION ($+0.0010$) is ++0.0091$, positive) dominate; CO\_ACTIVATION ($+0.0015$) is consistent-sign but at the per-row noise floor. In the -consolidation-only group, all nine deltas sit within the per-row -noise floor ($\approx \pm 0.002$ MRR at $n = 1986$ single-seed); -HOMEOSTATIC\_PLASTICITY ($-0.0025$) is the largest absolute, -EMOTIONAL\_DECAY ($+0.0015$) and TWO\_STAGE\_MODEL ($-0.0012$) -follow. The honest reading of the consolidation-only group is that +consolidation-only group, every delta is within the per-row noise +floor ($\approx \pm 0.002$ MRR at $n = 1986$ single-seed); +HOMEOSTATIC\_PLASTICITY and SCHEMA\_ENGINE (both $+0.0017$) have the +largest absolute magnitudes but remain at the noise boundary. The +honest reading of the consolidation-only group is that the consolidation pipeline as a whole contributes (the cadence fix narrative in \S\ref{sec:cadence-fix} is not undone by these deltas), but no single consolidation-time mechanism dominates at LoCoMo's @@ -1054,18 +1062,13 @@ \subsubsection{LoCoMo evidence: empirical resolution of the architectural-mismat \S\ref{sec:per-mechanism} already documented for LME-S. \paragraph{Limitations of the LoCoMo run.} Single-seed at -$n = 1986$; per-row noise floor $\approx \pm 0.002$ MRR. The -plasticity result-shape contract bug fixed in commit \texttt{5f737fe} -(\S\ref{sec:plasticity-fix} below) was discovered \emph{during} the -LoCoMo sweep and the run was launched on bytes pre-fix; the -\texttt{BASELINE\_WITH} and the nine consolidation-only rows -therefore ran with a logged-WARNING (not a crash) that may have -muted some consolidation deltas. The three longitudinal-read-path -rows ran with consolidation off, are not affected by the plasticity -bug, and constitute the empirical resolution finding above. The -follow-up re-run of the consolidation-only group on -post-\texttt{5f737fe} bytes is reported in -\S\ref{sec:locomo-postfix}. +$n = 1986$; per-row noise floor $\approx \pm 0.002$ MRR. The run is +on bytes that include both the cadence fix \texttt{6c51bce} and the +plasticity result-shape fix \texttt{5f737fe}. Magnitudes below the +noise floor are not interpreted causally. The predecessor sweep on +pre-\texttt{5f737fe} bytes is retained only for the transparent +pre-vs-post comparison in \S\ref{sec:locomo-postfix}; it is not the +source of the current headline or table. \subsubsection{Calibration rigor: Phase~A and Phase~B} \label{sec:calibration} @@ -1618,11 +1621,12 @@ \subsection{Why Decay Restores Discriminability} \subsection{Concrete Numbers (LongMemEval R@10)} \label{app:shannon-numbers} -Cortex measured (clean DB, April 2026): +Cortex measured in the current clean-DB run (2026-07-14; code SHA +\texttt{28145f0}; dirty=false): \begin{itemize} - \item LongMemEval R@10: \textbf{98.4\%} + \item LongMemEval R@10: \textbf{98.2\%} \item Best flat-RAG baseline (paper-best): \textbf{78.4\%} - \item Gap: \textbf{20.0\,pp}. + \item Gap: \textbf{19.8\,pp}. \end{itemize} LongMemEval-S has a \emph{per-query} candidate haystack of @@ -1849,7 +1853,8 @@ \subsection{Connection to LongMemEval} best $R@10 = 78.4\%$ matches: roughly $1 - 10^4/10^5 = 0.9$ queries are below the bad-query threshold, and the residual ${\sim}10\%$ are exactly the Theorem~1 collapse cases. Cortex measures -$R@10 = 98.4\%$. The 20.0-point gap is the predicted Zipf rescue. +$R@10 = 98.2\%$ in the clean run named above. The 19.8-point gap is +the predicted Zipf rescue. \subsection{Existence of an Optimal Decay Exponent} \label{app:erdos-thm3} @@ -1919,7 +1924,7 @@ \subsection{Summary of Bounds} our measurements: the clean-store dose--response and Zipf controls (\S\ref{sec:decay-dose}, \S\ref{sec:regime}) find no crossover up to $N = 10^5$ under single-instant benchmark ingestion, and the -observed $78.4\% \to 98.4\%$ LongMemEval improvement is carried by +observed $78.4\% \to 98.2\%$ LongMemEval improvement is carried by the integrated stack (fusion, gating, consolidation), not by decay (per-mechanism ablation, \S\ref{sec:per-mechanism}). The theorem's premises---a store with genuinely differential access history @@ -2039,10 +2044,10 @@ \subsection{C2 --- Heat Decay Alone is Sufficient to Prevent Collapse (severity: in a store with genuinely differential live access history, which no current benchmark instantiates. -\subsection{C5 --- 98.4\% LongMemEval R@10 Generalizes Beyond the Benchmark (severity: high)} +\subsection{C5 --- 98.2\% LongMemEval R@10 Generalizes Beyond the Benchmark (severity: high)} \label{app:popper-c5} -\textbf{The claim.} ``98.4\% LongMemEval R@10 generalizes beyond +\textbf{The claim.} ``98.2\% LongMemEval R@10 generalizes beyond the benchmark---the system is not overfit.'' \textbf{Riskiest prediction.} Calibration parameters tuned on @@ -2050,10 +2055,14 @@ \subsection{C5 --- 98.4\% LongMemEval R@10 Generalizes Beyond the Benchmark (sev reranker depth) transfer to LoCoMo without retuning, retaining $\ge 90\%$ of LongMemEval's R@10 (i.e., LoCoMo R@10 $\ge 0.88$). -\textbf{Current evidence.} \textbf{PARTIALLY TESTED.} Each -benchmark is currently scored independently with whatever defaults -exist at the time, but the calibration history is not version-pinned -to a single benchmark. +\textbf{Current evidence.} \textbf{PARTIALLY TESTED.} The current +protocol-specific scores are LongMemEval R@10 $=98.2\%$ (MRR $=0.9167$, +$n{=}500$, clean run at SHA \texttt{28145f0}), LoCoMo R@10 $=94.35\%$ +(MRR $=0.8279$, $n{=}1986$, clean post-fix run at SHA +\texttt{2f45bcb}), and BEAM-100K retrieval-proxy MRR $=0.591$ +($n{=}100$, five-conversation run at SHA \texttt{a071d89}). They were +scored independently, so they are consistent with generalization but +do not constitute the frozen-config cross-evaluation required by this claim. \subsection{C4 --- WSF Fusion of 5 Signals Decorrelates Noise Sources (severity: medium)} \label{app:popper-c4} @@ -2103,7 +2112,7 @@ \subsection{Summary} No claim in the paper is currently CONFIRMED in the strict Popperian sense (survived a severe test designed to refute it). -The 98.4\% / 94.2\% / 0.591 numbers are corroborations of the +The 98.2\% / 94.35\% / 0.591 numbers are corroborations of the system as configured, not corroborations of the causal claims (C1--C6) about \emph{why} it performs. @@ -2256,8 +2265,8 @@ \subsection{Verification surfaced a production fix: consolidation cadence} The fix was validated on smoke first; the \S\ref{sec:locomo-evidence} LoCoMo run is the $n = 1986$ validation. At full scale, -\texttt{BASELINE\_WITH\_CONSOLIDATION} reaches MRR $= 0.8264$ -against \texttt{BASELINE\_NO\_CONSOLIDATION} at MRR $= 0.8278$ +\texttt{BASELINE\_WITH\_CONSOLIDATION} reaches MRR $= 0.8265$ +against \texttt{BASELINE\_NO\_CONSOLIDATION} at MRR $= 0.8279$ ($\Delta$ vs.\ NO $= +0.0014$, within the per-row noise floor of $\approx \pm 0.002$ MRR). The two anchors agree at full $n$; the cadence fix holds, and the \S\ref{sec:locomo-evidence} @@ -2289,8 +2298,8 @@ \subsection{Verification surfaced a second production fix: plasticity result-sha compliance for the disabled path. Regression tests lock the result-shape invariant. -The \S\ref{sec:locomo-evidence} LoCoMo run was launched on bytes -\emph{before} commit \texttt{5f737fe}, which means the +The predecessor to the \S\ref{sec:locomo-evidence} LoCoMo run was +launched on bytes \emph{before} commit \texttt{5f737fe}, which means the consolidation-only ablation rows (CASCADE, INTERFERENCE, HOMEOSTATIC\_PLASTICITY, MICROGLIAL\_PRUNING, TWO\_STAGE\_MODEL, EMOTIONAL\_DECAY, TRIPARTITE\_SYNAPSE, SCHEMA\_ENGINE) may have a @@ -2310,9 +2319,9 @@ \subsection{Verification surfaced a second production fix: plasticity result-sha does not change as a result: the architectural-mismatch resolution holds (RECONSOLIDATION $\Delta$MRR $= +0.0091$, ADAPTIVE\_DECAY $\Delta$MRR $= -0.0163$ on clean bytes); two consolidation-only rows -(HOMEOSTATIC\_PLASTICITY, SCHEMA\_ENGINE) move from noise/negative to -positive-out-of-noise, recovering the contribution the contract bug -had silently muted; the cadence-fix anchor agreement +(HOMEOSTATIC\_PLASTICITY, SCHEMA\_ENGINE) move from negative/noise to +positive direction while remaining inside the post-fix MRR noise +floor; the cadence-fix anchor agreement (\S\ref{sec:cadence-fix}) is re-validated identically ($\Delta$ vs.\ NO $= +0.0014$). We declare both the bug and the re-run rather than amend silently because the verification campaign's @@ -2331,9 +2340,8 @@ \subsection{The plasticity-fix re-run on post-\texttt{5f737fe} bytes} The \S\ref{sec:plasticity-fix} plasticity result-shape contract bug was fixed in commit \texttt{5f737fe}; the same 14-row two-baseline sweep was re-run at full $n = 1986$ on a descendant SHA -(\texttt{2f45bcb}, dirty=false, finished 2026-05-04). Artefacts at -\texttt{benchmarks/results/ablation/locomo\_v3\_post\_plasticity\_fix/}; -detailed writeup at +(\texttt{2f45bcb}, dirty=false, finished 2026-05-04). The committed +run record and detailed writeup are at \texttt{docs/benchmarks/e1-v3-locomo-results-post-fix.md}. \paragraph{Headline.} \texttt{BASELINE\_NO\_CONSOLIDATION} reaches @@ -2348,8 +2356,8 @@ \subsection{The plasticity-fix re-run on post-\texttt{5f737fe} bytes} plasticity bug had no opportunity to exercise. \paragraph{Pre-vs-post-fix comparison.} The consolidation-only group -has three sign-flips relative to the pre-fix sweep, two of them -moving out of noise: +has three sign-flips relative to the pre-fix sweep; all post-fix +consolidation-only deltas remain inside the $\pm 0.002$ MRR floor: \begin{center} \small @@ -2368,29 +2376,25 @@ \subsection{The plasticity-fix re-run on post-\texttt{5f737fe} bytes} \end{tabular} \end{center} -Two rows (HOMEOSTATIC\_PLASTICITY at $+0.0042$, -SCHEMA\_ENGINE at $+0.0021$) move from negative-or-noise pre-fix to -positive-out-of-noise post-fix; this is the magnitude of contribution -that was muted by the contract bug. Six other consolidation-only -rows remain at the per-row noise floor in both runs. The +HOMEOSTATIC\_PLASTICITY and SCHEMA\_ENGINE move by $+0.0042$ and +$+0.0021$ respectively between runs and end at positive-direction +$\Delta$MRR $= +0.0017$, still within the post-fix noise floor. Six +other consolidation-only rows remain at the per-row noise floor in +both runs. The architectural-mismatch resolution (RECONSOLIDATION $\Delta$MRR $= +0.0091$, ADAPTIVE\_DECAY $\Delta$MRR $= -0.0163$) is -\textbf{strengthened} on clean bytes---the dominant longitudinal -contributions are intact and slightly larger in magnitude, as -predicted in \S\ref{sec:locomo-evidence} (the longitudinal rows were -never affected by the plasticity bug, so any movement is attributable -to single-seed run-to-run variance at the noise floor; the magnitudes -here move with that variance, not against the hypothesis). +\textbf{re-confirmed} on clean bytes---the dominant longitudinal +contributions are intact; their small between-run movement is +single-seed variation. \paragraph{The verification self-correcting.} \S\ref{sec:plasticity-fix} declared the plasticity-shape bug as a possible source of mute on the consolidation-only group and committed -to a follow-up re-run. This subsection is that re-run. Two -consolidation-only rows (HOMEOSTATIC\_PLASTICITY, SCHEMA\_ENGINE) -recover positive contributions that were silently masked; the -cadence-fix anchor agreement is re-confirmed; the load-bearing -architectural-mismatch finding is re-confirmed and slightly -strengthened. The verification campaign now comprises three +to a follow-up re-run. This subsection is that re-run. +HOMEOSTATIC\_PLASTICITY and SCHEMA\_ENGINE recover positive-direction +deltas at the noise boundary; the cadence-fix anchor agreement and +the load-bearing architectural-mismatch finding are re-confirmed. +The verification campaign now comprises three artefact sets at full $n$ (LME-S 17 rows + LoCoMo 14 rows pre-fix + LoCoMo 14 rows post-fix $=$ 45 per-mechanism evidence rows on the appropriate benchmark for each mechanism's mechanism-of-action), diff --git a/docs/benchmarks/arxiv-figure-audit-2026-08-02.md b/docs/benchmarks/arxiv-figure-audit-2026-08-02.md new file mode 100644 index 00000000..bf74ddfb --- /dev/null +++ b/docs/benchmarks/arxiv-figure-audit-2026-08-02.md @@ -0,0 +1,35 @@ +# arXiv benchmark-figure audit — 2026-08-02 + +This audit records the pre-submission check requested in issue #347. It covers the benchmark figures in `docs/arxiv-thermodynamic/main.tex`, `docs/arxiv-context-assembly/main.tex`, and their Markdown source documents. + +## Authoritative run records + +| Benchmark | Current figure | Run record | Code / protocol | +|---|---:|---|---| +| LongMemEval-S, current headline | MRR 0.9167, R@10 0.982, n=500 | `benchmarks/results/repro/20260714-v4.14.1-pretag/longmemeval-s.json` | code SHA `28145f0b7a113fc06e22568de6feea7f8444eaf5`, dirty=false, 2026-07-14 | +| LongMemEval-S, E1 v3 ablation snapshot | MRR 0.9124, R@10 0.984, n=500 | `docs/benchmarks/e1-v3-results.md`; `benchmarks/results/ablation/longmemeval-s_v3/` | code SHA `0e858e8db0f8a5dae0879fa0134113d101be19f8`, dirty=false, 2026-05-03; historical protocol-specific decomposition, not the current headline | +| LoCoMo | MRR 0.8279, R@10 0.9435, n=1986 | `docs/benchmarks/e1-v3-locomo-results-post-fix.md` | code SHA `2f45bcb39dbe15fa0ef857cc8c8c3783175d05db`, dirty=false, descendant of `5f737fe` | +| BEAM-100K, flat WRRF | retrieval-proxy MRR 0.591, R@10 0.790, n=100 | `benchmarks/results/a3_beam_100k_post_refactor.md` and `benchmarks/beam/variance/baseline_limit5.txt` | code SHA `a071d89`, five-conversation A/B protocol; result committed by `544abe7` | +| BEAM-100K, assembler | MRR 0.602, n=100 | `benchmarks/beam/variance/assembler_limit5.txt` | same five-conversation A/B protocol | +| BEAM-10M, flat / oracle / temporal | MRR 0.353 / 0.429 / 0.471, n=196 | `benchmarks/beam/variance/baseline_10m_fixed.txt`, `assembler_10m_stagefixed.txt`, `assembler_10m_temporal.txt` | original paired protocol | +| BEAM-10M reproduction, oracle / temporal | MRR 0.496 / 0.523, n=196 | `benchmarks/results/beam10m_paired/RESULTS.md` | later paired code revision; compare within this pair only | +| BEAM-500K / 1M crossover | flat 0.500 / 0.466; assembler 0.570 / 0.535 | `benchmarks/results/beam_crossover/RESULTS.md` | clean DB, 35 conversations per split | + +## Findings and resolution + +- **LongMemEval:** Opus 5's review found that the first audit had incorrectly promoted the May E1 v3 ablation baseline (MRR 0.9124, R@10 98.4%) to the current headline while `README.md` carried 98.2%. The current headline is now the latest committed clean run with an explicit clean flag: MRR 0.9167, R@10 98.2%, n=500, code SHA `28145f0`. The E1 v3 values remain only inside their named historical ablation snapshot and its per-row analysis. +- **LoCoMo:** the thermodynamic paper's headline, benchmark table, 14-row ablation table, contributor narrative, cadence appendix, and both context-assembly citations still used the pre-fix 0.8278 / 0.942 run. They now use the post-fix 0.8279 / 0.9435 run and its per-row deltas. Pre-fix values remain only in explicitly labelled historical comparisons. +- **Historical 0.794 / 0.926 comparator:** this pair is the April 2026 clean-DB Cortex result (n=1982) first published in commit `b4057a`. Its original per-query artefact is not committed. All active citations now label it as a superseded historical comparator rather than attributing it to the current `CLAUDE.md`. +- **BEAM:** the arXiv LaTeX headline already had the correct 0.591 retrieval-proxy MRR from the named five-conversation / 100-question protocol at code SHA `a071d89`. The standalone Popper appendix's stale 0.543 was replaced, and the thermodynamic Markdown source now matches the LaTeX paper by removing the invalid comparison against BEAM's incommensurable end-to-end 0.329 score. The later 395-question full split remains a separate protocol. +- **Publication surfaces:** the thermodynamic and context-assembly LaTeX sources, their Markdown sources, the Popper/Shannon/Erdős appendices, the README benchmark table, and the arXiv endorsement drafts now use the current LongMemEval and LoCoMo headlines. Historical values are retained only with a named run or an explicit historical label. +- **Noise-floor language:** the post-fix LoCoMo consolidation-only values are all within the stated ±0.002 MRR floor. HOMEOSTATIC_PLASTICITY and SCHEMA_ENGINE end at +0.0017; the papers now describe these as positive-direction observations within noise, not causal contributions outside noise. + +## Submission gate + +The source-level figure audit passes when: + +1. the stale-number grep contains no unlabeled LoCoMo headline (`0.805`, `91.5%`, `94.2%`) or current LongMemEval headline (`97.8%`, `98.4%`), and every retained occurrence is explicitly historical, belongs to a named protocol, or represents a different metric; +2. the two LaTeX sources compile without undefined references or citations; +3. `git diff --check` passes. + +**Result: PASS after Opus 5 follow-up on 2026-08-02.** The stale-number grep has no unlabeled current LoCoMo or LongMemEval headline; both PDFs were rebuilt with `pdflatex`/`bibtex` and the final passes contain no undefined references or citations; the 14-row LoCoMo values were checked across the post-fix writeup, Markdown paper, and LaTeX source; `scripts/check_doc_claims.py` and `git diff --check` pass. diff --git a/docs/benchmarks/e1-v3-locomo-results-post-fix.md b/docs/benchmarks/e1-v3-locomo-results-post-fix.md index 998c823c..f248ba74 100644 --- a/docs/benchmarks/e1-v3-locomo-results-post-fix.md +++ b/docs/benchmarks/e1-v3-locomo-results-post-fix.md @@ -3,7 +3,7 @@ ## Headline - **Cortex BASELINE_NO_CONSOLIDATION (longitudinal-read-path anchor): MRR = 0.8279, R@10 = 0.9435** on LoCoMo (n = 1986). -- vs. CLAUDE.md established LoCoMo baseline (MRR = 0.794, R@10 = 0.926): **+4.3% MRR, +1.7% R@10** — within rounding identical to the pre-fix sweep, as expected (the longitudinal-read-path rows ran with consolidation off in both sweeps and the plasticity bug cannot exercise there). +- vs. the historical April 2026 clean-DB Cortex comparator (MRR = 0.794, R@10 = 0.926; n=1982; provenance below): **+4.3% MRR, +1.75 percentage points R@10** — directionally identical to the pre-fix sweep, as expected (the longitudinal-read-path rows ran with consolidation off in both sweeps and the plasticity bug cannot exercise there). - **BASELINE_WITH_CONSOLIDATION (consolidation-cadence anchor): MRR = 0.8265, R@10 = 0.941.** ΔvsNO = +0.0014, **identical to the pre-fix value**, within the per-row noise floor. The cadence fix (commit `6c51bce`) re-validated at full n = 1986 a second time on post-`5f737fe` bytes. - The 14-row two-baseline ablation **re-confirms** the architectural-mismatch resolution from the pre-fix writeup (`docs/benchmarks/e1-v3-locomo-results.md`): RECONSOLIDATION ΔMRR = +0.0091, ADAPTIVE_DECAY ΔMRR = -0.0163. The longitudinal-read-path group is unchanged; the consolidation-only group has small sign flips on three rows (HOMEOSTATIC_PLASTICITY, SCHEMA_ENGINE, SYNAPTIC_PLASTICITY) — see "Pre-vs-post-fix comparison" below. @@ -43,13 +43,13 @@ Positive Δ ⇒ mechanism contributes positively (ablating it hurts). Negative | BASELINE_WITH_CONSOLIDATION | 0.8265 | 0.9410 | 0 | 0 | self | Reference (consolidation-cadence anchor); ΔvsNO = +0.0014, within noise — cadence fix `6c51bce` re-validated | | CASCADE | 0.8268 | 0.9425 | -0.0002 | -0.0015 | WITH | Within noise floor | | INTERFERENCE | 0.8271 | 0.9410 | -0.0005 | 0.0000 | WITH | Within noise floor | -| HOMEOSTATIC_PLASTICITY | 0.8248 | 0.9390 | +0.0017 | +0.0020 | WITH | **Sign flipped** vs pre-fix (-0.0025 → +0.0017); positive contribution unmasked once plasticity ran cleanly | +| HOMEOSTATIC_PLASTICITY | 0.8248 | 0.9390 | +0.0017 | +0.0020 | WITH | **Sign flipped** vs pre-fix (-0.0025 → +0.0017); positive direction, within the MRR noise floor | | SYNAPTIC_PLASTICITY | 0.8269 | 0.9405 | -0.0003 | +0.0005 | WITH | Null contribution (clean: ablation explicitly disables plasticity) | | MICROGLIAL_PRUNING | 0.8269 | 0.9420 | -0.0004 | -0.0010 | WITH | Within noise floor (sign flipped from +0.0011 but |Δ| at noise floor) | | TWO_STAGE_MODEL | 0.8267 | 0.9395 | -0.0002 | +0.0015 | WITH | Within noise floor | | EMOTIONAL_DECAY | 0.8263 | 0.9415 | +0.0002 | -0.0005 | WITH | Within noise floor | | TRIPARTITE_SYNAPSE | 0.8266 | 0.9415 | -0.0001 | -0.0005 | WITH | Within noise floor | -| SCHEMA_ENGINE | 0.8249 | 0.9395 | +0.0017 | +0.0015 | WITH | **Sign flipped** vs pre-fix (-0.0004 → +0.0017); positive contribution unmasked | +| SCHEMA_ENGINE | 0.8249 | 0.9395 | +0.0017 | +0.0015 | WITH | **Sign flipped** vs pre-fix (-0.0004 → +0.0017); positive direction, within the MRR noise floor | (Exact 6-decimal values at `benchmarks/results/ablation/locomo_v3_post_plasticity_fix/.json::overall_mrr` and `manifest.rows`.) @@ -66,15 +66,15 @@ The plasticity result-shape contract bug (commit `5f737fe`) silently dropped pla | BASELINE_WITH_CONSOLIDATION | 0 (anchor) | 0 (anchor) | — | ΔvsNO = +0.0014 in both runs — cadence fix re-confirmed | | CASCADE | -0.0008 | -0.0002 | +0.0006 | Within noise; closer to zero | | INTERFERENCE | +0.0004 | -0.0005 | -0.0009 | Within noise; sign flipped at noise floor | -| **HOMEOSTATIC_PLASTICITY** | **-0.0025** | **+0.0017** | **+0.0042** | **Sign flipped** — plasticity-bug-muted negative reading was an artefact; with clean plasticity, this row contributes positively | +| **HOMEOSTATIC_PLASTICITY** | **-0.0025** | **+0.0017** | **+0.0042** | **Sign flipped** — the between-run movement exceeds the floor, but the post-fix delta remains within the ±0.002 MRR floor | | SYNAPTIC_PLASTICITY | 0.0000 | -0.0003 | -0.0003 | Within noise; explicitly clean ablation | | MICROGLIAL_PRUNING | +0.0011 | -0.0004 | -0.0015 | Within noise; sign flipped at noise floor | | TWO_STAGE_MODEL | -0.0012 | -0.0002 | +0.0010 | Within noise; closer to zero | | EMOTIONAL_DECAY | +0.0015 | +0.0002 | -0.0013 | Within noise; closer to zero | | TRIPARTITE_SYNAPSE | -0.0004 | -0.0001 | +0.0003 | Within noise; near-identical | -| **SCHEMA_ENGINE** | **-0.0004** | **+0.0017** | **+0.0021** | **Sign flipped** — small but consistent-direction unmasking, mirrors HOMEOSTATIC_PLASTICITY | +| **SCHEMA_ENGINE** | **-0.0004** | **+0.0017** | **+0.0021** | **Sign flipped** — positive direction after the fix, still within the post-fix MRR noise floor | -**Reading.** Three sign-flips (HOMEOSTATIC_PLASTICITY, SCHEMA_ENGINE, SYNAPTIC_PLASTICITY) of which two (HOMEOSTATIC_PLASTICITY at +0.0042, SCHEMA_ENGINE at +0.0021) move out of noise and toward positive contribution. The longitudinal-read-path group (RECONSOLIDATION, CO_ACTIVATION, ADAPTIVE_DECAY) is essentially identical between runs because those rows ran with consolidation off and the plasticity bug had no opportunity to exercise — exactly as documented in the pre-fix limitations note. The cadence-fix anchor agreement (ΔvsNO = +0.0014) is identical to 4 decimals in both runs. +**Reading.** Three rows change sign (HOMEOSTATIC_PLASTICITY, SCHEMA_ENGINE, SYNAPTIC_PLASTICITY). HOMEOSTATIC_PLASTICITY and SCHEMA_ENGINE move by +0.0042 and +0.0021 between runs, ending at positive-direction ΔMRR = +0.0017; because |+0.0017| < 0.002, both post-fix row effects remain inside the stated MRR noise floor. The longitudinal-read-path group (RECONSOLIDATION, CO_ACTIVATION, ADAPTIVE_DECAY) is essentially identical between runs because those rows ran with consolidation off and the plasticity bug had no opportunity to exercise. The cadence-fix anchor agreement (ΔvsNO = +0.0014) is identical to 4 decimals in both runs. ## Architectural-mismatch hypothesis: re-confirmed on clean bytes @@ -96,11 +96,11 @@ The architectural-mismatch hypothesis (longitudinal mechanisms are foreclosed on **Consolidation-only (anchor: BASELINE_WITH_CONSOLIDATION)** -1. **HOMEOSTATIC_PLASTICITY: ΔMRR = +0.0017** (largest absolute; sign-flipped from pre-fix; positive contribution unmasked). +1. **HOMEOSTATIC_PLASTICITY: ΔMRR = +0.0017** (largest absolute; sign-flipped from pre-fix; positive direction within noise). 2. **SCHEMA_ENGINE: ΔMRR = +0.0017** (tied largest absolute; sign-flipped from pre-fix). 3. **INTERFERENCE: ΔMRR = -0.0005** (within noise floor; reported for completeness). -The consolidation-only group's deltas all sit at or just outside the per-row noise floor (≈ ±0.002 MRR at n = 1986 single-seed). The two newly-unmasked positive contributions (HOMEOSTATIC_PLASTICITY, SCHEMA_ENGINE) sit at the boundary of noise and effect; they are reportable as positive-direction contributions but no single consolidation-time mechanism dominates at LoCoMo's scale, the same calibrated-stack property documented for LME-S and the pre-fix LoCoMo run. +The consolidation-only group's deltas all sit inside the per-row noise floor (≈ ±0.002 MRR at n = 1986 single-seed). HOMEOSTATIC_PLASTICITY and SCHEMA_ENGINE are positive-direction observations, not causal contributions at this precision; no single consolidation-time mechanism dominates at LoCoMo's scale. ## Limitations and honest framing @@ -124,6 +124,10 @@ The consolidation-only group's deltas all sit at or just outside the per-row noi - **Summary CSV:** `summary.csv` (14 rows, anchor assignments, per-row delta_mrr_vs_anchor / delta_r10_vs_anchor). - **Total artefacts:** 14 row JSONs + 1 manifest + 1 summary = 16. +## Historical comparator provenance + +The MRR = 0.794 / R@10 = 0.926 pair is **not the current baseline** and is no longer sourced from `CLAUDE.md`. It is the historical clean-database, per-conversation-isolated LoCoMo result (n=1982) first published in commit [`b4057a`](https://github.com/cdeust/Cortex/commit/b4057a532da9c0eecd63ba72ad4174f5e54dab1c). The original per-query run artefact was not committed, so the pair is retained only as a transparent historical comparator; the post-fix run above is authoritative for current Cortex figures. + ## Sources - LoCoMo (Maharana et al., ACL 2024) — benchmark. diff --git a/docs/benchmarks/e1-v3-locomo-results.md b/docs/benchmarks/e1-v3-locomo-results.md index badf3d73..b02fc493 100644 --- a/docs/benchmarks/e1-v3-locomo-results.md +++ b/docs/benchmarks/e1-v3-locomo-results.md @@ -1,9 +1,12 @@ # E1 v3 — LoCoMo Ablation Results (n=1986, 14 rows) +> [!WARNING] +> **Superseded for current figures.** This sweep ran on code SHA `ef178da7418a05bcf7aeb3e66f5b3179fdad2c4d`, before the plasticity result-shape fix `5f737fe`. Use the [post-fix re-run](e1-v3-locomo-results-post-fix.md) (code SHA `2f45bcb39dbe15fa0ef857cc8c8c3783175d05db`) for every current LoCoMo value and per-mechanism delta. This document is retained as historical pre-fix evidence only. + ## Headline - **Cortex BASELINE_NO_CONSOLIDATION (longitudinal-read-path anchor): MRR = 0.8278, R@10 = 0.942** on LoCoMo (n = 1986). -- vs. CLAUDE.md established LoCoMo baseline (MRR = 0.794, R@10 = 0.926): **+4.3% MRR, +1.6% R@10**. +- vs. the historical April 2026 clean-DB Cortex comparator (MRR = 0.794, R@10 = 0.926; n=1982; provenance and limitations in the [post-fix re-run](e1-v3-locomo-results-post-fix.md#historical-comparator-provenance)): **+4.3% MRR, +1.6% R@10**. - **BASELINE_WITH_CONSOLIDATION (consolidation-cadence anchor): MRR = 0.8264, R@10 = 0.940.** ΔvsNO = +0.0014, within the per-row noise floor. This is the **n=1986 validation that the cadence fix (commit `6c51bce`) holds**: pre-fix smoke had MRR_with_cons collapse to 0.222 because of a wall-clock vs event-time confusion; the post-fix anchor sits indistinguishable from NO_CONSOLIDATION at full scale. - The 14-row two-baseline ablation **empirically resolves the architectural-mismatch hypothesis from the LME-S §6.3 writeup**: longitudinal mechanisms (RECONSOLIDATION, CO_ACTIVATION, ADAPTIVE_DECAY) that were at the noise floor on isolated-haystack LME-S **do show measurable effect on the longitudinal benchmark whose mechanism-of-action they target**. diff --git a/docs/benchmarks/e1-v3-locomo-smoke-finding.md b/docs/benchmarks/e1-v3-locomo-smoke-finding.md index bd33bdaa..7858549d 100644 --- a/docs/benchmarks/e1-v3-locomo-smoke-finding.md +++ b/docs/benchmarks/e1-v3-locomo-smoke-finding.md @@ -13,7 +13,7 @@ files do NOT touch the benchmark code path; benchmark uses ## Pre-registered validation gate From task #55 spec: -> BASELINE LoCoMo MRR: established baseline from CLAUDE.md is 0.794 R@10=0.926. +> BASELINE LoCoMo MRR: historical April 2026 clean-DB Cortex comparator is 0.794, R@10=0.926 (n=1982; first published in commit `b4057a`; original per-query artefact not committed; superseded by the [post-fix re-run](e1-v3-locomo-results-post-fix.md)). > With `--with-consolidation` enabled, this should be APPROXIMATELY similar > (consolidation may shift it slightly, but should be within ±0.05 MRR). If > WAY off — STOP and diagnose. @@ -26,7 +26,7 @@ From task #55 spec: | `--limit 1 --with-consolidation` | **0.222** | **54.8%** | 176.3 (incl. 127.7s consol) | Δ MRR = **−0.644** vs the no-consolidation anchor. -Δ MRR = **−0.572** vs the published 0.794 (CLAUDE.md headline). +Δ MRR = **−0.572** vs the superseded historical 0.794 comparator. This is **WAY off** the ±0.05 tolerance. Stop-and-diagnose triggered. diff --git a/docs/papers/appendix-erdos-bounds.md b/docs/papers/appendix-erdos-bounds.md index eba7ce67..60feda34 100644 --- a/docs/papers/appendix-erdos-bounds.md +++ b/docs/papers/appendix-erdos-bounds.md @@ -62,7 +62,7 @@ $$ LongMemEval-S (Wu et al. 2025, ICLR) has effective corpus size `N ≈ 10^5` across the test set (500 sessions × ~200 atomic events). Since `N_cross ≈ 10^4 < 10^5`, Theorem 1 predicts that any flat-importance retriever should suffer the `k/N` collapse on at least a constant fraction of queries. The paper's reported best `R@10 = 78.4%` matches: roughly `1 - 10^4/10^5 = 0.9` queries are below the bad-query threshold, and the residual `~10%` are exactly the Theorem 1 collapse cases. -Cortex measures `R@10 = 97.8%` (Apr 2026, clean DB, single process). The `19.4`-point gap is the predicted Zipf rescue. (proved by random argument for the upper bound; by direct measurement for the Cortex value.) +Cortex measures `R@10 = 98.2%` (clean run `benchmarks/results/repro/20260714-v4.14.1-pretag/longmemeval-s.json`, code SHA `28145f0`, dirty=false). The `19.8`-point gap is the predicted Zipf rescue. (proved by random argument for the upper bound; by direct measurement for the Cortex value.) --- @@ -84,7 +84,7 @@ Cortex measures `R@10 = 97.8%` (Apr 2026, clean DB, single process). The `19.4`- | Zipf `w`, `γ > 1` | `≥ 1 - k^{1-γ}` (constant) | direct computation (Thm 2) | | Optimal decay `λ*` | maximal, unique | random argument (Thm 3) | -The crossover `N_cross ≈ 10^4` (k=10, γ=1.2, σ=0.05) demarcates the regime in which decay is optional (small corpora) from the regime in which it is necessary (production-scale memory). LongMemEval at `N ≈ 10^5` is firmly in the latter, predicting the observed 78.4% → 97.8% improvement. +The crossover `N_cross ≈ 10^4` (k=10, γ=1.2, σ=0.05) demarcates the regime in which decay is optional (small corpora) from the regime in which it is necessary (production-scale memory). LongMemEval at `N ≈ 10^5` is firmly in the latter, predicting the observed 78.4% → 98.2% improvement. --- diff --git a/docs/papers/appendix-popper-falsifiability.md b/docs/papers/appendix-popper-falsifiability.md index 2313190e..57633f35 100644 --- a/docs/papers/appendix-popper-falsifiability.md +++ b/docs/papers/appendix-popper-falsifiability.md @@ -55,19 +55,19 @@ A claim with no falsification condition is unscientific in Popper's sense and mu --- -## C5 — 97.8% LongMemEval R@10 generalizes beyond the benchmark (SEVERITY: high) +## C5 — 98.2% LongMemEval R@10 generalizes beyond the benchmark (SEVERITY: high) -**The claim.** "97.8% LongMemEval R@10 generalizes beyond the benchmark — the system is not overfit." +**The claim.** "98.2% LongMemEval R@10 generalizes beyond the benchmark — the system is not overfit." **Riskiest prediction.** Calibration parameters tuned on LongMemEval (WRRF weights, intent classifier thresholds, FlashRank reranker depth) transfer to LoCoMo without retuning, retaining ≥ 90% of LongMemEval's R@10 (i.e., LoCoMo R@10 ≥ 0.88). **Falsification protocol.** - Freeze all hyperparameters at their LongMemEval-tuned values. -- Evaluate on LoCoMo (1986 Qs) and BEAM (200 Qs at 100k) without modification. -- Metric: R@10 on each held-out benchmark. -- Threshold: if frozen LongMemEval calibration yields LoCoMo R@10 < 0.83 or BEAM Overall < 0.45, the cross-benchmark generalization claim is refuted. +- Evaluate on LoCoMo (1,986 Qs) and the named BEAM-100K five-conversation subset (100 Qs) without modification. +- Metrics: R@10 on LoCoMo and retrieval-proxy MRR on BEAM-100K. +- Threshold: if frozen LongMemEval calibration yields LoCoMo R@10 < 0.83 or BEAM-100K retrieval-proxy MRR < 0.45, the cross-benchmark generalization claim is refuted. -**Current evidence.** **PARTIALLY TESTED.** Each benchmark is currently scored independently with whatever defaults exist at the time, but the calibration history is not version-pinned to a single benchmark. Current scores (LongMemEval 97.8%, LoCoMo 92.6%, BEAM 0.543) are *consistent with* generalization but do not prove it: per-benchmark drift in defaults could be hiding overfitting. A frozen-config cross-eval has not been run. +**Current evidence.** **PARTIALLY TESTED.** Each benchmark is currently scored independently, so the numbers are protocol-specific rather than a frozen-config cross-evaluation: LongMemEval R@10 98.2% / MRR 0.9167 (n=500, clean run `benchmarks/results/repro/20260714-v4.14.1-pretag/longmemeval-s.json`, code SHA `28145f0`, dirty=false); LoCoMo R@10 94.35% / MRR 0.8279 (n=1,986, clean post-fix run, code SHA `2f45bcb`); and BEAM-100K retrieval-proxy MRR 0.591 (n=100, five-conversation run, code SHA `a071d89`). These results are *consistent with* generalization but do not prove it: per-benchmark drift in defaults could be hiding overfitting. A frozen-config cross-eval has not been run. --- @@ -113,7 +113,7 @@ A claim with no falsification condition is unscientific in Popper's sense and mu | C4 WRRF decorrelates | medium | PARTIALLY TESTED | Correlation matrix + 6 ablations | | C6 read/write 100:1 | low | UNTESTED | One week of production counters | -No claim in the paper is currently CONFIRMED in the strict Popperian sense (survived a severe test designed to refute it). The 97.8% / 92.6% / 0.543 numbers are corroborations of the system as configured, not corroborations of the causal claims (C1–C6) about *why* it performs. +No claim in the paper is currently CONFIRMED in the strict Popperian sense (survived a severe test designed to refute it). The protocol-specific 98.2% / 94.35% / 0.591 numbers are corroborations of the system as configured, not corroborations of the causal claims (C1–C6) about *why* it performs. --- diff --git a/docs/papers/appendix-shannon-collapse.md b/docs/papers/appendix-shannon-collapse.md index 617c820f..0551a129 100644 --- a/docs/papers/appendix-shannon-collapse.md +++ b/docs/papers/appendix-shannon-collapse.md @@ -97,10 +97,10 @@ Decay is what generates and *maintains* the heavy tail: without it, repeated wri ## 5. Concrete numbers (LongMemEval R@10) -Cortex measured (clean DB, April 2026): -- LongMemEval R@10: **97.8%** +Cortex measured in the current clean-DB run (`benchmarks/results/repro/20260714-v4.14.1-pretag/longmemeval-s.json`, code SHA `28145f0`, dirty=false): +- LongMemEval R@10: **98.2%** - Best flat-RAG baseline (paper-best): **78.4%** -- Gap: **19.4 pp**. +- Gap: **19.8 pp**. LongMemEval has $N \approx 10^4$ (S variant: 500 questions, ~30 turns/session, ~10k candidate spans). Plugging into §3 with $\eta = 0.05$, $f_S \approx 4$: @@ -108,7 +108,7 @@ $$ N^\star \;=\; \frac{1}{0.05 \cdot 4} \;=\; 5{,}000. $$ -Test set size $10^4$ is $2 N^\star$. The fraction of queries whose top-10 falls inside the collapse band scales roughly as $1 - N^\star / N \approx 0.5$, but only items in the *boundary band* (between rank 10 and rank 50, where score gap is below $\eta$) are mis-ranked. Empirically that band holds $\sim 20$–$25\%$ of items. Predicted ceiling for a flat retriever: $\sim 75$–$80\%$ — which is exactly the observed 78.4% paper-best. The 19.4 pp gap is the discriminability that decay+heat preserves and uniform priors throw away. +Test set size $10^4$ is $2 N^\star$. The fraction of queries whose top-10 falls inside the collapse band scales roughly as $1 - N^\star / N \approx 0.5$, but only items in the *boundary band* (between rank 10 and rank 50, where score gap is below $\eta$) are mis-ranked. Empirically that band holds $\sim 20$–$25\%$ of items. Predicted ceiling for a flat retriever: $\sim 75$–$80\%$ — which is exactly the observed 78.4% paper-best. The 19.8 pp gap is the discriminability that the integrated stack preserves and uniform priors throw away. This is a back-of-envelope, not a tight bound. It survives because the order-of-magnitude $N^\star$ matches the test set, not because the constants are precisely calibrated. diff --git a/docs/papers/arxiv-endorsement-email.md b/docs/papers/arxiv-endorsement-email.md index c34b83ab..c788ad59 100644 --- a/docs/papers/arxiv-endorsement-email.md +++ b/docs/papers/arxiv-endorsement-email.md @@ -35,7 +35,7 @@ github.com/cdeust/Cortex (★52, growing — Perplexity surfaces it on **Paper 2 — Thermodynamic Memory vs. Flat-Importance Stores** (cs.IR or cs.CL) - 34 pages, ready to submit - 45 row per-mechanism ablation campaign on LongMemEval (n=500) and LoCoMo (n=1986) -- LongMemEval R@10 98.4% (vs 78.4% paper best), LoCoMo R@10 94.2% +- LongMemEval R@10 98.2% / MRR 0.9167 (clean run, code SHA `28145f0`), LoCoMo R@10 94.35% (post-fix run, code SHA `2f45bcb`) - Verification surfaced two real production bugs that were fixed and disclosed in the paper itself — the verification campaign improved the system, not just measured it Both PDFs: diff --git a/docs/papers/linkedin-endorser-post.md b/docs/papers/linkedin-endorser-post.md index c7aeffcb..8cfc0366 100644 --- a/docs/papers/linkedin-endorser-post.md +++ b/docs/papers/linkedin-endorser-post.md @@ -15,8 +15,8 @@ The architecture was designed in September 2025 for generating 9-page PRDs on Ap **Repo (MIT, open source):** github.com/cdeust/Cortex Other benchmark results (E1 v3 verification campaign, May 2026): -• 98.4% Recall@10 / 0.9124 MRR on LongMemEval (vs 78.4% paper best, n=500) -• 94.2% Recall@10 / 0.8278 MRR on LoCoMo (vs 92.6% / 0.794, n=1986) +• 98.2% Recall@10 / 0.9167 MRR on LongMemEval (vs 78.4% paper best, n=500; clean run, code SHA 28145f0) +• 94.35% Recall@10 / 0.8279 MRR on LoCoMo (post-fix run, code SHA 2f45bcb, n=1986; historical 92.6% / 0.794 clean-DB comparator was n=1982) • 45 row entries of per-mechanism ablation evidence (17 LME-S + 14 LoCoMo + 14 LoCoMo post-fix) • 41 paper citations, 26 biological mechanisms with faithful implementations • 2700+ tests passing @@ -41,7 +41,7 @@ Built a memory system that scores +33.4% on BEAM-10M (ICLR 2026) — without ora Paper: "Stage-Aware Context Assembly for Long-Context Memory Retrieval" Repo: github.com/cdeust/Cortex (MIT, LaTeX source in docs/arxiv-context-assembly/) -98.4% R@10 LongMemEval | 94.2% R@10 LoCoMo | +33.4% BEAM-10M +98.2% R@10 LongMemEval | 94.35% R@10 LoCoMo | +33.4% BEAM-10M If you can endorse on cs.IR, cs.CL, or cs.AI — DM me. Paper is ready. diff --git a/docs/papers/research-post-context-assembly.md b/docs/papers/research-post-context-assembly.md index 6a3355ca..2d552cae 100644 --- a/docs/papers/research-post-context-assembly.md +++ b/docs/papers/research-post-context-assembly.md @@ -294,8 +294,10 @@ procedure (`recall_memories()`), returning pre-fused results. Client-side, FlashRank (ONNX cross-encoder) reranks the top-3k candidates to produce the final ranking. -This pipeline is strong at moderate scale: 97.8% R@10 on LongMemEval, -92.6% R@10 on LoCoMo. The five-signal fusion mitigates any single +This pipeline is strong at moderate scale: 98.2% R@10 on LongMemEval +(clean run, July 2026; code SHA `28145f0`) and 94.35% R@10 on LoCoMo +(E1 v3 post-fix run, May 2026; code SHA `2f45bcb`). The five-signal +fusion mitigates any single signal's weakness (e.g., vector similarity misses lexical matches that trigram catches; FTS misses paraphrases that vectors catch). But at BEAM-10M scale, all five signals suffer from the same underlying @@ -1128,8 +1130,10 @@ management -- the regime where our architecture is designed to help. **WRRF baseline.** Cortex's production pipeline without the assembler: 5-signal server-side fusion + FlashRank client-side -reranking. This is a strong baseline: 97.8% R@10 on LongMemEval, -92.6% R@10 on LoCoMo, and 0.591 MRR on BEAM-100K. It represents the +reranking. This is a strong baseline: 98.2% R@10 on LongMemEval +(clean run, July 2026; code SHA `28145f0`), 94.35% R@10 on LoCoMo +(E1 v3 post-fix run, May 2026; code SHA `2f45bcb`), and 0.591 MRR on +BEAM-100K (five-conversation protocol, n=100; code SHA `a071d89`). It represents the state of the art for multi-signal hybrid retrieval without structural organization. diff --git a/docs/papers/thermodynamic-memory-vs-flat-importance.md b/docs/papers/thermodynamic-memory-vs-flat-importance.md index 5310d7f6..77ed5170 100644 --- a/docs/papers/thermodynamic-memory-vs-flat-importance.md +++ b/docs/papers/thermodynamic-memory-vs-flat-importance.md @@ -9,7 +9,7 @@ ## Abstract -External memory for large language models is dominated by *flat-importance* stores: vector indexes, BM25 corpora, and long-context buffers in which every item carries the same long-term retrieval prior. We argue this design is asymptotically broken. As the corpus grows, top-k retrieval over an undifferentiated pool degenerates into near-arbitrary tie-breaking among items with comparable surface similarity, and the discriminative information delivered to the consumer LLM approaches zero — a failure mode that compounds the position bias of long-context decoding (Liu et al., 2023). We formalise the flat-importance failure, then describe Cortex, a memory architecture that maintains a non-flat priority distribution across N by coupling four mechanisms: (i) continuously decaying *heat* on every item (Ebbinghaus, 1885), (ii) a hierarchical predictive-coding write gate (Friston, 2010), (iii) consolidation cascades that compress episodic into semantic memory (Kandel, 2001; McClelland et al., 1995), and (iv) WRRF fusion with heat as a tie-breaker. On three independent long-term-memory benchmarks, Cortex reaches LongMemEval R@10 = 98.4% (vs. 78.4% paper-best), LoCoMo R@10 = 94.2%, and BEAM Overall = 0.591 (vs. 0.329 paper-best). We discuss when flat memory remains adequate (small N, single-session contexts), the calibration cost of decay, and the per-write overhead of biological consolidation. We position the work within the broader Cortex ecosystem — paired with **cortex-beam-abstain** (a learned retrieval-abstention model for the residual cases where decay cannot prevent collapse), the **automatised-pipeline** AST backend that gives memories code-structural anchors, and **prd-spec-generator** as the downstream read-heavy workload that justifies thermodynamic write costs. +External memory for large language models is dominated by *flat-importance* stores: vector indexes, BM25 corpora, and long-context buffers in which every item carries the same long-term retrieval prior. We argue this design is asymptotically broken. As the corpus grows, top-k retrieval over an undifferentiated pool degenerates into near-arbitrary tie-breaking among items with comparable surface similarity, and the discriminative information delivered to the consumer LLM approaches zero — a failure mode that compounds the position bias of long-context decoding (Liu et al., 2023). We formalise the flat-importance failure, then describe Cortex, a memory architecture that maintains a non-flat priority distribution across N by coupling four mechanisms: (i) continuously decaying *heat* on every item (Ebbinghaus, 1885), (ii) a hierarchical predictive-coding write gate (Friston, 2010), (iii) consolidation cascades that compress episodic into semantic memory (Kandel, 2001; McClelland et al., 1995), and (iv) WRRF fusion with heat as a tie-breaker. On three independent long-term-memory benchmarks, Cortex reaches LongMemEval R@10 = 98.2% (vs. 78.4% paper-best), LoCoMo R@10 = 94.35%, and a BEAM-100K retrieval-proxy MRR of 0.591. BEAM's published end-to-end LLM-as-judge score is not commensurable with this retrieval metric, so we make no head-to-head BEAM claim. We discuss when flat memory remains adequate (small N, single-session contexts), the calibration cost of decay, and the per-write overhead of biological consolidation. We position the work within the broader Cortex ecosystem — paired with **cortex-beam-abstain** (a learned retrieval-abstention model for the residual cases where decay cannot prevent collapse), the **automatised-pipeline** AST backend that gives memories code-structural anchors, and **prd-spec-generator** as the downstream read-heavy workload that justifies thermodynamic write costs. --- @@ -21,7 +21,7 @@ The reason is not retrieval speed but *retrieval semantics*. A vector store trea This paper argues that the missing mechanism is decay, and that decay is not a heuristic but a structural requirement: it is what keeps the priority distribution over memories non-flat at any N. We characterise the flat-importance failure mode, describe Cortex's thermodynamic architecture, and report empirical results on three published long-term-memory benchmarks. -**Teaser.** On LongMemEval (ICLR 2025), Cortex reaches Recall@10 of 98.4%, vs. the paper-best of 78.4%. On BEAM (ICLR 2026), Cortex reaches 0.591 Overall, vs. 0.329 paper-best. All numbers are from runs against Cortex's production database, single-process, with the exact PL/pgSQL retrieval code path used in deployment. +**Teaser.** On LongMemEval (ICLR 2025), Cortex reaches Recall@10 of 98.2%, vs. the paper-best of 78.4%. On BEAM-100K, Cortex reaches retrieval-proxy MRR 0.591; this is reported only for within-system comparison because BEAM's published score is end-to-end and incommensurable. All numbers are from runs against Cortex's production database, single-process, with the exact PL/pgSQL retrieval code path used in deployment. ## 2. Background and related work @@ -148,25 +148,25 @@ We evaluate Cortex on three independent long-term-memory benchmarks. All numbers | Benchmark | Venue | Metric | Cortex | Paper-best | |---|---|---|---|---| -| LongMemEval | ICLR 2025 | R@10 | **98.4%** | 78.4% | -| LongMemEval | ICLR 2025 | MRR | **0.9124** | — | -| LoCoMo | ACL 2024 | R@10 | **94.2%** | — | -| LoCoMo | ACL 2024 | MRR | **0.8278** | — | -| BEAM | ICLR 2026 | Overall | **0.591** | 0.329 | +| LongMemEval | ICLR 2025 | R@10 | **98.2%** | 78.4% | +| LongMemEval | ICLR 2025 | MRR | **0.9167** | — | +| LoCoMo | ACL 2024 | R@10 | **94.35%** | — | +| LoCoMo | ACL 2024 | MRR | **0.8279** | — | +| BEAM-100K | ICLR 2026 | MRR (retrieval-proxy) | **0.591** | — | -The +20.0 pp absolute gain on LongMemEval R@10 and the +79.6% relative gain on BEAM Overall are the headline results. Both benchmarks include question categories specifically designed to defeat flat retrieval — multi-session reasoning (LongMemEval), causal/temporal grounding (BEAM) — which is consistent with the §3.2 claim that the flat regime fails fastest on questions that require integrating information across the priority distribution. +The +19.8 pp absolute gain on LongMemEval R@10 is the headline result against a published baseline. The BEAM-100K retrieval-proxy MRR is reported only for within-system comparison. Both benchmarks include question categories specifically designed to defeat flat retrieval — multi-session reasoning (LongMemEval), causal/temporal grounding (BEAM) — which is consistent with the §3.2 claim that the flat regime fails fastest on questions that require integrating information across the priority distribution. **Where Cortex wins.** The largest gaps appear on temporal questions ("what did I decide first about X?"), causal-chain questions ("why did Y change?"), and multi-hop knowledge integration. These are the question categories that require traversal of the entity/causal graph and that benefit most from heat-modulated tie-breaking. **Where the gap is smaller.** On surface-fact retrieval ("what is the value of X?") with a small corpus, flat baselines do reasonably well — there is no priority disambiguation to do because the fact is uniquely identified by similarity. Cortex's advantage on these categories is primarily from the FTS/trigram channels, not the thermodynamic ones. -**Caveats on these numbers.** (i) We do not have head-to-head re-runs of every published baseline on our exact protocol; we report Cortex's numbers and the highest paper-reported number on each benchmark. (ii) These benchmarks are retrieval-quality benchmarks; downstream end-task accuracy with a specific LLM may differ. (iii) BEAM's Overall is a composite of seven sub-metrics — see `benchmarks/beam/` for the per-subset breakdown. +**Caveats on these numbers.** (i) We do not have head-to-head re-runs of every published baseline on our exact protocol; we report Cortex's numbers and the highest paper-reported retrieval number only where the metric is commensurable. (ii) These benchmarks are retrieval-quality benchmarks; downstream end-task accuracy with a specific LLM may differ. (iii) BEAM's published score is an end-to-end LLM-as-judge metric, whereas our 0.591 is retrieval-proxy MRR; the two are not compared. See `benchmarks/beam/` for the per-ability retrieval breakdown. ### 6.3 Per-mechanism evidence (LongMemEval-S, n=500) -The headline §6 table reports the integrated stack against published baselines. This subsection opens the integrated number and asks which mechanisms in §4 carry the lift on two benchmarks — LongMemEval-S (§6.3.1–6.3.3) and LoCoMo (§6.3.4) — at the calibrated equilibrium. +The headline §6 table reports the current integrated stack against published baselines. This subsection reports a separate, earlier clean E1 v3 ablation snapshot and asks which mechanisms in §4 carry the lift in that snapshot on two benchmarks — LongMemEval-S (§6.3.1–6.3.3) and LoCoMo (§6.3.4) — at the calibrated equilibrium. -**Headline against the established Cortex baseline.** On LongMemEval-S at n=500, the calibrated integrated stack reaches **MRR = 0.9124** and **R@10 = 0.984** (artefact: `benchmarks/results/ablation/longmemeval-s_v3/BASELINE.json`; manifest: `benchmarks/results/ablation/longmemeval-s_v3/manifest.json`, code SHA `0e858e8`, dirty=false, finished 2026-05-03). Against the previously established CLAUDE.md reference (MRR = 0.882, R@10 = 0.978) this is **+3.0% MRR and +0.6% R@10**. The single-seed limitation of §6 still applies; the per-row noise floor on n=500 is empirically ≈ ±0.001 MRR. +**Historical E1 v3 ablation snapshot.** On LongMemEval-S at n=500, the calibrated integrated stack reaches **MRR = 0.9124** and **R@10 = 0.984** (artefact: `benchmarks/results/ablation/longmemeval-s_v3/BASELINE.json`; manifest: `benchmarks/results/ablation/longmemeval-s_v3/manifest.json`, code SHA `0e858e8`, dirty=false, finished 2026-05-03). These values belong to this named ablation snapshot, not to the current headline run. Against the previously established CLAUDE.md reference (MRR = 0.882, R@10 = 0.978) this is **+3.0% MRR and +0.6% R@10**. The single-seed limitation of §6 still applies; the per-row noise floor on n=500 is empirically ≈ ±0.001 MRR. #### 6.3.1 Sign convention and the 17-row table @@ -236,40 +236,40 @@ The LoCoMo ablation is a 14-row, two-baseline, single-seed sweep on the full ben Sign convention is unchanged from §6.3.1: ΔMRR = anchor − ablated, so positive ΔMRR ⇒ mechanism contributes positively. -**Headline.** `BASELINE_NO_CONSOLIDATION` reaches MRR = 0.8278, R@10 = 0.942 on LoCoMo (n = 1986). Against the established LoCoMo baseline (MRR = 0.794, R@10 = 0.926) this is +4.3% MRR, +1.6% R@10. `BASELINE_WITH_CONSOLIDATION` reaches MRR = 0.8264, R@10 = 0.940 — ΔvsNO = +0.0014, within the per-row noise floor. The two anchors agreeing at full n confirms that the consolidation cadence fix described in §6.3.6 holds on the full benchmark, not only on smoke. +**Headline.** `BASELINE_NO_CONSOLIDATION` reaches MRR = 0.8279, R@10 = 0.9435 on LoCoMo (n = 1986). Against the historical April 2026 clean-DB Cortex comparator (MRR = 0.794, R@10 = 0.926, n = 1982; first published in commit `b4057a`, with no committed per-query artefact) this is +4.3% MRR and +1.75 percentage points R@10. `BASELINE_WITH_CONSOLIDATION` reaches MRR = 0.8265, R@10 = 0.9410 — ΔvsNO = +0.0014, within the per-row noise floor. The current values come from the post-plasticity-fix run at code SHA `2f45bcb39dbe15fa0ef857cc8c8c3783175d05db`, recorded in `docs/benchmarks/e1-v3-locomo-results-post-fix.md`; the historical pair is a comparator, not the current baseline. **14-row LoCoMo table.** | Mechanism | MRR (ablated) | R@10 (ablated) | ΔMRR | ΔR@10 | Anchor | Note | |-----------------------------|--------------:|---------------:|--------:|--------:|--------|------| -| BASELINE_NO_CONSOLIDATION | 0.8278 | 0.942 | 0 | 0 | self | Reference (longitudinal read-path anchor) | -| RECONSOLIDATION | 0.8202 | 0.931 | +0.0076 | +0.011 | NO | Strongest positive contribution in the table | -| CO_ACTIVATION | 0.8268 | 0.940 | +0.0010 | +0.001 | NO | Confirmed positive contribution | -| ADAPTIVE_DECAY | 0.8441 | 0.962 | -0.0163 | -0.020 | NO | Strongest counterproductive; ablating improves the score | -| BASELINE_WITH_CONSOLIDATION | 0.8264 | 0.940 | 0 | 0 | self | Reference (consolidation-cadence anchor); ΔvsNO = +0.0014 (within noise) | -| CASCADE | 0.8272 | 0.941 | -0.0008 | -0.001 | WITH | Within noise floor | -| INTERFERENCE | 0.8260 | 0.939 | +0.0004 | +0.001 | WITH | Within noise floor | -| HOMEOSTATIC_PLASTICITY | 0.8289 | 0.945 | -0.0025 | -0.005 | WITH | Largest absolute in consolidation-only group | -| SYNAPTIC_PLASTICITY | 0.8264 | 0.940 | 0.0000 | 0 | WITH | Null contribution (clean: full plasticity disable) | -| MICROGLIAL_PRUNING | 0.8253 | 0.939 | +0.0011 | +0.001 | WITH | Within noise floor | -| TWO_STAGE_MODEL | 0.8276 | 0.941 | -0.0012 | -0.001 | WITH | Within noise floor | -| EMOTIONAL_DECAY | 0.8249 | 0.940 | +0.0015 | -0.000 | WITH | Within noise floor | -| TRIPARTITE_SYNAPSE | 0.8268 | 0.941 | -0.0004 | -0.001 | WITH | Within noise floor | -| SCHEMA_ENGINE | 0.8268 | 0.941 | -0.0004 | -0.001 | WITH | Within noise floor | +| BASELINE_NO_CONSOLIDATION | 0.8279 | 0.9435 | 0 | 0 | self | Reference (longitudinal read-path anchor) | +| RECONSOLIDATION | 0.8188 | 0.9289 | +0.0091 | +0.0146 | NO | Strongest positive contribution in the table | +| CO_ACTIVATION | 0.8264 | 0.9400 | +0.0015 | +0.0035 | NO | Confirmed positive direction; within MRR noise floor | +| ADAPTIVE_DECAY | 0.8442 | 0.9622 | -0.0163 | -0.0187 | NO | Strongest counterproductive; ablating improves the score | +| BASELINE_WITH_CONSOLIDATION | 0.8265 | 0.9410 | 0 | 0 | self | Reference (consolidation-cadence anchor); ΔvsNO = +0.0014 (within noise) | +| CASCADE | 0.8268 | 0.9425 | -0.0002 | -0.0015 | WITH | Within noise floor | +| INTERFERENCE | 0.8271 | 0.9410 | -0.0005 | 0.0000 | WITH | Within noise floor | +| HOMEOSTATIC_PLASTICITY | 0.8248 | 0.9390 | +0.0017 | +0.0020 | WITH | Sign flipped post-fix; within MRR noise floor | +| SYNAPTIC_PLASTICITY | 0.8269 | 0.9405 | -0.0003 | +0.0005 | WITH | Within noise floor | +| MICROGLIAL_PRUNING | 0.8269 | 0.9420 | -0.0004 | -0.0010 | WITH | Within noise floor | +| TWO_STAGE_MODEL | 0.8267 | 0.9395 | -0.0002 | +0.0015 | WITH | Within noise floor | +| EMOTIONAL_DECAY | 0.8263 | 0.9415 | +0.0002 | -0.0005 | WITH | Within noise floor | +| TRIPARTITE_SYNAPSE | 0.8266 | 0.9415 | -0.0001 | -0.0005 | WITH | Within noise floor | +| SCHEMA_ENGINE | 0.8249 | 0.9395 | +0.0017 | +0.0015 | WITH | Sign flipped post-fix; within MRR noise floor | **Empirical resolution of the LME-S architectural-mismatch hypothesis.** The three longitudinal mechanisms LME-S could not exercise show up on LoCoMo, with consistent signs and magnitudes that match the mechanism-of-action argument: | Mechanism | LME-S ΔMRR | LoCoMo ΔMRR | Resolution | |------------------|-----------:|------------:|------------| -| RECONSOLIDATION | +0.0000 | +0.0076 | Confirmed: mechanism fires on multi-session recall | -| CO_ACTIVATION | +0.0000 | +0.0010 | Confirmed; smaller magnitude | +| RECONSOLIDATION | +0.0000 | +0.0091 | Confirmed: mechanism fires on multi-session recall | +| CO_ACTIVATION | +0.0000 | +0.0015 | Confirmed; smaller magnitude | | ADAPTIVE_DECAY | -0.0014 | -0.0163 | Same sign, amplified ~11× — decay is counterproductive on both, more so on the longitudinal benchmark | This is the load-bearing finding of §6.3.4. The §6.3.3 argument (that 13 LME-S rows were *predicted-null by construction*, not failed mechanisms) is now empirically substantiated for the longitudinal subset: when the benchmark exercises the mechanism-of-action, the mechanism shows up in the deltas. -**Top contributors per anchor group.** In the longitudinal-read-path group, ADAPTIVE_DECAY (|ΔMRR| = 0.0163, counterproductive) and RECONSOLIDATION (ΔMRR = +0.0076, positive) dominate; the third row CO_ACTIVATION (+0.0010) is consistent-sign but at the per-row noise floor. In the consolidation-only group, all nine deltas sit within the per-row noise floor (≈ ±0.002 MRR at n = 1986 single-seed); HOMEOSTATIC_PLASTICITY (-0.0025) is the largest absolute, EMOTIONAL_DECAY (+0.0015) and TWO_STAGE_MODEL (-0.0012) follow. The honest reading of the consolidation-only group is that the consolidation pipeline as a whole contributes (the cadence fix narrative in §6.3.6 is not undone by these deltas), but no single consolidation-time mechanism dominates at LoCoMo's scale — the same calibrated-stack property §6.3.1 already documented for LME-S. +**Top contributors per anchor group.** In the longitudinal-read-path group, ADAPTIVE_DECAY (|ΔMRR| = 0.0163, counterproductive) and RECONSOLIDATION (ΔMRR = +0.0091, positive) dominate; the third row CO_ACTIVATION (+0.0015) is consistent-sign but at the per-row noise floor. In the consolidation-only group, every delta is within the per-row noise floor (≈ ±0.002 MRR at n = 1986 single-seed); HOMEOSTATIC_PLASTICITY and SCHEMA_ENGINE (both +0.0017) have the largest absolute magnitudes but remain at the noise boundary. The honest reading is that no single consolidation-time mechanism dominates at LoCoMo's scale — the same calibrated-stack property §6.3.1 already documented for LME-S. -**Limitations of the LoCoMo run.** Single-seed at n = 1986; per-row noise floor ≈ ±0.002 MRR. The plasticity result-shape contract bug fixed in commit `5f737fe` (§6.3.7 below) was discovered *during* the LoCoMo sweep and the run was launched on bytes pre-fix; the BASELINE_WITH and the nine consolidation-only rows therefore ran with a logged-WARNING (not a crash) that may have muted some consolidation deltas. The three longitudinal-read-path rows ran with consolidation off, are not affected by the plasticity bug, and constitute the empirical resolution finding above. The follow-up re-run on post-`5f737fe` bytes is reported in §6.3.4.1. +**Limitations of the LoCoMo run.** Single-seed at n = 1986; per-row noise floor ≈ ±0.002 MRR. The current table is the post-fix run on bytes containing both `6c51bce` and `5f737fe`; magnitudes below the noise floor are not interpreted causally. The pre-`5f737fe` predecessor is retained only for the transparent comparison in §6.3.4.1, not as the source of current figures. #### 6.3.4.1 The plasticity-fix re-run on post-`5f737fe` bytes @@ -277,7 +277,7 @@ The §6.3.7 plasticity result-shape contract bug was fixed in commit `5f737fe`; **Headline.** `BASELINE_NO_CONSOLIDATION` reaches MRR = 0.8279, R@10 = 0.9435; `BASELINE_WITH_CONSOLIDATION` reaches MRR = 0.8265, R@10 = 0.941; ΔvsNO = +0.0014 — **identical to four decimals** to the pre-fix value. The cadence-fix anchor agreement (§6.3.6) is therefore re-validated at full n on a second independent run on bytes that include the plasticity fix. The longitudinal-read-path group is essentially unchanged between runs, as expected: those rows ran with consolidation off, so the plasticity bug had no opportunity to exercise. -**Pre-vs-post-fix comparison.** The consolidation-only group has three sign-flips relative to the pre-fix sweep, two of them moving out of noise: +**Pre-vs-post-fix comparison.** The consolidation-only group has three sign-flips relative to the pre-fix sweep; all post-fix consolidation-only deltas remain at the ≈ ±0.002 MRR noise floor: | Mechanism | Pre-fix ΔMRR | Post-fix ΔMRR | Reading | |--------------------------|-------------:|--------------:|---------| @@ -289,9 +289,9 @@ The §6.3.7 plasticity result-shape contract bug was fixed in commit `5f737fe`; | SYNAPTIC_PLASTICITY | 0.0000 | -0.0003 | Within noise; explicitly clean ablation | | Other 6 consolidation rows | within noise | within noise | All deltas remain within ≈ ±0.002 MRR | -Two rows (HOMEOSTATIC_PLASTICITY at +0.0042, SCHEMA_ENGINE at +0.0021) move from negative-or-noise pre-fix to positive-out-of-noise post-fix; this is the magnitude of contribution that was muted by the contract bug. Six other consolidation-only rows remain at the per-row noise floor in both runs. The architectural-mismatch resolution (RECONSOLIDATION ΔMRR = +0.0091, ADAPTIVE_DECAY ΔMRR = -0.0163) is **strengthened** on clean bytes — the dominant longitudinal contributions are intact and slightly larger in magnitude, as predicted in §6.3.4 (the longitudinal rows were never affected by the plasticity bug, so any movement is attributable to single-seed run-to-run variance at the noise floor; the magnitudes here move with that variance, not against the hypothesis). +HOMEOSTATIC_PLASTICITY and SCHEMA_ENGINE move by +0.0042 and +0.0021 respectively between runs and end at positive-direction ΔMRR = +0.0017, still inside the post-fix run's noise floor. Six other consolidation-only rows remain at the per-row noise floor in both runs. The architectural-mismatch resolution (RECONSOLIDATION ΔMRR = +0.0091, ADAPTIVE_DECAY ΔMRR = -0.0163) is **re-confirmed** on clean bytes; the dominant longitudinal contributions are intact, while their small between-run movement is single-seed variation. -**The verification self-correcting.** §6.3.7 declared the plasticity-shape bug as a possible source of mute on the consolidation-only group and committed to a follow-up re-run. This subsection is that re-run. Two consolidation-only rows (HOMEOSTATIC_PLASTICITY, SCHEMA_ENGINE) recover positive contributions that were silently masked; the cadence-fix anchor agreement is re-confirmed; the load-bearing architectural-mismatch finding is re-confirmed and slightly strengthened. The verification campaign now comprises three artefact sets at full n (LME-S 17 rows + LoCoMo 14 rows pre-fix + LoCoMo 14 rows post-fix = 45 per-mechanism evidence rows on the appropriate benchmark for each mechanism's mechanism-of-action), with the only contract bug surfaced during verification re-measured on clean bytes and the result documented either way. +**The verification self-correcting.** §6.3.7 declared the plasticity-shape bug as a possible source of mute on the consolidation-only group and committed to a follow-up re-run. This subsection is that re-run. HOMEOSTATIC_PLASTICITY and SCHEMA_ENGINE recover positive-direction deltas at the noise boundary; the cadence-fix anchor agreement and the load-bearing architectural-mismatch finding are re-confirmed. The verification campaign now comprises three artefact sets at full n (LME-S 17 rows + LoCoMo 14 rows pre-fix + LoCoMo 14 rows post-fix = 45 per-mechanism evidence rows on the appropriate benchmark for each mechanism's mechanism-of-action), with the only contract bug surfaced during verification re-measured on clean bytes and the result documented either way. #### 6.3.5 Calibration rigor: Phase A and Phase B @@ -308,7 +308,7 @@ During the same verification campaign the team discovered a production-relevant The fix (commit `6c51bce`) introduces `memories.ingested_at TIMESTAMPTZ NOT NULL DEFAULT NOW()`, with an idempotent migration backfilling `ingested_at = created_at` for legacy rows, and routes the cadence gate, ACT-R lifetime computation, synaptic-tagging window, and temporal-novelty signal through `ingested_at` rather than `created_at`. Regression tests in `test_compression.py`, `test_decay_cycle.py`, and `test_pg_ingested_at.py` lock the new behaviour. The fix is independent of the LME-S evaluation reported in §6.3.1–6.3.3 (LME-S is not consolidation-dependent) but is necessary for the LoCoMo half (§6.3.4) and for any production backfill scenario where memories are ingested with historical timestamps. -The fix was validated on smoke first; the §6.3.4 LoCoMo run is the n = 1986 validation. At full scale, `BASELINE_WITH_CONSOLIDATION` reaches MRR = 0.8264 against `BASELINE_NO_CONSOLIDATION` at MRR = 0.8278 (ΔvsNO = +0.0014, within the per-row noise floor of ≈ ±0.002 MRR). The two anchors agree at full n; the cadence fix holds, and the §6.3.4 consolidation-only deltas are measured against a stable post-fix baseline. +The fix was validated on smoke first; the §6.3.4 post-plasticity-fix LoCoMo run is the authoritative n = 1986 validation. At full scale, `BASELINE_WITH_CONSOLIDATION` reaches MRR = 0.8265 against `BASELINE_NO_CONSOLIDATION` at MRR = 0.8279 (ΔvsNO = +0.0014, within the per-row noise floor of ≈ ±0.002 MRR). The two anchors agree at full n; the cadence fix holds, and the §6.3.4 consolidation-only deltas are measured against a stable post-fix baseline. We mention this not to recount engineering, but because it tightens the §1 framing: a verification campaign is not just *was the system as designed correct?* but *did verification improve the system?* In this instance it did. @@ -318,15 +318,15 @@ A second production-relevant bug surfaced during the same LoCoMo verification ca The fix (commit `5f737fe`) makes the ablation no-op return result-shaped dicts with `action="none"`, restoring contract compliance for the disabled path. Regression tests lock the result-shape invariant. -The §6.3.4 LoCoMo run was launched on bytes *before* commit `5f737fe`, which means the consolidation-only ablation rows (CASCADE, INTERFERENCE, HOMEOSTATIC_PLASTICITY, MICROGLIAL_PRUNING, TWO_STAGE_MODEL, EMOTIONAL_DECAY, TRIPARTITE_SYNAPSE, SCHEMA_ENGINE) may have a slightly muted plasticity contribution. The SYNAPTIC_PLASTICITY ablation row is not affected: that row explicitly disables the plasticity mechanism entirely, so the no-op shape bug cannot exercise. The three longitudinal-read-path rows (RECONSOLIDATION, CO_ACTIVATION, ADAPTIVE_DECAY) ran with consolidation off and are likewise not affected. The empirical resolution of the architectural-mismatch hypothesis (§6.3.4 above) does not depend on the consolidation-only group; it rests on the longitudinal-read-path rows, which are clean. +The predecessor to the §6.3.4 LoCoMo run was launched on bytes *before* commit `5f737fe`, which means the consolidation-only ablation rows (CASCADE, INTERFERENCE, HOMEOSTATIC_PLASTICITY, MICROGLIAL_PRUNING, TWO_STAGE_MODEL, EMOTIONAL_DECAY, TRIPARTITE_SYNAPSE, SCHEMA_ENGINE) may have had a slightly muted plasticity contribution. The SYNAPTIC_PLASTICITY ablation row was not affected: that row explicitly disabled the plasticity mechanism entirely, so the no-op shape bug could not exercise. The three longitudinal-read-path rows (RECONSOLIDATION, CO_ACTIVATION, ADAPTIVE_DECAY) ran with consolidation off and were likewise not affected. The current §6.3.4 table uses the clean post-fix re-run. -The re-run of the BASELINE_WITH and consolidation-only rows on post-`5f737fe` bytes is reported in §6.3.4.1. The §6.3 narrative does not change as a result: the architectural-mismatch resolution holds (RECONSOLIDATION ΔMRR = +0.0091, ADAPTIVE_DECAY ΔMRR = -0.0163 on clean bytes); two consolidation-only rows (HOMEOSTATIC_PLASTICITY, SCHEMA_ENGINE) move from noise/negative to positive-out-of-noise, recovering the contribution the contract bug had silently muted; the cadence-fix anchor agreement (§6.3.6) is re-validated identically (ΔvsNO = +0.0014). We declare both the bug and the re-run rather than amend silently because the verification campaign's evidence is the load-bearing argument of §6.3, and the integrity of that argument requires disclosing every code-path artefact that touched the numbers — and remeasuring on clean bytes once the artefact is removed. +The post-`5f737fe` re-run is reported in §6.3.4.1. The architectural-mismatch resolution holds (RECONSOLIDATION ΔMRR = +0.0091, ADAPTIVE_DECAY ΔMRR = -0.0163 on clean bytes); HOMEOSTATIC_PLASTICITY and SCHEMA_ENGINE move from negative/noise to positive direction while remaining inside the post-fix noise floor; the cadence-fix anchor agreement (§6.3.6) is re-validated identically (ΔvsNO = +0.0014). We declare both the bug and the re-run rather than amend silently because the integrity of the evidence requires disclosing every code-path artefact that touched the numbers and remeasuring after removal. The §1 framing applies again: verification did not just confirm the system; it surfaced two real bugs (cadence and plasticity result-shape) that are now fixed. #### 6.3.8 Caveats specific to §6.3 -- **Single-seed per run.** Each row in §6.3.1 (17 rows, LME-S, n = 500) and §6.3.4 (14 rows, LoCoMo, n = 1986, pre-fix bytes) is run once; the §6.3.4.1 LoCoMo post-fix re-run adds 14 more rows on independent bytes (`2f45bcb`, descendant of `5f737fe`). Per-question noise averages down by $\sqrt{n}$; empirical per-row noise floor is ≈ ±0.001 MRR on LME-S and ≈ ±0.002 MRR on LoCoMo. ΔMRR magnitudes below the relevant threshold are not interpretable as causal contributions; the paper-bearing claims of §6.3 are the *category-specialization pattern* (LME-S), the *empirical resolution of the architectural-mismatch hypothesis* (LoCoMo, confirmed on two runs straddling the plasticity-shape fix), and the *integrated stack lift over the published baselines on both benchmarks*, not the per-row sub-noise deltas. +- **Single-seed per run.** Each row in §6.3.1 (17 rows, LME-S, n = 500), the historical LoCoMo pre-fix sweep (14 rows), and the authoritative §6.3.4 LoCoMo post-fix sweep (14 rows, `2f45bcb`, descendant of `5f737fe`) is run once. Per-question noise averages down by $\sqrt{n}$; empirical per-row noise floor is ≈ ±0.001 MRR on LME-S and ≈ ±0.002 MRR on LoCoMo. ΔMRR magnitudes below the relevant threshold are not interpretable as causal contributions; the paper-bearing claims of §6.3 are the *category-specialization pattern* (LME-S) and the *empirical resolution of the architectural-mismatch hypothesis* (LoCoMo, confirmed on two runs straddling the plasticity-shape fix), not the per-row sub-noise deltas. - **Two benchmarks, complementary architectures.** LME-S §6.3.1 captures saturated-rerank and integrated-stack behaviour; LoCoMo §6.3.4 captures longitudinal mechanism behaviour. The two together cover the read-path / write-path / consolidation-path stack; neither alone would. - **Calibration-conditional.** The integrated lift is reported at the Phase A/B calibrated equilibrium. Re-calibration on a different workload (e.g. an emotion-laden corpus that exercises the affect-side gates) would shift the per-mechanism contributions; §8 already notes that *the model is general; its constants are not.* @@ -444,4 +444,3 @@ The ablation work needed to make this story tight — per-mechanism contribution - Deust, C. (2026c). *prd-spec-generator — Stateless reducer turning feature descriptions into multi-judge-verified PRDs*. https://github.com/cdeust/prd-spec-generator - Deust, C. (2026d). *zetetic-team-subagents — 116 reasoning agents with commit-time source-citation enforcement*. https://github.com/cdeust/zetetic-team-subagents - Deust, C. (2026e). *Cortex-cowork — SQLite variant of Cortex for sandboxed environments*. https://github.com/cdeust/Cortex-cowork - diff --git a/docs/program/github-reply-issue-14-a3-delivery.md b/docs/program/github-reply-issue-14-a3-delivery.md index 4a4770d7..fd9e9258 100644 --- a/docs/program/github-reply-issue-14-a3-delivery.md +++ b/docs/program/github-reply-issue-14-a3-delivery.md @@ -58,6 +58,8 @@ from the README. | BEAM-100K MRR (100 Q) | **0.591** | 0.591 | 0.000 | PASS (exact) | | BEAM-100K R@10 | **79.0%** | 79.0% | 0.0 pp | PASS (exact) | +These are the historical v3.11 regression floors used by this A3 run. The LoCoMo 0.794 / 92.6% comparator (n=1982) is superseded for current reporting by the post-fix E1 v3 run (`2f45bcb`, n=1986); it remains here to preserve the original gate evaluation. + All deltas are within the 0.5pp measurement-noise tolerance set in the design doc §8. BEAM-10M runs overnight; adding that result in a follow-up. diff --git a/docs/program/gitnexus-competitive-analysis.md b/docs/program/gitnexus-competitive-analysis.md index f99b4c26..14a91e48 100644 --- a/docs/program/gitnexus-competitive-analysis.md +++ b/docs/program/gitnexus-competitive-analysis.md @@ -57,7 +57,7 @@ are the consensus. | Clustering | Per-domain cognitive profile + cross-domain bridges | Louvain + Traag C2 repair (Blondel 2008, Traag 2019 — cited) | | Scale | see docs/module-inventory.md for module counts (single source, per #127) · 47 MCP tools · 2500+ tests | 12 046 LOC, 23 MCP tools, 220 tests | | Scientific grounding | Every mechanism cites papers: cascade (Kandel 2001), homeostatic (Turrigiano 2008), neuromodulation (Doya 2002), synaptic tagging (Frey & Morris 1997), microglial pruning (Wang 2020), predictive coding (Friston 2010), … | Every stage cites papers: Louvain (Blondel 2008), Traag (2019), RRF K=60 (Cormack et al 2009), Tarjan SCC, tree-sitter … | -| Benchmarks | **LongMemEval R@10 97.8%** (paper SOTA 78.4%); **LoCoMo 92.6%**; **BEAM 0.543** (paper SOTA 0.329) — all on clean DB, reproducible | 220 unit tests; no external benchmark yet | +| Benchmarks | **LongMemEval R@10 98.2%** (paper SOTA 78.4%; clean run SHA `28145f0`); **LoCoMo 94.35%** (post-fix SHA `2f45bcb`); **BEAM retrieval-proxy MRR 0.591** (five-conversation protocol, SHA `a071d89`) | 220 unit tests; no external benchmark yet | | Unique features | persistent cross-session memory, thermodynamic decay, cascade consolidation, neuromodulation, synaptic tagging, cognitive profile per domain, predictive-coding write gate, hippocampal replay | PRD validator (symbol hallucination check), security gates (auth-critical/unsafe/public API), Tarjan-SCC semantic diff, 5-layer resolver with LSP, macro expansion, stdlib indexing | | License | MIT | MIT-equivalent | @@ -87,7 +87,7 @@ The structural asymmetry. |---|---|---|---| | **C1** | Persistent memory across sessions | Full thermodynamic store (`core/thermodynamics.py`) + decay (`core/decay_cycle.py`) + reconsolidation (`core/reconsolidation.py`) — GitNexus is stateless-per-query code intelligence | The headline moat — see §6 | | **C2** | Paper-cited mechanisms | 100+ citations across `core/*.py`; GitNexus cites zero papers | Every challenge to our implementation has a paper retreat; every challenge to theirs has nothing | -| **C3** | Reproducible benchmarks that beat published SOTA | LongMemEval 97.8% vs paper's 78.4%; BEAM 0.543 vs 0.329 | Concrete track record; GitNexus has none | +| **C3** | Reproducible benchmarks that beat published SOTA | LongMemEval R@10 98.2% vs paper's 78.4%; BEAM retrieval-proxy MRR 0.591 on the named five-conversation protocol | Concrete track record; GitNexus has none | | **C4** | Cognitive profile per domain (Felder-Silverman style) | `core/style_classifier.py` + `core/domain_detector.py` + behavioural persona vector | Tailors retrieval to the agent's actual reasoning pattern | | **C5** | Predictive-coding write gate (Friston 2010) | 4-signal novelty filter prevents contaminated memory | Their re-index-on-change model has no write gate — garbage accumulates | | **C6** | Security gates + PRD validator + Tarjan-SCC semantic diff (via AP) | AP `prd_validator.rs`, `security_gates.rs`, `semantic_diff.rs` | Structural-truth layer shields PRDs from symbol hallucination | @@ -101,7 +101,7 @@ The structural asymmetry. Paper citations prevent one specific failure mode: **silent constant drift under benchmark pressure**. -When a benchmark goes from 94% → 97.8%, the temptation is to tune one more constant to get 98.3%. Without a paper anchor, the constant becomes corpus-fitted — a form of overfitting invisible until the next distribution shift (new corpus, new user, new language). With a paper anchor, moving the constant requires either (a) a new paper or (b) a public benchmark measurement — both leave an audit trail. GitNexus has no anchors. Their BM25+RRF fusion has no cited weights; their Leiden resolution parameter is unstated. They can tune freely, overfit invisibly, and collapse silently on the first independent evaluation. +When a benchmark moves from one measured score to another, the temptation is to tune one more constant for a headline gain. Without a paper anchor, the constant becomes corpus-fitted — a form of overfitting invisible until the next distribution shift (new corpus, new user, new language). With a paper anchor, moving the constant requires either (a) a new paper or (b) a public benchmark measurement — both leave an audit trail. GitNexus has no anchors. Their BM25+RRF fusion has no cited weights; their Leiden resolution parameter is unstated. They can tune freely, overfit invisibly, and collapse silently on the first independent evaluation. The moat is **provenance forces honesty**. It's the same moat peer-reviewed science has over blog-driven opinion. diff --git a/docs/provenance/bounded-io-phase2-design.md b/docs/provenance/bounded-io-phase2-design.md index 51cbf305..d275368c 100644 --- a/docs/provenance/bounded-io-phase2-design.md +++ b/docs/provenance/bounded-io-phase2-design.md @@ -133,6 +133,8 @@ New `core/gist_extraction.py` (pure logic) + `infrastructure/artifact_store.py` (fails before F2, passes after); (b) garbage keyword trigger + query containing a partial-word match ⇒ no injection after F1. 2. Full suite: `.venv/bin/pytest tests_py -q` (3,173+ passing baseline). -3. Benchmarks: LongMemEval s, LoCoMo, BEAM on clean DB vs recorded baselines - (R@10 98.4 / 94.2, BEAM 0.591). No regression accepted. +3. Benchmarks: LongMemEval s, LoCoMo, BEAM on clean DB vs the then-recorded + protocol-specific baselines (R@10 98.4 / 94.35 post-fix, BEAM 0.591). + These are historical regression gates, not current publication headlines. + No regression accepted. 4. open_visualization + commit per repo; PUSH NOTHING. diff --git a/docs/provenance/bounded-io-plan.md b/docs/provenance/bounded-io-plan.md index 7d0be1a6..eba14e4b 100644 --- a/docs/provenance/bounded-io-plan.md +++ b/docs/provenance/bounded-io-plan.md @@ -66,8 +66,8 @@ most of this plan ALREADY LANDED in prior sessions; the checkboxes below were st amplification (M2) and the source/confidence structural gap (M3) sat beneath it. - Gate results: full suite 3207 passed (baseline 3173 + exactly 34 new tests); repro tests fail pre-fix / pass post-fix (verified via git stash); ruff clean. -- Benchmarks (clean DB): LongMemEval R@10 98.4% (=), MRR 0.916 (≥0.9124); - LoCoMo MRR 0.828 (=0.8278), R@10 94.1% (94.2 baseline, 1982 vs 1986 Qs); +- Historical Phase 2 validation benchmarks (clean DB; protocol-specific, not current headlines): LongMemEval R@10 98.4% (=), MRR 0.916 (≥0.9124); + LoCoMo MRR 0.828 (=0.8279), R@10 94.1% (94.35% post-fix reference, 1982 vs 1986 Qs); BEAM 100K re-based to 395 Qs — A/B old 0.502 vs new 0.501, regression-free. - Production data ops: 319 keyword triggers deactivated (reversible); created_by column arrives at server restart (DDL); 6,799 existing raw blobs diff --git a/docs/provenance/verification-measurement-discipline.md b/docs/provenance/verification-measurement-discipline.md index 63fa81a5..02263fc3 100644 --- a/docs/provenance/verification-measurement-discipline.md +++ b/docs/provenance/verification-measurement-discipline.md @@ -102,9 +102,9 @@ Forbidden: filling a cell with an estimate, an interpolation, or a "should be ro ## Measurement-Debt List (ordered by impact-if-wrong) -1. **LongMemEval R@10 = 97.8% / MRR = 0.882 (CLAUDE.md April 2026).** Provenance unknown — single run? Aggregated? 95% CI? **Status: I don't know.** Required: rerun ×5 on locked embedding model + commit `bench/results/longmemeval__.json` with per-run scores. -2. **BEAM Overall = 0.543.** Same questions. Higher impact-if-wrong because the headline beats best-in-paper (0.329) by a large margin — a single-run inflation here is the most damaging citation in the campaign. -3. **LoCoMo R@10 = 92.6% / MRR = 0.794.** Same questions. Same fix. +1. **Historical LongMemEval R@10 = 97.8% / MRR = 0.882 (CLAUDE.md April 2026).** Provenance of that original pair remains unknown. It is superseded for current reporting by the clean run at SHA `28145f0` (R@10 98.2%, MRR 0.9167, n=500); the multi-run noise-floor experiment remains open. +2. **Historical BEAM Overall = 0.543.** Provenance of that old headline remains unresolved. It is superseded for current reporting by the named five-conversation / 100-question retrieval-proxy run at SHA `a071d89` (MRR 0.591); do not conflate it with the later 395-question full split. +3. **Historical LoCoMo R@10 = 92.6% / MRR = 0.794 (superseded; April 2026, n=1982).** The publication commit is `b4057a`, but the original per-query artefact is not committed. Same fix. 4. **All ablation Δs in `bench/beam/ablation_results.json`.** No σ. Cannot distinguish 1pp signal from 1pp noise. 5. **All latency claims (`<50ms`, `<100ms`, `<200ms` in CLAUDE.md tool table).** Targets, not measurements. No published p99. Required: emit p50/p99/p999 from a real workload trace. 6. **Cross-benchmark transfer claims.** Currently zero published numbers; any future claim must follow E5 protocol from day one. diff --git a/docs/provenance/verification-protocol.md b/docs/provenance/verification-protocol.md index ee45dbc1..610905ce 100644 --- a/docs/provenance/verification-protocol.md +++ b/docs/provenance/verification-protocol.md @@ -308,18 +308,22 @@ SHA-256 of generated corpus. **Hypothesis (confirmatory).** With ALL hyperparameters frozen at their LongMemEval-tuned values (committed at protocol freeze, hashed), LoCoMo MRR is within 0.08 of its -standalone-tuned MRR (currently 0.794). I.e., MRR(LoCoMo, frozen-config) ≥ 0.714. +protocol-frozen historical standalone-tuned MRR (0.794, superseded for current +reporting). I.e., MRR(LoCoMo, frozen-config) ≥ 0.714. -**Falsification.** MRR(LoCoMo, frozen) < 0.714 → C5 refuted. The 97.8% LongMemEval is -a calibration artefact, not a generalizable result. +**Falsification.** MRR(LoCoMo, frozen) < 0.714 → C5 refuted. The current +LongMemEval result (R@10 98.2%, MRR 0.9167; clean run at SHA `28145f0`) +would then be a calibration artefact, not a generalizable result. **Design.** - Factor: configuration source ∈ {LongMemEval-tuned (frozen)}. No tuning loop on LoCoMo. -- Single arm; the comparator is the historical LoCoMo standalone-tuned score from - CLAUDE.md (MRR=0.794). This is a one-arm pre-registered claim. +- Single arm; the comparator is the historical April 2026 clean-DB LoCoMo + standalone-tuned score (MRR=0.794, n=1982), first published in commit + `b4057a`. This superseded value remains frozen here because changing a + pre-registered comparator after the fact would invalidate the protocol. - Replications: 3 seeds (DB load order). LoCoMo is 1986 Q — high statistical power. -- Zero-cell: prior published MRR=0.794 (read from `docs/benchmarks/` snapshot - at protocol freeze). +- Zero-cell: superseded historical MRR=0.794 (read from the repository snapshot + at protocol freeze; original per-query artefact was not committed). - Blocking: same hardware, embedding cache, Postgres binary as E1's LoCoMo block. **Sample size / power.** 1986 Q × 3 seeds = 5958 evaluations. Paired bootstrap on MRR.