Model-capability guardrails + larger-model prerequisites (anti-fabrication, KV-cache quant, per-model n_ctx, Gemma-12B tool parsing)

[malibio]

Overview

The A/B trial in #1329 found the local ai-chat agent is capability-bound in two distinct ways, and that simply swapping in a larger model (Gemma 4 12B Q4_K_M) does not help on the 16 GB target hardware — it overshoots memory at the 32K context budget, and at a reduced 8K context it stops calling tools and fabricates tool results (claims it created/updated/searched nodes it never touched). See the trial results.

This issue tracks model-capability guardrails plus the concrete engine/loop unlocks the trial surfaced, so a future larger-model attempt is actually testable.

Problem statement

On the corrected loop (#1331) the remaining rough edges are model-capability limits, not infrastructure bugs:

• E4B over-eagerly calls tools on conversational turns, duplicates create_schema, is verbose, and sometimes mis-routes (couldn't filter "list all", update_node errored).
• 12B@8K is worse on correctness: it narrates fictional successes ("created invoice ID 104", "marked invoice 124 paid", "found the Project Alpha $500 payment") with zero tool calls — actively misleading — and leaks a bare thought preamble on every reply.

We need (a) runtime guardrails that keep the agent honest within whatever model is loaded, and (b) the prerequisites that would let a larger model be re-evaluated fairly.

Proposed work

A. Capability guardrails (original #1329 follow-up intent)

• Detect requests beyond the loaded model's reliable envelope (e.g. multi-step CRUD across several custom types in one turn) and ask the user to simplify / break it up, rather than attempting and failing/fabricating.
• Anti-fabrication guard: never present "I created/updated/found X" unless a corresponding tool actually executed in this turn. If the model claims an action with tool_calls=0, suppress the claim (or convert to "I can do that — confirm?"). This directly addresses the 12B fabrication failure and also catches E4B narrating beyond what its tools returned.
• Surface tool failures honestly to the user instead of papering over them (cf. E4B's update_node error and "couldn't list all").

B. Larger-model prerequisites (discovered in the trial — without these, no Gemma > E4B is viable here)

• KV-cache quantization: add type_k/type_v (q8_0/q4_0) to ChatConfig and wire them into the llama.cpp context build (packages/nlp-engine/src/chat/). The 12B blocker was the f16 KV cache (10.24 GB @32k), not the Q4_K_M weights — quantizing K/V is the real headroom unlock.
• Per-model n_ctx: source the runtime context window from the catalog entry's context_window instead of the hardcoded n_ctx: 32_768 in ChatConfig::default() (packages/nlp-engine/src/chat/types.rs), so a larger model can run a smaller context without a global edit.
• Gemma-4-12B tool-call parsing: the loop's tool-call/turn-structure handling (tuned for E4B) does not recognize 12B's chat template (leading thought channel, <|tool_response>/<turn|> EOG tokens), so tool calls are emitted as prose. Generalize parsing across Gemma-4 sizes, including stripping the unmarked bare-thought narration (deferred from Capture and surface the local agent's reasoning as a structured, collapsible section #1330 because it carries no <|channel> marker to regex-strip safely).

Acceptance criteria

• Agent never claims an action it did not execute via a tool (verified on the A/B test Gemma 4 12B (Q4_K_M) vs E4B for the local ai-chat agent #1329 scenario matrix).
• Over-scope requests get a "please simplify" response instead of a failed/fabricated attempt.
• ChatConfig supports KV-cache quantization and per-model n_ctx; documented memory footprints for E4B and 12B at the chosen settings.
• 12B (or another candidate) re-run on the A/B test Gemma 4 12B (Q4_K_M) vs E4B for the local ai-chat agent #1329 matrix with tool calls correctly parsed, as a gate for any default-model change.

Related

• A/B test Gemma 4 12B (Q4_K_M) vs E4B for the local ai-chat agent #1329 (A/B trial that produced these findings)
• Capture and surface the local agent's reasoning as a structured, collapsible section #1330 (reasoning UI; the marked-<|channel> half — the unmarked bare-thought strip belongs here)

[malibio]

Frontend baseline: 134 files, 3844 tests passed