disagg: RPC batching, KV prefetch streaming, sync fence, disagg fixes

[denmrnngp-cloud]

Hi, thanks for the disagg prefill/decode work — I've been running it on a direct 10GbE link between a Mac Studio (M2 Ultra, Metal decode) and a DGX Spark (GB10, CUDA prefill), on Qwen3.6-27B and Qwen3.6-35B-A3B. This PR is a set of patches on top of poc/server-disagg-prefill-decode (based on f4a0744), plus the numbers I measured along the way, in case any of it is useful upstream.

Patches

1. has_mtmd = !files.empty() in process_mtmd_prompt (server-common.cpp) — as written, having an --mmproj loaded unconditionally disabled disagg prefill even for pure-text requests. This mattered a lot since vision models are common now (Qwen3.6-VL etc.) and I didn't want to give up disagg just because the model can see images.

2. Skip disagg when the slot has a cached common prefix (server-context.cpp) — disagg re-establishes the whole prefix KV on every request by design, which is right for a cold prompt but wasteful for multi-turn conversations where most of the prefix is already resident locally. Added a check on get_common_prefix() before triggering the disagg path, so follow-up turns extend the local KV instead of re-shipping everything through the remote prefill device.

3. Batched RPC tensor fetch — new RPC_CMD_GET_TENSOR_BATCH (proto v4.1). The KV handoff after disagg prefill was doing one round-trip per tensor, which capped wire throughput at ~3.8 Gbps on our 10GbE link regardless of MTU/buffer tuning. Batching the fetch into one round trip got us to ~9.8 Gbps (essentially wire speed).

4. 32MB SO_RCVBUF/SNDBUF on the RPC sockets (transport.cpp) — default OS buffers were bottlenecking the bulk tensor transfers independent of (3).

5. ggml_backend_rpc_synchronize is currently a no-op — graph_compute over RPC is fire-and-forget (no response expected), so there's nothing to block on. I think this is worth flagging as a correctness issue, not just a perf one: on a slot doing multiple RPC graphs back to back (e.g. disagg prefill chunks), a caller relying on synchronize as a barrier can read a buffer before the queued work has actually completed on the remote side. I made it issue a cheap round-trip command (DEVICE_COUNT) as a real fence — since the remote processes commands in order, getting a response means the queue up to that point has drained.

6. KV prefetch streaming during prefill — this was the biggest win. Added llama_state_seq_prefetch_ext → llama_memory_i::state_prefetch (implemented for llama_kv_cache, forwarded through llama_memory_hybrid — Qwen3.6 is a hybrid recurrent/attention architecture, so the forwarding turned out to be required, not optional) → ggml_backend_rpc_prefetch_tensor_batch, a fire-and-forget prefetch request whose response is drained lazily into a client-side region cache keyed by (remote tensor ptr, offset). Called per-chunk inside disagg_prefill_batch, with the per-chunk llama_synchronize removed since queue ordering is enough to guarantee correctness. Net effect: the attention-KV transfer for a chunk is fully hidden behind the next chunk's remote compute, instead of happening as a separate blocking step after prefill finishes.

Numbers (Qwen3.6-27B UD-Q4_K_XL, Mac Studio M2 Ultra ↔ DGX Spark GB10, direct 10GbE, MTU 8000)

KV handoff, 28.8K-token prompt: 1264ms → 544ms after patch 6 (86% of the ~1.1GB transfer served from the prefetch cache; the residual ~150ms is the fixed 149.6MiB recurrent DeltaNet state, which is not prefetchable, plus the final chunk's compute).

End-to-end, same 28.8K-token prompt: prefill 48.9s on Spark (~590 tok/s) + 0.57s handoff + 31.3s decode on Mac (~12.8 tok/s at that depth).

Cluster vs. single-node (same ~2200-token prompt, temp=0, same model):

Mode	Prompt processing	Decode	Total (300 tok)
Cluster (Spark prefill + Mac decode)	4.4s	26.8 tok/s	15.6s
Mac only (Metal, no RPC)	8.3s	24.2 tok/s	20.7s
Spark only (CUDA via RPC, --device RPC0)	4.3s	9.1 tok/s	37.3s

Spark-only decode being ~2.7x slower than Mac isn't a disagg issue — it's GB10 being memory-bandwidth-bound on single-batch autoregressive decode, consistent with what others have reported for this hardware. The disaggregated split plays to each device's strength: CUDA for parallel prefill, Metal for decode.

Depth scaling (llama-bench, tg128, cluster prefill device):

Depth	pp512 (tok/s)	tg128 (tok/s)
0	333-335	24.9-25.0
16384	246-250	16.0-16.2
65536	133-134	7.8

Speculative decoding (--spec-type ngram-mod) stacks fine with disagg, but with a caveat: on the first disagg request in a session, draft-accept rate is low (observed ~9%) because the CUDA prefill and Metal decode produce numerically slightly different logits at temp=0 (verified — outputs diverge starting a few dozen tokens in on identical greedy-decode requests), so n-grams learned from one don't transfer cleanly to the other on that first pass. On follow-up turns disagg is skipped anyway (see patch 2), so ngram-mod runs at full effectiveness there. Net gain on repetitive/code-edit workloads: +39% (24→33.4 tok/s) on a code-edit task, up to 2.1x on highly repetitive prompts, neutral (no regression) on fresh text.

Layer-split mode (--device MTL0,RPC0 --split-mode layer --tensor-split, no --prefill-device) for models too large for either machine alone — tested on Qwen3.6-35B-A3B-Q8 (34GB) split 55/45: decode 18.6 tok/s vs. 65.6 tok/s running the same quant Mac-only. Expected — every token's activations now cross the 10GbE link — but it's the only way to run something that genuinely doesn't fit in 147GB (Mac) or 119GB (Spark) alone, giving ~260GB combined. Not something I'd use unless the model actually requires it.

Network / deployment notes (not code changes, but cost me real debugging time)

• Direct link, no switch: two 10GbE NICs cabled point-to-point on a dedicated /24, separate from the management LAN used for SSH. Keeps the RPC path off any shared network.
• MTU 8000, not 9000: the Spark's onboard Realtek NIC caps out around 8178 bytes — MTU 9000 silently doesn't pass end-to-end on that chip. 8000 is the largest value that reliably works on both ends.
• Socket tuning beyond the buffer patch above: net.core.wmem_max/rmem_max = 16MB and net.ipv4.tcp_slow_start_after_idle=0 on the Spark side (persisted via /etc/sysctl.d/), kern.ipc.maxsockbuf=16777216 on the Mac (/etc/sysctl.conf) plus networksetup -setMTU. Without tcp_slow_start_after_idle=0 specifically, throughput on a request after any idle gap re-ramps from a slow-start window instead of going straight to line rate — noticeable since our traffic pattern is bursty (idle between requests, then a burst per prefill).
• macOS "Local Network" permission silently breaks RPC — worth a README callout for other Mac users of this fork. On macOS 15.x, any non-system process (including a plain llama-server binary run as a normal user) that tries to connect() to the RPC port gets errno 65 ("No route to host") even though the host is reachable — nc/system tools connect fine, confirming it's an app-level Local Network permission gate, not routing. There's no user-facing prompt for this in our testing; it just fails. The workaround we're running with is launching llama-server from a root-owned LaunchDaemon (root is exempt from the check), but that's a workaround, not a fix — a note in the README about this would save the next Mac user a confusing debugging session, since the error message points at routing/firewall, not permissions.
• Both binaries must be rebuilt together on any RPC protocol change — the hello handshake checks major equal / server minor <= client, so a stale rpc-server on the remote box fails closed rather than degrading, which is good, but easy to trip over when only rebuilding one side.
• Reliability: rpc-server on the CUDA box under a systemd unit (Restart=always) and the router llama-server on macOS under the LaunchDaemon mentioned above — both survive a reboot and a crash without manual intervention. Worth doing before treating this as more than a demo, since a single stuck RPC connection previously required killing and restarting both processes.

Scope of this PR

Self-contained on top of f4a0744, ~15 files touched across ggml/src/ggml-rpc/, src/llama-kv-cache.{cpp,h}, src/llama-memory-hybrid.{cpp,h}, and tools/server/server-context.cpp. Happy to split it up by concern if you'd rather review RPC batching / prefetch streaming / disagg fixes separately — they're fairly independent of each other.