CUDA streaming: per-size-class expert caches, keep cache warm across prefills

[iCreil]

Fixes #503.

See the issue for the full analysis and benchmarks. Summary of the three changes, all in ds4_cuda.cu:

1. Per-size-class expert caches. g_stream_expert_cache becomes a fixed array of 4 caches keyed by (gate_expert_bytes, down_expert_bytes). note_size() is replaced by class_index() (assigns a free slot on first sight; recycles class 0 in the — never observed — case of more than 4 distinct sizes). Runtime caps and cap-notice dedup move to per-class arrays. prepare() operates on the class cache; cross-class transitions no longer release anything.

2. Conditional release in begin_compact_load. The unconditional release_all() on the allow_global_cache=0 path becomes: try ensure_slots() first; on allocation failure release the expert caches and retry once. Long-prompt VRAM pressure is still handled, short prompts keep the warm cache.

3. Prefill batch reads the cache. prepare_selected_batch now passes allow_global_cache=1 plus a new allow_cache_evict=0 mode: batch loads copy hits device-to-device and may append to free capacity, but never evict valid slots (an LRU cache smaller than a long prompt's expert union would otherwise be fully cycled, destroying the decode working set). The single-token decode path keeps full LRU behaviour (allow_cache_evict=1).

Notes:

• The memset over the struct containing a std::vector is replaced by explicit member resets (vector().swap()), which also fixes a latent leak of the slots buffer.
• Budget semantics: a count budget now applies per size class, so classes can overcommit in aggregate; the lazily-growing model-range arena can then fail mid-decode. A shared byte budget across classes would be safer — left out of scope here to keep the diff focused, happy to follow up. Practical guidance for the 96GB + Flash q2-q4 case is in the issue (budget 9472, reserve ≥ 12 GB).
• Tested on RTX PRO 6000 Blackwell 96GB / CUDA 13.1 / make cuda-generic with Flash q2-imatrix (uniform: behaviour unchanged) and Flash q2-q4-imatrix (fixed: ~0.9 → ~25 t/s sustained decode, 97% cache hit rate per DS4_CUDA_STREAMING_EXPERT_CACHE_VERBOSE).

[iCreil]

Some real-workload numbers from running this PR against the mixed-quant file it targets (DeepSeek V4 Flash, 37 routed layers IQ2XXS/Q2K + 6 layers with Q4_K experts, 91 GB GGUF fully page-cache resident; RTX PRO 6000 Blackwell 96 GB, --ssd-streaming --ssd-streaming-cache-experts 9472).

Chat-shaped test against ds4-server (4 short questions, 160 generated tokens each, 3 rounds, default temperature):

• without this PR: 1.36 tok/s decode, flat across all rounds — the cache never warms up, even when the exact same question is repeated;
• with this PR: 19.3–19.4 tok/s from the second round on (~14x), ~99% cache hit rate.

A timing probe around the selected-expert load shows where the time was going: without the fix each layer's load costs ~10.6 ms, of which only ~1.1 ms is the actual H2D wait — the other ~9.5 ms is host-side cudaFree/cudaMalloc churn from the release_all() on every size-class flip, with a permanent 0% hit rate. With the per-class caches the same load drops to ~0.17 ms/layer.

Happy to share the exact scripts and configs if useful.