CUDA: batch gate/up/down uploads for selected expert cache misses

[fmolara]

Motivation

In the CUDA direct-I/O path, a selected expert cache miss uploads the gate/up/down tensors independently, synchronizing the selected upload stream after each tensor.

This patch batches the three uploads belonging to the same expert and synchronizes the selected upload stream once after all three have been enqueued.

Properties

• no API changes
• no kernel changes
• no cache policy changes
• resident and compact cache layouts are unchanged
• unsupported cases fall back to the existing path

Validation

Tested after rebasing onto efdadd4, which was the current main at the time, on an NVIDIA A100 80GB PCIe with CUDA 12.4 (sm_80).

• Baseline commit: efdadd4
• PR commit: 6ece876
• Clean CUDA builds: passed
• Deterministic output: identical
• Normalized output SHA-256: identical
• No CUDA staging, upload, or synchronization errors

Targeted miss-heavy Nsight Systems validation

Expert cache	Metric	Baseline	PR	Difference
8GB	cudaStreamSynchronize calls	108,276	36,092	−66.67%
8GB	cudaStreamSynchronize API time	1.649 s	0.587 s	−64.42%
16GB	cudaStreamSynchronize calls	108,276	36,092	−66.67%
16GB	cudaStreamSynchronize API time	1.742 s	0.584 s	−66.49%

The number of cudaMemcpyAsync calls remained unchanged at 108,276.

This confirms that the patch preserves the same gate/up/down uploads while reducing synchronization from once per tensor upload to once per three-upload group.

End-to-end throughput differences were within normal run-to-run noise:

Expert cache	Metric	Baseline median	PR median
8GB	Prefill	2.45 t/s	2.41 t/s
8GB	Generation	0.65 t/s	0.65 t/s
8GB	Wall time	220.53 s	221.08 s
16GB	Prefill	2.40 t/s	2.42 t/s
16GB	Generation	0.64 t/s	0.64 t/s
16GB	Wall time	221.89 s	222.31 s

This change should therefore be considered a synchronization-granularity improvement, not a demonstrated overall throughput improvement on the current upstream code.

[fmolara]

Rebased onto current main and adapted the patch to the refactored selected expert cache path.

The change remains narrowly scoped:

• gate/up/down use three distinct staging slots;
• all three copies are enqueued on the selected upload stream;
• the stream is synchronized once after the three uploads;
• unsupported cases transparently use the existing three-copy fallback;
• pending stream work is drained on partial failure;
• cache policy, tensor layouts, kernels, and diagnostics are unchanged.

Both the current baseline (efdadd4) and the PR commit (6ece876) build successfully with CUDA 12.4 on an NVIDIA A100 80GB PCIe (sm_80).

All validation runs produced identical deterministic output and the same normalized output SHA-256, with no CUDA staging, upload, or synchronization errors.

A targeted Nsight Systems comparison confirmed the intended effect:

• cudaMemcpyAsync: unchanged at 108,276 calls

• cudaStreamSynchronize: 108,276 → 36,092 calls (−66.67%)

• cudaStreamSynchronize API time:

  • 8GB expert cache: 1.649 s → 0.587 s (−64.42%)
  • 16GB expert cache: 1.742 s → 0.584 s (−66.49%)

This removes exactly 72,184 stream synchronizations in the profiled workload while preserving the upload count and output correctness.

End-to-end throughput remained within run-to-run noise, so the validation notes no longer claim a measurable overall throughput improvement on the current main.