ggml : implement fast walsh-hadamard transform for kv rotation (#21352)#22631
Merged
Conversation
github-actions
Bot
added
the
ggml
am17an
requested changes
May 3, 2026
am17an
left a comment
am17an
left a comment
Contributor
There was a problem hiding this comment.
You should use ggml_vec* methods
AlrIsmail
force-pushed
the
feature/fwht-kv-rotation
branch
from
7eafd3b to
490a22c
Compare
am17an
reviewed
May 3, 2026
AlrIsmail
force-pushed
the
feature/fwht-kv-rotation
branch
2 times, most recently
from
5eed37c to
ba5ce01
Compare
Contributor
Author
|
@am17an I initially tried using the ggml_vec_add_f32 and ggml_vec_sub_f32 functions from vec.h, but the benchmark tanked. Is there a reason to not to use the internal GGML_F32_VEC_* macros instead? |
Contributor
|
Yes any generic API is fine |
am17an
reviewed
May 3, 2026
am17an
left a comment
am17an
left a comment
Contributor
There was a problem hiding this comment.
you need to also add a test-case in test-backend-ops which exercise this path
AlrIsmail
force-pushed
the
feature/fwht-kv-rotation
branch
2 times, most recently
from
fcb2c79 to
4320d7d
Compare
am17an
reviewed
May 3, 2026
am17an
left a comment
am17an
left a comment
Contributor
There was a problem hiding this comment.
remove the binary file benchmark_fwht_test
AlrIsmail
force-pushed
the
feature/fwht-kv-rotation
branch
2 times, most recently
from
88c6e08 to
d1a46e4
Compare
am17an
approved these changes
May 4, 2026
Contributor
AlrIsmail
force-pushed
the
feature/fwht-kv-rotation
branch
from
d1a46e4 to
3abc7f8
Compare
ggerganov
approved these changes
May 4, 2026
samuraieng
pushed a commit
to samuraieng/llama.cpp
that referenced
this pull request
May 6, 2026
4 tasks
1 task
cetarthoriphros
pushed a commit
to cetarthoriphros/llama.cpp
that referenced
this pull request
May 9, 2026
meh
pushed a commit
to meh/llama.cpp
that referenced
this pull request
May 10, 2026
muxanick
added a commit
to muxanick/llama.cpp
that referenced
this pull request
May 19, 2026
Closes the CUDA gap left by PR ggml-org#22631 (CPU FWHT for KV cache rotation, merged 2026-05-05). When ggml_mul_mat_set_hint(t, GGML_HINT_SRC0_IS_HADAMARD) is set on a MUL_MAT node, the CPU backend replaces the O(N^2) literal matmul against the Hadamard matrix with an O(N log N) Fast Walsh-Hadamard Transform via ggml_compute_forward_fwht. The CUDA backend previously fell through to a literal mul_mat which produced numerically equivalent output (since the scaled Hadamard matrix dotted with the input equals FWHT mathematically) but did O(N^2) work. This commit adds: - ggml/src/ggml-cuda/fwht.cuh, fwht.cu: a CUDA FWHT kernel mirroring the CPU algorithm — one block per row, threads cooperate via shared memory across log2(n) butterfly passes after a 1/sqrt(n) scale. Block size = 128 (4 warps); shared-memory budget caps n at 1024 elements per row (current upstream callers in src/llama-graph.cpp and src/llama-kv-cache.cpp use head_dim, typically 64-128, so this is fine). - ggml-cuda.cu: dispatch hook at the start of ggml_cuda_mul_mat that checks op_params[1] against GGML_HINT_SRC0_IS_HADAMARD and routes to ggml_cuda_op_fwht when set. - common.cuh: added #include <atomic> next to the other standard includes. The std::atomic field added in the prior bundle commit is reachable from translation units that don't transitively pull in <atomic> via other headers; the new fwht.cu translation unit is one such case and surfaced the missing include during build. - BUNDLE_README.md: c4 entry added; the corresponding entry under "What's deliberately NOT in this bundle" updated to acknowledge the gap is now closed. Validation: test-backend-ops MUL_MAT_HADAMARD on CUDA passes byte-identically to CPU reference for n = 64, 128, 256 (single batch) and n = 128 (batch = 32) — the 4 test cases added by PR ggml-org#22631. test-backend-ops GET_ROWS continues to pass 67/67 (regression check for c1 from the prior bundle commit). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
baramofme
pushed a commit
to baramofme/llama-cpp-turboquant
that referenced
this pull request
May 23, 2026
carlosfundora
pushed a commit
to carlosfundora/llama.cpp-1-bit-turbo
that referenced
this pull request
May 24, 2026
…org#21352) (ggml-org#22631) (cherry picked from commit a817a22)
winstonma
pushed a commit
to winstonma/llama.cpp
that referenced
this pull request
May 27, 2026
fewtarius
pushed a commit
to fewtarius/CachyLLama
that referenced
this pull request
May 30, 2026
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jun 20, 2026
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Resolves #21352
Replaces the$O(N^2)$ matrix multiplication for Hadamard KV-cache rotation with an $O(N \log N)$ Fast Walsh-Hadamard Transform (FWHT).
Instead of adding a new
ggml_op, this uses a hint flag (GGML_HINT_SRC0_IS_HADAMARD) to dynamically route the computation to the FWHT kernel inggml-cpuAdditional information
The FWHT kernel uses a hybrid SIMD approach: a clean scalar loop for initial passes, and AVX/AVX-512/NEON intrinsics for the main workload. This maximizes performance while avoiding complex code.
Benchmarks (Apple Silicon M-series, N=128, M=4096, 8 Threads):
mul_mat): 1285.84 msRequirements