opencl: add q5_K gemm and gemv kernels for Adreno#21595
Merged
Conversation
github-actions
Bot
added
ggml
lhez
approved these changes
Apr 16, 2026
max-krasnyansky
approved these changes
Apr 16, 2026
max-krasnyansky
left a comment
max-krasnyansky
left a comment
Member
There was a problem hiding this comment.
Nice to see Q5_K. Will get started on the Hexagon version too :)
cnsiva
pushed a commit
to saas-home/llama.cpp
that referenced
this pull request
Apr 17, 2026
samuraieng
pushed a commit
to samuraieng/llama.cpp
that referenced
this pull request
Apr 19, 2026
mengqin
pushed a commit
to mengqin/llama.cpp
that referenced
this pull request
Apr 20, 2026
ArberSephirotheca
pushed a commit
to ArberSephirotheca/llama.cpp
that referenced
this pull request
Apr 21, 2026
arthw
pushed a commit
to arthw/llama.cpp
that referenced
this pull request
Apr 23, 2026
ljubomirj
pushed a commit
to ljubomirj/llama.cpp
that referenced
this pull request
May 6, 2026
my-other-github-account
pushed a commit
to my-other-github-account/llama.cpp
that referenced
this pull request
May 15, 2026
my-other-github-account
pushed a commit
to my-other-github-account/llama.cpp
that referenced
this pull request
May 15, 2026
fewtarius
pushed a commit
to fewtarius/CachyLLama
that referenced
this pull request
May 30, 2026
wanghqc
added a commit
to qualcomm/llama.cpp
that referenced
this pull request
Jun 11, 2026
a2366b5 sent all large-N (n_tokens >= 512) Q5_K mul_mats to CPU on the premise that Q5_K has no Adreno GEMM and therefore always takes the GEMV that corrupts large-batch prefill. That premise was wrong: Q5_K has had a trans-weight GEMM + GEMV since PR ggml-org#21595 (e45dbde), fully wired through ggml_cl_mul_mat_q5_K_f32_adreno and the set_tensor noshuffle convert. Q5_K weights that pass use_adreno_kernels (dims >= 512) already dispatch to that GEMM at prefill; the GEMV path is only reached by small/odd-shaped projections. Verified the GEMM is correct at large N: greedy generation over a 720-token prompt on Qwen3.5-0.8B (Q5_K SSM in/out proj) is coherent and matches the CPU reference (modulo expected FP drift in the tail). The garbage batched perplexity is a separate head_dim>=256 intermediate-logit issue that persists with the CPU fallback too and does not affect generation. Add Q5_K to type_has_gemm so large-N Q5_K weights stay on the GPU GEMM instead of falling back to CPU; small/odd Q5_K projections still fall back (they would take the buggy GEMV). Removes the per-layer GPU->CPU->GPU round-trip the fallback forced on the SSM projections. Clean serial A/B on Qwen3.5-0.8B (fa=1, ngl=99, -r 3): pp1024 1275->1381 (+8.3%), pp4096 957->1070 (+11.8%), pp4096@d8192 453->490 (+8.2%). Decode and N<512 unaffected.
wanghqc
added a commit
to qualcomm/llama.cpp
that referenced
this pull request
Jun 15, 2026
a2366b5 sent all large-N (n_tokens >= 512) Q5_K mul_mats to CPU on the premise that Q5_K has no Adreno GEMM and therefore always takes the GEMV that corrupts large-batch prefill. That premise was wrong: Q5_K has had a trans-weight GEMM + GEMV since PR ggml-org#21595 (e45dbde), fully wired through ggml_cl_mul_mat_q5_K_f32_adreno and the set_tensor noshuffle convert. Q5_K weights that pass use_adreno_kernels (dims >= 512) already dispatch to that GEMM at prefill; the GEMV path is only reached by small/odd-shaped projections. Verified the GEMM is correct at large N: greedy generation over a 720-token prompt on Qwen3.5-0.8B (Q5_K SSM in/out proj) is coherent and matches the CPU reference (modulo expected FP drift in the tail). The garbage batched perplexity is a separate head_dim>=256 intermediate-logit issue that persists with the CPU fallback too and does not affect generation. Add Q5_K to type_has_gemm so large-N Q5_K weights stay on the GPU GEMM instead of falling back to CPU; small/odd Q5_K projections still fall back (they would take the buggy GEMV). Removes the per-layer GPU->CPU->GPU round-trip the fallback forced on the SSM projections. Clean serial A/B on Qwen3.5-0.8B (fa=1, ngl=99, -r 3): pp1024 1275->1381 (+8.3%), pp4096 957->1070 (+11.8%), pp4096@d8192 453->490 (+8.2%). Decode and N<512 unaffected.
wanghqc
added a commit
to qualcomm/llama.cpp
that referenced
this pull request
Jul 2, 2026
a2366b5 sent all large-N (n_tokens >= 512) Q5_K mul_mats to CPU on the premise that Q5_K has no Adreno GEMM and therefore always takes the GEMV that corrupts large-batch prefill. That premise was wrong: Q5_K has had a trans-weight GEMM + GEMV since PR ggml-org#21595 (e45dbde), fully wired through ggml_cl_mul_mat_q5_K_f32_adreno and the set_tensor noshuffle convert. Q5_K weights that pass use_adreno_kernels (dims >= 512) already dispatch to that GEMM at prefill; the GEMV path is only reached by small/odd-shaped projections. Verified the GEMM is correct at large N: greedy generation over a 720-token prompt on Qwen3.5-0.8B (Q5_K SSM in/out proj) is coherent and matches the CPU reference (modulo expected FP drift in the tail). The garbage batched perplexity is a separate head_dim>=256 intermediate-logit issue that persists with the CPU fallback too and does not affect generation. Add Q5_K to type_has_gemm so large-N Q5_K weights stay on the GPU GEMM instead of falling back to CPU; small/odd Q5_K projections still fall back (they would take the buggy GEMV). Removes the per-layer GPU->CPU->GPU round-trip the fallback forced on the SSM projections. Clean serial A/B on Qwen3.5-0.8B (fa=1, ngl=99, -r 3): pp1024 1275->1381 (+8.3%), pp4096 957->1070 (+11.8%), pp4096@d8192 453->490 (+8.2%). Decode and N<512 unaffected.
wanghqc
added a commit
to qualcomm/llama.cpp
that referenced
this pull request
Jul 2, 2026
a2366b5 sent all large-N (n_tokens >= 512) Q5_K mul_mats to CPU on the premise that Q5_K has no Adreno GEMM and therefore always takes the GEMV that corrupts large-batch prefill. That premise was wrong: Q5_K has had a trans-weight GEMM + GEMV since PR ggml-org#21595 (e45dbde), fully wired through ggml_cl_mul_mat_q5_K_f32_adreno and the set_tensor noshuffle convert. Q5_K weights that pass use_adreno_kernels (dims >= 512) already dispatch to that GEMM at prefill; the GEMV path is only reached by small/odd-shaped projections. Verified the GEMM is correct at large N: greedy generation over a 720-token prompt on Qwen3.5-0.8B (Q5_K SSM in/out proj) is coherent and matches the CPU reference (modulo expected FP drift in the tail). The garbage batched perplexity is a separate head_dim>=256 intermediate-logit issue that persists with the CPU fallback too and does not affect generation. Add Q5_K to type_has_gemm so large-N Q5_K weights stay on the GPU GEMM instead of falling back to CPU; small/odd Q5_K projections still fall back (they would take the buggy GEMV). Removes the per-layer GPU->CPU->GPU round-trip the fallback forced on the SSM projections. Clean serial A/B on Qwen3.5-0.8B (fa=1, ngl=99, -r 3): pp1024 1275->1381 (+8.3%), pp4096 957->1070 (+11.8%), pp4096@d8192 453->490 (+8.2%). Decode and N<512 unaffected.
wanghqc
added a commit
to qualcomm/llama.cpp
that referenced
this pull request
Jul 8, 2026
a2366b5 sent all large-N (n_tokens >= 512) Q5_K mul_mats to CPU on the premise that Q5_K has no Adreno GEMM and therefore always takes the GEMV that corrupts large-batch prefill. That premise was wrong: Q5_K has had a trans-weight GEMM + GEMV since PR ggml-org#21595 (e45dbde), fully wired through ggml_cl_mul_mat_q5_K_f32_adreno and the set_tensor noshuffle convert. Q5_K weights that pass use_adreno_kernels (dims >= 512) already dispatch to that GEMM at prefill; the GEMV path is only reached by small/odd-shaped projections. Verified the GEMM is correct at large N: greedy generation over a 720-token prompt on Qwen3.5-0.8B (Q5_K SSM in/out proj) is coherent and matches the CPU reference (modulo expected FP drift in the tail). The garbage batched perplexity is a separate head_dim>=256 intermediate-logit issue that persists with the CPU fallback too and does not affect generation. Add Q5_K to type_has_gemm so large-N Q5_K weights stay on the GPU GEMM instead of falling back to CPU; small/odd Q5_K projections still fall back (they would take the buggy GEMV). Removes the per-layer GPU->CPU->GPU round-trip the fallback forced on the SSM projections. Clean serial A/B on Qwen3.5-0.8B (fa=1, ngl=99, -r 3): pp1024 1275->1381 (+8.3%), pp4096 957->1070 (+11.8%), pp4096@d8192 453->490 (+8.2%). Decode and N<512 unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Add Q5_K GEMM and GEMV kernels to the Adreno backend to improve performance for Q5_K quantized models.
Additional information
With Qwen3.5-9B-Q5_K_M.gguf on 8 elite gen 5:
master,
this PR,
Requirements