Skip to content

ggml: fix cuda kernel launch configuration for k_compute_batched_ptrs to support large batch#16744

Merged
JohannesGaessler merged 4 commits into
ggml-org:masterfrom
leejet:cuda_mul_mat_fix
Oct 26, 2025
Merged

ggml: fix cuda kernel launch configuration for k_compute_batched_ptrs to support large batch#16744
JohannesGaessler merged 4 commits into
ggml-org:masterfrom
leejet:cuda_mul_mat_fix

Conversation

@leejet

@leejet leejet commented Oct 23, 2025

Copy link
Copy Markdown
Contributor

This PR fixes an invalid CUDA kernel launch issue for k_compute_batched_ptrs when ne12 or ne13 is large.

  • The previous launch used dim3 block(ne13, ne12), which can exceed the maximum threads per block (1024) and trigger cudaErrorInvalidConfiguration.
  • Updated the launch to use a fixed block size (e.g., 32×32 threads) and compute the required number of blocks in each dimension, ensuring the total threads per block do not exceed CUDA limits.
  • No changes were made to the kernel logic itself; only the grid/block configuration was adjusted.

@leejet leejet requested a review from slaren as a code owner October 23, 2025 15:00
@leejet leejet mentioned this pull request Oct 23, 2025
Merged
@github-actions github-actions Bot added Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning labels Oct 23, 2025
@jeffbolznv

jeffbolznv commented Oct 23, 2025

Copy link
Copy Markdown
Contributor

Same comment for this one. Please add a backend test that hits this case. The bug may exist in other backends, too.

@leejet

leejet commented Oct 23, 2025

Copy link
Copy Markdown
Contributor Author

The backend test has been added.

@github-actions github-actions Bot added the testing Everything test related label Oct 23, 2025
JohannesGaessler
JohannesGaessler reviewed Oct 23, 2025
View reviewed changes
Comment thread ggml/src/ggml-cuda/ggml-cuda.cu Outdated
slaren
slaren reviewed Oct 23, 2025
View reviewed changes
Comment thread tests/test-backend-ops.cpp Outdated

// test cases with large batch size
test_cases.emplace_back(new test_mul_mat(type_a, type_b, 16, 8, 256, {1024, 2}, {1, 1}));
test_cases.emplace_back(new test_mul_mat(type_a, type_b, 16, 8, 256, {4096, 1}, {1, 1}));

@slaren slaren Oct 23, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't a batch size of 1024 be enough to test this?

@leejet leejet Oct 24, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1024 happens to be the maximum number of threads, so it won’t trigger any issues. I’ve now changed the batch size to 1536, which is smaller than before.

leejet and others added 2 commits October 25, 2025 00:50
@leejet @JohannesGaessler
Update ggml/src/ggml-cuda/ggml-cuda.cu
96e2320 
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
@leejet
reduce the batch size
1da9854
@am17an

am17an commented Oct 26, 2025

Copy link
Copy Markdown
Contributor

@JohannesGaessler merge?

@JohannesGaessler JohannesGaessler merged commit bbac6a2 into ggml-org:master Oct 26, 2025
71 of 72 checks passed
Anico2 added a commit to Anico2/llama.cpp that referenced this pull request Jan 15, 2026
@Anico2 @JohannesGaessler
ggml: fix cuda kernel launch configuration for k_compute_batched_ptrs…
13eef59 
… to support large batch (ggml-org#16744)

* fix k_compute_batched_ptrs

* add backend ops test

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* reduce the batch size

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
blime4 referenced this pull request in blime4/llama.cpp Feb 5, 2026
@leejet @JohannesGaessler
ggml: fix cuda kernel launch configuration for k_compute_batched_ptrs…
a0414ad 
… to support large batch (#16744)

* fix k_compute_batched_ptrs

* add backend ops test

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* reduce the batch size

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
@leejet @JohannesGaessler
ggml: fix cuda kernel launch configuration for k_compute_batched_ptrs…
e0a23e0 
… to support large batch (ggml-org#16744)

* fix k_compute_batched_ptrs

* add backend ops test

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* reduce the batch size

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
ljubomirj pushed a commit to ljubomirj/llama.cpp that referenced this pull request May 6, 2026
@leejet @JohannesGaessler
ggml: fix cuda kernel launch configuration for k_compute_batched_ptrs…
982ddd8 
… to support large batch (ggml-org#16744)

* fix k_compute_batched_ptrs

* add backend ops test

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* reduce the batch size

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
my-other-github-account pushed a commit to my-other-github-account/llama.cpp that referenced this pull request May 15, 2026
@leejet @JohannesGaessler
ggml: fix cuda kernel launch configuration for k_compute_batched_ptrs…
9f43c86 
… to support large batch (ggml-org#16744)

* fix k_compute_batched_ptrs

* add backend ops test

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* reduce the batch size

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
my-other-github-account pushed a commit to my-other-github-account/llama.cpp that referenced this pull request May 15, 2026
@leejet @JohannesGaessler
ggml: fix cuda kernel launch configuration for k_compute_batched_ptrs…
24b77a6 
… to support large batch (ggml-org#16744)

* fix k_compute_batched_ptrs

* add backend ops test

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* reduce the batch size

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
phibya pushed a commit to ziee-ai/llama.cpp that referenced this pull request May 29, 2026
@leejet @JohannesGaessler
ggml: fix cuda kernel launch configuration for k_compute_batched_ptrs…
845b6c9 
… to support large batch (ggml-org#16744)

* fix k_compute_batched_ptrs

* add backend ops test

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* reduce the batch size

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
fewtarius pushed a commit to fewtarius/CachyLLama that referenced this pull request May 30, 2026
@leejet @JohannesGaessler
ggml: fix cuda kernel launch configuration for k_compute_batched_ptrs…
b6b0bcc 
… to support large batch (ggml-org#16744)

* fix k_compute_batched_ptrs

* add backend ops test

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* reduce the batch size

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
fukuro-kun pushed a commit to fukuro-kun/fukuro-llama-cpp-turboquant that referenced this pull request Jul 5, 2026
@leejet @JohannesGaessler
ggml: fix cuda kernel launch configuration for k_compute_batched_ptrs…
edb7443 
… to support large batch (ggml-org#16744)

* fix k_compute_batched_ptrs

* add backend ops test

* Update ggml/src/ggml-cuda/ggml-cuda.cu

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>

* reduce the batch size

---------

Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@slaren slaren slaren left review comments
@JohannesGaessler JohannesGaessler JohannesGaessler approved these changes

Assignees

No one assigned

Labels

ggml changes relating to the ggml tensor library for machine learning Nvidia GPU Issues specific to Nvidia GPUs testing Everything test related

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@leejet @jeffbolznv @am17an @slaren @JohannesGaessler