Skip to content

Fix Gemma 3/3n KV cache to honor per-batch end_index - #765

Open
Ayush7614 wants to merge 1 commit into
google-deepmind:mainfrom
Ayush7614:feat/per-batch-kv-cache-gemma3
Open

Fix Gemma 3/3n KV cache to honor per-batch end_index#765
Ayush7614 wants to merge 1 commit into
google-deepmind:mainfrom
Ayush7614:feat/per-batch-kv-cache-gemma3

Conversation

@Ayush7614

@Ayush7614 Ayush7614 commented Jul 28, 2026

Copy link
Copy Markdown

Summary

  • Fixes a real correctness bug: Gemma 3 and Gemma 3n attention still wrote KV cache updates using only end_index[0], so every batch row was written at the same offset whenever fill lengths diverged.
  • Ports the Gemma 4 per-example scatter update (batch_indices + modulo indices) into gm/nn/_modules.py and gm/nn/gemma3n/_modules.py.
  • Also improves rolling-cache behavior for multi-token writes via per-position % cache_size (matching Gemma 4).
  • Adds regression tests that set divergent end_index values and assert each row updates its own slot (and that the old shared-slot corruption no longer happens).

This is independent of existing rolling-cache / OOM issues and does not claim any open issue.

Test plan

  • Verified Gemma 3 Attention with divergent end_index=[1,4]: row0 writes slot1, row1 writes slot4; row1 slot1 unchanged
  • Verified uniform end_index still updates both rows at the shared offset
  • Verified Gemma 3n Attention with the same divergent-index scenario
  • Reviewers: pytest gemma/gm/nn/_modules_test.py::test_attention_cache_uses_per_batch_end_index gemma/gm/nn/gemma3n/_modules_test.py::test_attention_cache_uses_per_batch_end_index

Gemma 4 already updates cache slots with per-example indices, but Gemma 3 and
Gemma 3n still used end_index[0], so divergent fill lengths in a batch silently
corrupted every row. Align both paths with the Gemma 4 scatter update and add
regression tests.
@google-cla

google-cla Bot commented Jul 28, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@Ayush7614
Ayush7614 force-pushed the feat/per-batch-kv-cache-gemma3 branch from d14faf4 to b811f90 Compare July 28, 2026 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant