Eval bug: Second interaction hangs / generates gibberish after commit 0988acc ([SYCL] Add Q8_0 reorder optimization)

[tunmaker]

Name and Version

commit 0988acc

version: 8685 (0988acc)
built with IntelLLVM 2025.3.2 for Linux x86_64

• build command
  cmake -B build -DGGML_SYCL=ON -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx DCMAKE_BUILD_TYPE=Release

Operating systems

Linux

GGML backends

SYCL

Hardware

Intel Arc A770
Intel Arc A380

Models

unsloth Qwen3.5 9b q8
unsloth Qwen3.5 2b q8

Problem description & steps to reproduce

Title: Second interaction hangs / generates gibberish after commit 0988acc ([SYCL] Add Q8_0 reorder optimization)

Description
After commit 0988acc ([SYCL] Add Q8_0 reorder optimization), I see a performance improvement for the first completion (≈17 t/s → ≈25 t/s) but the second interaction hangs in an endless loop or produces gibberish output.

Reproduction steps

1. Build from tip including commit 0988acc.
2. Run either:
   • cli mode, or
   • server mode
3. Use model: qwen3.5 9b q8_0 on Intel Arc A770 (16 GB) and qwen3.5 2b q8_0 on Intel Arc A380 (6 GB).
4. Request a completion once — first completion returns correctly and with improved throughput.
5. Make a second interaction (another prompt / follow-up completion). The second interaction either:
   • enters an endless loop (nothing generated), or
   • generates gibberish output.

What I tested / observed

• i test older commits with rebuilding all, commit (ggml web gpu) works as intended
• First completion works correctly and is faster (17 t/s → 25 t/s).
• Second interaction consistently fails (endless loop or gibberish).
• Reproduced in both cli and server modes.
• Tested with "thinking on" and "thinking off".
• Collected logs for all combinations; I cleaned logs to reduce the endless-loop spam and marked the point where the endless loop occurs.
• Behavior observed on two GPU configurations:
  • Intel Arc A770 (16 GB), model qwen3.5 9b q8_0
  • Intel Arc A380 (6 GB), model qwen3.5 2b q8_0

Attached logs

• Cleaned logs for:
  • cli thinking on
  • cli thinking off
  • server thinking on
  • server thinking off
• In each log I marked where the endless loop begins.

Expected behavior

• Second and subsequent interactions should behave the same as the first: produce coherent output and not hang.

Actual behavior

• Second interaction either loops forever (no output produced) or emits gibberish tokens.

Notes / hypotheses

• The regression started with commit 0988acc (SYCL Q8_0 reorder optimization).
• The optimization likely changes the memory layout or reorders quantized data for Q8_0. Possible issues:
  • buffer/stride mismatch on subsequent inference calls,
  • state (KV cache or related buffers) not being reset or handled correctly after the first run,
  • thread-safety or synchronization problem in SYCL kernel(s) during repeated runs,
  • an indexing or pointer error when reusing reordered buffers for a second inference,
  • an uninitialized memory path exposed only on the second run.
• Problem reproduces on two different Arc GPUs and both 2b/9b Q8_0 models, suggesting the problem is implementation-level rather than model-specific.

What I’ve tried

• Running cli and server variants.
• Toggling thinking on/off.
• Captured and cleaned logs; marked the loop point.
• Compared first vs second run behavior to confirm regression point is after commit 0988acc.

Request / help wanted

• Guidance on how to debug SYCL Q8_0 reorder interactions and where to look for root cause.
• If maintainers can point to likely files/functions affected by the reorder (quantization/reorder kernels, buffer init, KV cache handling, inference loop, or reuse of device buffers), that would help.
• If you can review the attached cleaned logs and point out where the kernel, synchronization, or buffer misuse is visible, I’m happy to try fixes and open a PR.
• Any hints about typical pitfalls in SYCL device buffer reuse, kernel ordering, or memory lifetime that cause first-run OK but second-run failure.

Relevant info

• Commit: 0988acc
• Hardware:
  • Intel Arc A770, 16 GB
  • Intel Arc A380, 6 GB
• Models:
  • qwen3.5 9b q8_0
  • qwen3.5 2b q8_0
• Logs: (attached cleaned logs covering cli/server × thinking on/off; loop points marked)

Suggested immediate debugging steps I can run if you point me to the right area

• Add validation checks after reorder kernels to confirm data layout/strides match expectations.
• Zero or reinitialize KV cache and device buffers between runs to check if reuse is the issue.
• Run with SYCL debug/env options to enable sync or stricter validation, if available.
• Insert checksums/hash of reordered buffers after the first and before the second run to detect corruption.
• Add debug prints in the inference loop to locate where output generation stalls.

If helpful, I can attach the cleaned logs here and run any suggested debug builds or tests. I'm new to contributing; any pointers about which files to inspect first would be appreciated.

First Bad Commit

0988acc

Relevant log output

log_server_0988acc_thinkingON_cleaned.txt
log_server_0988acc_thinkingOFF_cleaned.txt
log_cli_0988acc_thinkingON_cleaned.txt
log_cli_0988acc_thinkingOFF_cleaned.txt

Logs

[NeoZhangJianyu]

@tunmaker
Could you try with PR: #21638?

Thank you!

[tunmaker]

@tunmaker Could you try with PR: #21638?

Thank you!

just tested with qwen3.5 2B q8_0 on intel Arc A380
works great 💯 with thinking on or off 👍
17 t/s not 25 t/s but its fixed no more gibberish

log_server_pr-21638_thinkingON.txt

[articuno1au]

I'm using the intel-server docker container with

1. Qwen3.5-9B-Q4_K.gguf
2. Qwen3.5-9B-Q6_K.gguf
3. Qwen3-8B-UD-Q6_K_XL.gguf
4. gemma-4-E4B-it-Q8_0.gguf

And I'm seeing the issue discussed here. Given the first 3 are Q4-6, I just wanted to flag there might be a broader issue here.

I also noticed that if I include web search as part of the context, it immediately starts generating junk (but I'm guessing that's because it's functionally a second request to the backend).

If you have a container/tag you want me to test against, I can do that.

[tunmaker]

there is this PR in progress not merged yet i hope they merge soon #21638

i checked out this pull request for my running server (not in a container)

but for a quick test you can

git fetch origin pull/21638/head:pr-21638
git checkout pr-21638

then rebuild llama.cpp

tested it fixes the issue on the master branch

[articuno1au]

Yeah, I understand. Just given the description and the changed files are all around Q8, just wanted to flag that the current state is broken for Q4 and Q6 models.

Unfortunately I don't have an environment I can build in at the moment. Thanks for the pointer though, appreciate it.

[cclecle]

confirming same problem here with oneAPI 2025.2.2 / Intel A380 , using Q4 models. Haven't tried the PR.

[arthw]

The PR can fix same issue of wen3.5-4B-Q4_0.gguf.

[tunmaker]

#21638

[cclecle]

The PR can fix same issue of wen3.5-4B-Q4_0.gguf.

it works !