When benchmarking ds4-server through /v1/chat/completions in default thinking mode with max_tokens, length-stopped responses can trigger a synchronous post-generation thinking checkpoint canonicalization rebuild.
That rebuild is included in request wall time, so external benchmarks can report a large apparent generation TPS cliff even though the decode loop itself is materially faster.
Observed locally on Apple M5 Max:
- Raw
ds4-bench does not reproduce the huge cliff.
/v1/completions does not reproduce the huge cliff./v1/chat/completions with default thinking mode does reproduce it.
Server logs show decode completing, followed by:
thinking checkpoint canonicalization needs rebuild ...
thinking checkpoint canonicalized ... via=rebuild
Expected behavior:
A finish=length / incomplete thinking response should not synchronously rebuild the canonical thinking checkpoint before returning the response. It should either keep the live sampled checkpoint or defer rebuilding until a later request actually needs it.
A branch-side fix is to skip thinking checkpoint canonicalization when the response ends with finish=length or thinking is still open, while preserving canonicalization for complete stopped thinking turns.
This is the result on client side.
| Context |
Prompt TPS |
Gen TPS |
Gen Tokens |
Total Time |
| 0.5k |
230.0 |
23.6 |
128 |
7.3s |
| 1k |
296.5 |
20.4 |
128 |
9.0s |
| 2k |
366.3 |
12.4 |
128 |
15.7s |
| 4k |
308.2 |
7.4 |
128 |
30.0s |
When benchmarking
ds4-serverthrough/v1/chat/completionsin default thinking mode withmax_tokens, length-stopped responses can trigger a synchronous post-generation thinking checkpoint canonicalization rebuild.That rebuild is included in request wall time, so external benchmarks can report a large apparent generation TPS cliff even though the decode loop itself is materially faster.
Observed locally on Apple M5 Max:
ds4-benchdoes not reproduce the huge cliff./v1/completionsdoes not reproduce the huge cliff./v1/chat/completionswith default thinking mode does reproduce it.Server logs show decode completing, followed by:
thinking checkpoint canonicalization needs rebuild ...thinking checkpoint canonicalized ... via=rebuildExpected behavior:
A
finish=length/ incomplete thinking response should not synchronously rebuild the canonical thinking checkpoint before returning the response. It should either keep the live sampled checkpoint or defer rebuilding until a later request actually needs it.A branch-side fix is to skip thinking checkpoint canonicalization when the response ends with
finish=lengthor thinking is still open, while preserving canonicalization for complete stopped thinking turns.This is the result on client side.