spec: add spec metrics mean acceptance length and acceptance rate per position

[ruixiang63]

Overview

This PR adds mean_acceptance_length and acceptance_rate_per_position metrics for speculative decoding.
These metrics make it easier to understand how well a speculative decoding method is performing and help tune speculative decoding parameters such as --spec-draft-n-max and --spec-draft-p-min.

The implementation follows the same metric definitions used by vLLM's spec metrics:

• mean_acceptance_length = 1 + accepted_tokens / draft_verification_steps
• acceptance_rate_per_position[i] = accepted_tokens_at_position_i / draft_verification_steps

This PR reports the metrics in two places:

• Per-request server timing logs
• Aggregated speculative decoding statistics

The per-request metrics show speculative decoding behavior for an individual completion, while the aggregated metrics accumulate results across requests for each speculative decoding implementation, making them useful for long-running performance monitoring and parameter tuning.

Example output from EAGLE3 speculative decoding:

• Per-request server timeing logs

0.09.401.679 I slot print_timing: id  1 | task 90 | draft acceptance = 0.54671 (  158 accepted /   289 generated), mean acceptance length =  2.63, acceptance rate per position = (0.763, 0.515, 0.351)

• Aggregated speculative decoding statistics

0.09.401.699 I statistics     draft-eagle3: #calls(b,g,a) =    2    184    184, #gen drafts =    184, #acc drafts =   144, #gen tokens =    548, #acc tokens =   325, #mean acc len = 2.77, #acc rate/pos = (0.783, 0.571, 0.413), dur(b,g,a) = 0.005, 696.140, 0.191 ms

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, mainly for cross checking and code review.

[ruixiang63]

cc @ggerganov

[ggerganov]

This is the same as n_call_accept - no need for new member.

[ruixiang63]

fixed. Thanks for pointing out.

[ruixiang63]

Thanks for the review! @ggerganov I fixed all the issues, and checked the results again. It looks good to me.

0.06.895.980 I slot print_timing: id  1 | task 0 | prompt eval time =     300.11 ms /    20 tokens (   15.01 ms per token,    66.64 tokens per second)
0.06.895.998 I slot print_timing: id  1 | task 0 |        eval time =    2296.72 ms /   256 tokens (    8.97 ms per token,   111.46 tokens per second)
0.06.895.999 I slot print_timing: id  1 | task 0 |       total time =    2596.83 ms /   276 tokens
0.06.896.009 I slot print_timing: id  1 | task 0 |    graphs reused =         85
0.06.896.015 I slot print_timing: id  1 | task 0 | draft acceptance = 0.64479 (  167 accepted /   259 generated), mean acceptance length =  2.92, acceptance rate per position = (0.805, 0.632, 0.483)
0.06.896.057 I statistics     draft-eagle3: #calls(b,g,a) =    1     87     87, #gen drafts =     87, #acc drafts =    70, #gen tokens =    259, #acc tokens =   167, #mean acc len = 2.92, #acc rate/pos = (0.805, 0.632, 0.483), dur(b,g,a) = 0.005, 325.555, 0.114 ms
0.06.896.125 I slot      release: id  1 | task 0 | stop processing: n_tokens = 275, truncated = 0
0.06.896.140 I srv  update_slots: all slots are idle
0.06.920.935 I srv  params_from_: Chat format: peg-native
0.06.921.210 I slot get_availabl: id  1 | task -1 | selected slot by LCP similarity, sim_best = 0.200 (> 0.100 thold), f_keep = 0.011
0.06.921.219 I srv  get_availabl: updating prompt cache
0.06.921.330 W srv   prompt_save:  - saving prompt with length 275, total state size = 39.749 MiB (draft: 1.073 MiB)
0.06.946.260 I srv          load:  - looking for better prompt, base f_keep = 0.011, sim = 0.200
0.06.946.276 I srv        update:  - cache state: 1 prompts, 39.749 MiB (limits: 8192.000 MiB, 60928 tokens, 60928 est)
0.06.946.277 I srv        update:    - prompt 0x556b6de010b0:     275 tokens, checkpoints:  0,    39.749 MiB
0.06.946.279 I srv  get_availabl: prompt cache update took 25.06 ms
0.06.946.403 I slot launch_slot_: id  1 | task 90 | processing task, is_child = 0
0.06.946.411 I slot process_sing: id  0 | task -1 | saving idle slot to prompt cache
0.08.485.288 I slot print_timing: id  1 | task 90 | n_decoded =    152, tg = 100.70 t/s
0.09.419.325 I slot print_timing: id  1 | task 90 | prompt eval time =      29.44 ms /    12 tokens (    2.45 ms per token,   407.61 tokens per second)
0.09.419.343 I slot print_timing: id  1 | task 90 |        eval time =    2443.41 ms /   256 tokens (    9.54 ms per token,   104.77 tokens per second)
0.09.419.344 I slot print_timing: id  1 | task 90 |       total time =    2472.85 ms /   268 tokens
0.09.419.345 I slot print_timing: id  1 | task 90 |    graphs reused =        179
0.09.419.349 I slot print_timing: id  1 | task 90 | draft acceptance = 0.54671 (  158 accepted /   289 generated), mean acceptance length =  2.63, acceptance rate per position = (0.763, 0.515, 0.351)
0.09.419.374 I statistics     draft-eagle3: #calls(b,g,a) =    2    184    184, #gen drafts =    184, #acc drafts =   144, #gen tokens =    548, #acc tokens =   325, #mean acc len = 2.77, #acc rate/pos = (0.783, 0.571, 0.413), dur(b,g,a) = 0.006, 690.478, 0.234 ms
0.09.419.402 I slot      release: id  1 | task 90 | stop processing: n_tokens = 270, truncated = 0

[ggerganov]

Is it correct to have 1.0 + here?

For example, if I see in the logs mean acc len = 3.0, should I use --spec-draft-n-max 3 or --spec-draft-n-max 2. I think it is the latter, which is a bit confusing.

[ruixiang63]

Yes, the 1.0 + is intentional. It follows the conventional definition from vLLM: mean acceptance length = tokens committed per target forward pass, including the bonus token that's always produced even when no draft is accepted — so the range is [1, n_max+1], not [0, n_max]. see https://github.com/vllm-project/vllm/blob/b997071ec493765abbed990c65843ed05e4708a8/vllm/v1/spec_decode/metrics.py#L114
mean acc len = 3.0 corresponds to about 2 accepted draft tokens on average, and a total advancement of 3 tokens per target pass.

It is basically the speedup over normal decoding, which only produces 1 token per pass. The bonus token is included because the target always emits one even if every draft gets rejected, so AL = 1 means "no speedup".

That said, it's only a convention. If you think it reads better without the bonus token, I can report it without +1 instead.

[ruixiang63]

More specifically, the +1 represents the one additional token produced by the target model: a recovered token if some draft token is rejected, or a bonus token if all draft tokens are accepted. So this metric is closer to "tokens advanced per speculative verification step".
For choosing n_max, the per-position acceptance rates are more preferred.