Added support of MTP speculative decoding on zai/glm-4.7-flash

[TriDefender]

Overview

Incorporated MTP support for glm4_moe_lite architechture, currently unsupported by the temporary solution of borrowing deepseek2 architechture

Additional information

Current code borrows functionality from deepseek2 arch, which has fundamental flaws that lacks MTP support. I could've added something to deepseek2 but i decided not to bring potential breaking changes.

I also added a few lines to deepseek2 that reveals the hidden state t_h_nextn for the MTP layer. This is because the graph function is imported from deepseek2 via using graph = llama_model_deepseek2::graph to reduce redundant code. Of course, if necessary I can just copy the entire section over.

src/models/glm4-moe-lite.cpp is a heavilly modified version of deepseek2.cpp so the MTP layer will be loaded or you'll crash by getting more tensors than expected

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure:
  YES, Artificial intelligence has been used to do pre-research on llm arches and information gathering

[ggml-gh-bot]

Hi @TriDefender, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[TriDefender]

I know zai/glm-4.7-flash is outdated and ppl stopped adding MTP support models for it but I would like to contribute on this matter

[TriDefender]

Hi @TriDefender, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

Edited PR to conform

[TriDefender]

Will perform some individual tests later on model conversion

[TriDefender]

This PR has been fully tested with actual model weights:

> python convert_hf_to_gguf.py "D:\llama.cpp\4.7" --outtype bf16 --outfile "D:\llama.cpp\4.7\glm-4.7-flash-bf16.gguf"
> .\llama-quantize.exe "D:\llama.cpp\4.7\glm-4.7-flash-bf16.gguf" "D:\llama.cpp\4.7\glm-4.7-flash-Q4_K_M.gguf" Q4_K_M
... ...
[ 860/ 868] blk.47.ffn_norm.weight               - [  2048,      1,      1,      1], type =    f32, size =    0.008 MiB
[ 861/ 868] blk.47.ffn_up_exps.weight            - [  2048,   1536,     64,      1], type =   bf16, converting to q4_K .. size =   384.00 MiB ->   108.00 MiB
[ 862/ 868] blk.47.ffn_up_shexp.weight           - [  2048,   1536,      1,      1], type =   bf16, converting to q4_K .. size =     6.00 MiB ->     1.69 MiB
[ 863/ 868] blk.47.nextn.eh_proj.weight          - [  4096,   2048,      1,      1], type =   bf16, converting to q4_K .. size =    16.00 MiB ->     4.50 MiB
[ 864/ 868] blk.47.nextn.embed_tokens.weight     - [  2048, 154880,      1,      1], type =   bf16, converting to q4_K .. size =   605.00 MiB ->   170.16 MiB
[ 865/ 868] blk.47.nextn.enorm.weight            - [  2048,      1,      1,      1], type =    f32, size =    0.008 MiB
[ 866/ 868] blk.47.nextn.hnorm.weight            - [  2048,      1,      1,      1], type =    f32, size =    0.008 MiB
[ 867/ 868] blk.47.nextn.shared_head_head.weight - [  2048, 154880,      1,      1], type =   bf16, converting to q4_K .. size =   605.00 MiB ->   170.16 MiB
[ 868/ 868] blk.47.nextn.shared_head_norm.weight - [  2048,      1,      1,      1], type =    f32, size =    0.008 MiB
llama_model_quantize_impl: model size  = 59562.53 MiB (16.00 BPW)
llama_model_quantize_impl: quant size  = 18020.27 MiB (4.84 BPW)
llama_model_quantize_impl: WARNING: 48 of 868 tensor(s) required fallback quantization

llama_quantize: quantize time = 260087.15 ms
llama_quantize:    total time = 260087.15 ms

And tested with llama-bench (no MTP)

>.\llama-bench.exe -m "D:\llama.cpp\4.7\glm-4.7-flash-Q4_K_M.gguf" -p 512 -n 128 -ngl 99
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 16379 MiB):
  Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9, VMM: yes, VRAM: 16379 MiB
| model                          |       size |     params | backend    | ngl |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| glm4-moe-lite 30B.A3B Q4_K - Medium |  17.60 GiB |    31.22 B | CUDA       |  99 |           pp512 |        192.38 ± 0.45 |
| glm4-moe-lite 30B.A3B Q4_K - Medium |  17.60 GiB |    31.22 B | CUDA       |  99 |           tg128 |         24.23 ± 0.34 |

build: d2462f8f7 (9590)

And llama-server inference test:

> cd .\llama-server.exe -m "D:\llama.cpp\4.7\glm-4.7-flash-Q4_K_M.gguf" --spec-type draft-mtp -md "D:\llama.cpp\4.7\glm-4.7-flash-Q4_K_M.gguf" --spec-draft-n-max 3 --fit on -fitt 128
0.00.026.346 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.026.350 I device_info:
0.00.079.036 I   - CUDA0   : NVIDIA GeForce RTX 4060 Ti (16379 MiB, 15233 MiB free)
0.00.079.044 I   - CPU     : AMD Ryzen 9 9900X 12-Core Processor             (47770 MiB, 33294 MiB free)
0.00.079.112 I system_info: n_threads = 12 (n_threads_batch = 12) / 24 | CUDA : ARCHS = 890 | USE_GRAPHS = 1 | FA_ALL_QUANTS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
... ...
0.59.284.140 I slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
0.59.284.141 I srv  get_availabl: updating prompt cache
0.59.284.148 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.59.284.154 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 4096 tokens, 8589934592 est)
0.59.284.155 I srv  get_availabl: prompt cache update took 0.01 ms
0.59.284.188 I slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
0.59.284.188 I slot process_sing: id  0 | task -1 | saving idle slot to prompt cache
0.59.284.189 I slot prompt_clear: id  0 | task -1 | clearing prompt with 0 tokens
0.59.284.392 I slot process_sing: id  1 | task -1 | saving idle slot to prompt cache
0.59.284.393 I slot prompt_clear: id  1 | task -1 | clearing prompt with 0 tokens
0.59.284.475 I slot process_sing: id  2 | task -1 | saving idle slot to prompt cache
0.59.284.475 I slot prompt_clear: id  2 | task -1 | clearing prompt with 0 tokens
1.03.242.481 W srv          stop: cancel task, id_task = 0
1.03.491.421 I slot      release: id  3 | task 0 | stop processing: n_tokens = 34, truncated = 0
1.03.491.442 I srv  update_slots: all slots are idle

Therefore, I can conclude that this PR is review ready

[am17an]

duplicate #24868?

[TriDefender]

duplicate #24868?

I did considered the options of adding MTP to deepseek2 but I didn't want to introduce potential breaking changes. I think it's better to respect arch differences and do not affect current 4.7-flash models created under deepseek2 arch that is without MTP support.