Upstream llama.cpp recently added native support for MTP-style speculative decoding, via --spec-type draft-mtp and --spec-draft-model (-md). This is used with models like Gemma 4's assistant/MTP drafter GGUFs and DeepSeek-style MTP modules — you load a small drafter alongside the target model and llama.cpp handles drafting and verification together in one batched decode call, which gives a solid speedup.
It'd be great to have something equivalent in llama-cpp-python. Right now the binding already has a speculative decoding setup with LlamaDraftModel and things like LlamaPromptLookupDecoding and LlamaNGramMapDecoding, but that's a different architecture from llama.cpp's native draft-mtp — the existing drafters generate their guesses in Python and hand them back for verification, while native draft-mtp keeps the whole loop inside C++ without any per-step round trip to Python.
Not sure whether it'd make more sense to bind the native path directly, or add a Python-side drafter that just loads the MTP GGUF as a second model and reuses the existing verification loop — the native route would likely perform better but probably means touching the vendored llama.cpp and the ctypes layer, while a Python-side version would be more self-contained. I'll leave the actual approach up to you since you know the codebase far better than I do — just wanted to flag that MTP support would be a nice addition and see if it's something you'd be interested in adding.
Upstream llama.cpp recently added native support for MTP-style speculative decoding, via
--spec-type draft-mtpand--spec-draft-model(-md). This is used with models like Gemma 4's assistant/MTP drafter GGUFs and DeepSeek-style MTP modules — you load a small drafter alongside the target model and llama.cpp handles drafting and verification together in one batched decode call, which gives a solid speedup.It'd be great to have something equivalent in llama-cpp-python. Right now the binding already has a speculative decoding setup with LlamaDraftModel and things like LlamaPromptLookupDecoding and LlamaNGramMapDecoding, but that's a different architecture from llama.cpp's native draft-mtp — the existing drafters generate their guesses in Python and hand them back for verification, while native draft-mtp keeps the whole loop inside C++ without any per-step round trip to Python.
Not sure whether it'd make more sense to bind the native path directly, or add a Python-side drafter that just loads the MTP GGUF as a second model and reuses the existing verification loop — the native route would likely perform better but probably means touching the vendored llama.cpp and the ctypes layer, while a Python-side version would be more self-contained. I'll leave the actual approach up to you since you know the codebase far better than I do — just wanted to flag that MTP support would be a nice addition and see if it's something you'd be interested in adding.