docs(blog): final figures for the 4.8 post, and the MLX provider - #11362
Merged
Conversation
The cycle closed at 374 PRs over twenty-one days, not the 321 over eighteen the post was written against. Corrects the summary, the opening line, the contributor count and the gallery total, and moves the date to the day the release is cut. Adds the MLX GEMM provider (#11137), which merged after the post was written and is the one number an Apple Silicon reader wants: 1.54x to 2.19x on an M4 with time to first token roughly halving, both arms toggled on one binary. The +/-10% caveat travels with the table rather than being left in the PR. Two lines edited against the no-ai-slop skill while I was in the file, the same pass #11324 ran over the engines post: - The opener balanced two clauses across a colon and closed on "without lying to you", which is the built-to-be-quoted shape readers picked out of the HN thread. It is a flat statement now. - "This is a new modality rather than a new backend under an existing one" is a binary contrast that says nothing the next clause does not. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
localai-bot
added
the
kind/documentation
vllm.cpp is not a released backend and the post read like it was. The old wording buried the caveat in a block quote at the end of the section and still said "first release of a young engine". It now says plainly, before the caveat can be skipped, that these are alpha development builds, that shipping them in 4.8 is about letting people try the thing rather than recommending it, and that llama-cpp stays the default. Also completes the no-ai-slop pass I had only half run. Counting the lines built to be quoted, headings and section endings included, the post is in reasonable shape: long flat informational stretches, tables followed by a plain finding, headings that are labels rather than epigram-verdicts. Three patterns survived, each one an item in eval.md: - "and inverts that:" set the usual shape against ours across a colon. The sentence works without the frame. - "Two things were conflated there: a signal, which needs one line, and the detail, which needs somewhere to put it" is a role-assignment pair. Says what happens instead. - "The maturity statement from the release notes is worth repeating in full" is throat-clearing in front of a quote, and the quote is gone. Left the rest alone. Minimum effective edit, not a rewrite. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Collaborator
Author
|
Two changes since the PR was opened. vllm.cpp is now described as alpha. It is not a released backend and the post read like it was: the caveat sat in a block quote at the end of the section and still called it "a first release of a young engine". It now says plainly, before anyone can skip it, that these are alpha development builds, that shipping them in 4.8 is about letting people try the thing rather than recommending it, and that Finished the no-ai-slop pass, which I had only half run the first time: I spot-fixed two lines and ran greps, but skipped the workflow's actual step, counting the lines built to be quoted (headings, subtitle and every section's last sentence) and checking against Having done that count, the post is in reasonable shape: long flat informational stretches, tables followed by a plain finding rather than a verdict, headings that are labels rather than epigram-verdicts. Three patterns survived, each an item in
Left the rest alone deliberately. Minimum effective edit. |
…bers The post described vllm.cpp as "a from-scratch port of vLLM, written and maintained by the LocalAI team". Two things wrong with that. It is a community project, and it has stopped being only a port: it loads GGUF, runs on CPU, Metal and Vulkan, ships speculative decoding and KV offload, and its benchmark page measures against llama.cpp, MLX-LM and DwarfStar as well as vLLM, because those are the engines it competes with on that hardware. vLLM's role is now stated for what it is, the reference implementation. Correctness is checked against it and the scoreboard is kept against it. Also flags that the name will probably change, since it is drifting far enough that vllm.cpp will eventually mislead. Adds real numbers from the project's own docs/BENCHMARKS.md rather than adjectives: 1.045x vLLM at concurrency 1 on Qwen3.6-27B NVFP4 with token-for-token identical output, 1.010x and 1.013x at c16 and c32 on the 35B MoE and behind below that, prefill 1.18x over llama.cpp on CPU aarch64, 97.6% of MLX-LM warm total on an M4. Upstream's own caution travels with them: it treats c2 through c32 as ties because its noise band is 0.5% and those margins are 0.7% to 1.7%. Every figure was checked against ~/_git/vllm.cpp/docs/BENCHMARKS.md rather than restated from memory. The heading is marked alpha to match the section body. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
…sult Two corrections to the previous commit. "A community project" says nothing and was not quite true either. The LocalAI team maintains vllm.cpp. Community-first is the intent, not a description, so it now says that and says what backs it: its own repository, its own docs, benchmark record and issue tracker, and it runs without LocalAI anywhere in the picture. Adds the DeepSeek-V4-Flash result, which makes the divergence point better than any of the prose around it. That model does not run on vLLM on a single GB10: every vLLM-loadable checkpoint is 156 GB or more against a 119 GiB unified pool, and the only quant that fits is an extreme-low-bit GGUF that vLLM cannot load. vllm.cpp reads GGUF and runs it at 16.28 tok/s against ds4's 16.33, a parity result. Also notes MTP speculative decoding, token-identical to vLLM's and about 4% faster at concurrency 1. Both figures checked against ~/_git/vllm.cpp/docs/BENCHMARKS.md. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
…vLLM cannot The previous version opened on "that model does not run on vLLM on a single GB10 at all". Wrong emphasis twice over: it makes a strong negative claim about another project the headline, and it buries the actual result, which is that vllm.cpp runs DeepSeek-V4-Flash at roughly 2-bit (IQ2_XXS mixed, about 80 GB) on a single DGX Spark and decodes at 16.28 tok/s against DwarfStar's 16.33. The size constraint is still there, stated as the reason the quant is what it is rather than as a point about vLLM: at 300B+ total parameters even a 4-bit checkpoint is 156 GB or more, so a 2-bit GGUF is what fits the Spark's 119 GiB unified pool. The table row now names the quant and the box (IQ2_XXS, one DGX Spark) instead of just "GGUF, GB10", since that is the part a reader with a Spark wants. Figures unchanged and still from ~/_git/vllm.cpp/docs/BENCHMARKS.md. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
"The name will probably change at some point" invited the obvious question. It now says the rename is expected and the name is still to be decided, which is the actual state. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Figures
The cycle closed at 374 PRs over twenty-one days, not the 321 over eighteen the post was written against. Corrects the summary, the opening line, the contributor count (24 to 25) and the gallery total (1,505 to 1,515), and moves the date to the day the release is cut.
MLX GEMM provider
#11137 merged after the post was written, and it is the one number an Apple Silicon reader wants. Qwen3-1.7B-bf16 on an M4, both arms toggled on a single binary so a build difference cannot explain the gap:
Time to first token roughly halves. The +/-10% caveat (two reps, 9.4% spread) travels with the table rather than staying buried in the PR.
Two prose fixes
Ran the same no-ai-slop pass over the lines I was already touching that #11324 ran over the engines post:
Left the rest of the post alone. The point was the minimum effective edit, not a rewrite.
Verification
hugobuilds clean; the post renders with all three clips resolving (vllm-race,3d-generation,magpie).