llama.cpp self-speculation

Speculative decoding is the trick that lets a language model run faster than it "should": a small, cheap model guesses the next few tokens and the big model checks them all in one pass, accepting the good guesses for free. The catch has always been the small model — you had to find one, ship it, and configure it alongside the real one. As of a May merge, llama.cpp skips that step: it reads extra prediction heads that recent models like Qwen3.6 already carry inside their own weights file and uses them as the draft. The model speculates on itself.

One run accepted 270 of 270 drafted tokens — a perfect score — and still ran slower than not speculating at all.

On a single high-end consumer card, decode on a dense 27-billion-parameter model jumped from 38 to 65 tokens a second — about 1.7x, with the model's self-drafts accepted roughly three times in four. It is off by default and set with one flag; nothing new needs downloading beyond a model that ships the heads.

Where it stops working is instructive. On mixture-of-experts models — the kind that only wake a slice of themselves per token — the speedup thins out for a single user, and an independent benchmark of the older separate-draft approach found the deeper reason: one run accepted every single drafted token, a perfect 270 for 270, and still decoded slower than plain generation. Acceptance was never the bottleneck. Each guessed-ahead token drags in a different expert, so the verification pass has to load the union of all of them at once — and pays for the lot. Faster drafting, more experts to fetch, no net win.

So the honest headline is narrow and real: the dominant local-inference engine gained free self-speculation, worth about a doubling on dense models and much less — sometimes nothing — on the sparse ones, decided entirely by memory traffic rather than how good the guesses are.