Speculative decoding on Apple GPUs was slower than not using it. The fix landed this morning.

#llama-cpp #metal #speculative-decoding #kernels

PR #29869 was merged into llama.cpp this morning. The commit message sums up the problem in one sentence:

Speculative decoding verifies a few draft tokens per step. Without the tensor API, Metal ran these mat-muls with the mat-vec kernels, whose time grows with every src1 row, so DFlash2 decoding on an M3 Ultra was slower than serial decoding.

Here is what that cost. M3 Ultra, 60-core GPU, macOS 15.7.9, Qwen3.8-27B Q4_0 as target with a Qwen3.8-27B DFlash2 Q8_0 drafter, -ngl 99 -fa on -c 8192 -np 1 --jinja:

modepromptmasterthis PR
DFlash2code30.3 t/s113.4 t/s
DFlash2prose16.8 t/s64.4 t/s
serialcode32.1 t/s32.0 t/s

Read the third row against the second. Before today, turning on speculative decoding for prose on an M3 Ultra took you from 32 t/s to 16.8 t/s. The draft acceptance rate was fine. The method was fine. The verification mat-mul was being run by a kernel that charges you linearly per row.

Why the mat-vec kernel was the wrong kernel

A mat-vec kernel is built for one src1 row: load a weight block, dequantize it, multiply by the single activation vector, move on. Memory-bound, bandwidth-optimal, exactly right at batch 1.

Hand it eight rows and it does the dequantize eight times, because nothing in its structure holds a dequantized weight for reuse. So verifying a 5-token draft costs roughly five single-token decodes of compute, plus the draft model's own work, which is how throughput ends up below serial. Speculative decoding only pays if verifying N tokens costs meaningfully less than N sequential steps, and on Metal that precondition was simply false.

The fix adds mat-mul kernels for 2–16 src1 rows on 8×8 simdgroup matrices: each weight is dequantized once for all rows, and the simdgroups of a threadgroup split K. Q4_0, Q8_0 and Q5_K get dedicated kernels; F32, F16, Q4_1, Q5_0, Q5_1, Q4_K and Q6_K go through a generic path over the 16-weight dequantizers.

From test-backend-ops perf -o MUL_MAT, m=4096, k=14336, reported as PR time divided by master time (lower is better):

typen=2n=3n=4n=5n=8
F321.000.971.000.990.57
F160.990.970.710.590.34
Q4_00.580.580.440.360.22
Q4_10.930.650.500.410.25

Note the shape of that table. For Q4_0 the win is already 1.7× at two rows and 4.5× at eight. For F32 there is nothing until n=8. That difference is the whole reason there is a threshold table rather than a flag:

use them only on MTLGPUFamilyApple7+ without the tensor API, from the row count at which they beat the mat-vec kernels on an M3 Ultra (F32: 6, F16, Q4_K, Q5_0, Q5_1: 3, other types: 2)

Dequantization cost decides the crossover. With Q4_0 you were paying to unpack 4-bit blocks once per row, so amortizing that across two rows already wins. F32 has no unpacking to amortize, so the MMA path only pays once you are far enough from bandwidth-bound for ALU throughput to matter at all.

What I'm less sure about

The thresholds are one device's measurements. F32:6, F16:3, others:2, tuned on an M3 Ultra with 60 GPU cores. The crossover row count is a function of the ratio between dequantize cost and memory bandwidth per core, and an A18 in a phone has a very different one. Nobody has published iPhone numbers. If you ship speculative decoding on iOS, the honest position is that your crossover is unmeasured and the committed constants are a guess for your device.

It only runs when the tensor API is absent. MTLGPUFamilyApple7+ without the tensor API, which, given that llama.cpp disables the tensor API unless the device name contains M5, M6, A19 or A20, means every Apple GPU older than those. The Metal mat-mul path has now forked into two worlds split by a device-name check: new chips go through matmul2d, everything older goes through these MMA kernels. Two code paths, one of them tuned on one machine.

Worth noting as a review outcome: the original PR also fused up to 16 adjacent same-layout f32 copies into a single dispatch. ggerganov measured it on an M2 Ultra, found under 1% difference, and asked for it to be removed. It was, along with the memory-range changes. A fusion that does not clear noise is complexity you maintain forever, and the right call is the one that was made.

What to do about it

If you benchmarked speculative decoding on Apple silicon and shelved it, that measurement is out of date. Re-run it on master after today. And if you are on an M5, you were never in this regime: your numbers were already fine, and nothing in this PR touches you.

The general point is that small-N batches behave differently from N=1 and need their own kernels. If a kernel's cost grows linearly with N, anything that verifies N tokens at once (speculative decoding, MTP, draft trees) ends up slower, and the method gets blamed for what is really a kernel problem.