MLX's nvfp4 prefill went 1,267 → 4,256 tok/s in two PRs. Decode went 81.9 → 81.7.

#mlx #nvfp4 #quantization #apple-silicon #m5

Two PRs merged into MLX within eleven minutes of each other on September 14: #4483 ("Load global scales in qmm_t kernels", +215/−12, 9 files) and #4481 ("Use matrix kernels for global-scale gather_qqmm", +234/−1, 4 files). Both benchmark the same configuration: Qwen3.6-35B-A3B quantized with NVIDIA Model Optimizer, nvfp4_mlp_only, prompt 2048 / generate 128.

The numbers, from #4483's table:

main prompt t/s#4483#4483 + #4481
M5 Max1,267.51,365.94,256.2
M3 Ultra1,603.21,683.52,819.9

And from #4481's own table, which measures #4481 against main and adds the decode column:

main ppPR ppmain tgPR tg
M5 Max1,257.03,105.381.981.7
M3 Ultra1,627.32,673.268.568.6

Two tables, slightly different baselines (1,267.5 vs 1,257.0 on M5 Max; that's run-to-run noise of roughly 0.8%), which is itself a useful reminder that ±1% on these benchmarks is not signal.

The shape of it

The generation column is the point. Prefill got 3.4× on M5 Max and decode moved by −0.2%, which is zero.

This is the clearest compute-bound/bandwidth-bound split you'll see in a real PR pair. Prefill at 2048 tokens is a batch of 2048 through each matmul: arithmetic intensity is high, you're limited by how fast the ALUs consume the weights, and putting the quantized matmul on the matrix units instead of the general SIMD path is worth 3×. Decode is a batch of one: every weight is touched exactly once per token, arithmetic intensity is about 1, and you are limited by how fast you can stream the weights out of unified memory. Faster multipliers do not help you read memory faster. So decode doesn't move, and it shouldn't, and a PR that claimed it did would be the suspicious one.

Put another way, decode only gets faster when you read fewer bytes per token: harder quantization, shared KV, pruned experts, speculation. Kernel work mostly helps prefill, which is your time-to-first-token. Both matter, but they're separate budgets, and mixing them up is how a 3× speedup ships and nobody notices.

Why the split between two PRs matters

#4483 on its own (teaching qmm_t to load the global scale) is 1.08× on M5 Max. On its own you'd call that a modest win and move on. It isn't the win; it's the enabling change. #4481 wires the new gather_qmm kernels into gather_qqmm (the MoE path, where each token's experts are gathered before the matmul), and that's where 1.36k becomes 4.26k.

That ordering is worth noticing because it's the usual shape of quantization work and it keeps catching people out. A new format lands. The first implementation is a correct-but-scalar dequant-and-multiply path. Benchmarks say the format is barely faster than what you had, and half the internet concludes the format is overhyped. Then someone puts it on the matrix units and it's 3×. The number you measure is a property of the kernel, not the format.

Three weeks ago the story on MLX's nvfp4 was a correctness one: the global scale term was missing entirely, so the fp8 block scales had to cover a tensor's absolute magnitude as well as its internal spread, underflowed on small-weight tensors, and produced garbage on Granite. That got fixed. This week the same term got fast. Correctness first, then the tensor units. That's the right order, and it took about two weeks end to end.

Does any of this reach a phone?

Mostly no, and I'll keep saying it. nvfp4's payoff is on hardware with matrix units that consume it natively, and the MLX kernels here are the NAX-path quantized matmuls. The benchmark machines are an M5 Max and an M3 Ultra. On an iPhone you are shipping affine int4 at group size 64, or a GGUF k-quant, and none of this touches your build.

What does reach you is the reasoning. If you're deciding whether a new low-precision format is worth adopting, benchmark it when it's on the matrix path and not before. Also benchmark prefill and decode separately, because they will disagree, and one of them is the one your users feel.