In bf16, a decay of 0.9995 is 1.0

#mlx #metal #numerics #linear-attention #on-device

MLX merged PR #4760 this morning. Five files, +19/−17, and four of the five files do nothing but swap a buffer declaration from InT to float. The whole fix in mlx/fast.cpp is two lines:

- auto g = astype(gates, out_dtype, s);
- auto beta = astype(beta_, out_dtype, s);
+ // The decay sits close to 1, where a 16-bit type has too few mantissa
+ // bits, so gates and beta stay float32 regardless of the type of q.
+ auto g = astype(gates, float32, s);
+ auto beta = astype(beta_, float32, s);

out_dtype is promoted from queries and keys. So if you ran a bf16 model, which is the usual case, MLX took your float32 decay and rounded it down to bf16 before the kernel ever saw it.

Why that specific cast is fatal

The docstring is explicit about what gamma is: “Decay rate in linear space”. Not a log. A number just under 1.

bf16 has 8 head mantissa bits. In [0.5, 1) the spacing is 2−8 = 0.00390625, so the largest bf16 strictly below 1.0 is 0.99609375. Everything above that rounds to exactly 1.0:

true gbf16fp16
0.990.988281250.990234375
0.9950.996093750.9951171875
0.99951.00.99951171875
0.99991.01.0

A decay of exactly 1.0 is not “slightly wrong”. It is a different model. The recurrent kernel at mlx/backend/metal/kernels/gated_delta_update.h does the multiply directly:

state[i] = state[i] * g_[hv_idx];

and the chunked kernel takes the log first, prefix-sums it across the chunk, then exponentiates:

float g_val = (thread_index_in_simdgroup < (uint)valid_rows)
    ? metal::fast::log(metal::max(
          static_cast<float>(g_[thread_index_in_simdgroup * Hv + hv_idx]), 1e-6f))
    : 0.0f;
float gamma_val = simd_prefix_inclusive_sum(g_val);

log(1.0f) is 0. Every gamma_* downstream becomes exp(0) == 1. The head doesn't decay a little less; for that chunk it is a plain delta rule with infinite memory.

The bug report

Issue #4725, filed October 19 by jvmenen, is the kind of bug report you wish you got. T = 4096, 16 key heads, 48 value heads, head dim 128, NAX route, M5 Pro, errors against a float64 reference:

inputsoutput errstate errms
bf16 q/k/v (then-current)4.8e−25.0e−24.91
all float321.8e−31.3e−36.45
bf16 q/k/v, fp32 g/beta2.4e−31.3e−35.20

Two numbers from that issue are the ones worth carrying around. First: in his test distribution, 27% of bf16 decay values landed on exactly 1.0. Second: a token 1,000 steps back should retain about 61% of its weight at g = 0.9995; after rounding it retains 100%.

He also isolated it properly: a float64 reference with only g rounded to bf16 reproduces the full bf16 error. That is how you prove which cast is the culprit instead of guessing.

The cost of the fix is 0.29 ms on 4.91, about 6%, for a 20× error reduction. In the issue thread zcbenz asked what the CUDA gated-delta implementations do; the answer in the thread was that they all pass the decay in float32. MLX was the outlier.

Caveats

The merged commit has no test. One of the branch commits is titled “no need of complicated test for trivial change”, and the final diff touches four kernel headers and fast.cpp, and nothing under python/tests/. The issue reports a test that fails on v0.32.4 with state error 2.3e−2. A two-line dtype change is exactly the kind of thing a future refactor silently reverts, and the regression costs you nothing to keep. Nobody caught this in review either.

Second: the issue was labeled low priority. For a sampled chat response, 5e−2 state error will mostly look like a different seed. The place it bites is long-context recall, which is the main reason hybrid linear attention exists, and it bites in a way that no loss curve or perplexity sweep on short sequences will surface. If you benchmarked a Qwen3.5-class gated delta net on MLX in the last six weeks (the kernels landed September 15) and concluded its long-range recall was mediocre, you were measuring 8 mantissa bits.

Not in v0.32.4. You need main.

My takeaway: a framework shouldn't cast a parameter to the activation dtype just because it's passed in next to the activations. A decay value that lives in a narrow band just below 1.0 has nothing to do with how Q and K are stored.