llama.cpp's BF16 attention kernel: the prefill number is the headline, the accumulator is the story
PR #30491 was merged this morning and made it into v0.7.0 the same afternoon. It adds BF16 instantiations of the Metal 4 tensor-API flash-attention kernel that landed two weeks ago, so -ctk bf16 -ctv bf16 gets the same path -ctk f16 already had. The diff is small: a new kernels/fa_tensor.metal (298 lines, moved out of fa_f16.metal at ggerganov's request), a type suffix in the pipeline name, and one line in the dispatch gate.
const char * type = op->src[1]->type == GGML_TYPE_BF16 ? "bf16" : "f16";
snprintf(base, 256, "kernel_flash_attn_ext_tensor_%s_dk%d_dv%d", type, dk, dv);
Twelve instantiations now exist: f16 and bf16 across dk/dv 64/64, 128/128, 192/128, 256/256, 512/512, 576/512. Full coverage, which is worth noting because the comparable MLX kernel work last week shipped three shapes out of twenty-four.
The performance numbers are prefill-only, and the gate says why
From the PR body, M5 Pro with a 20-core GPU, Qwen3.8-27B Q8_0, -fa 1:
| test | master, BF16 KV | PR, BF16 KV | PR, F16 KV |
|---|---|---|---|
| pp512 @ d8192 | 323.8 | 355.3 | 354.2 |
| pp512 @ d32768 | 184.2 | 265.6 | 268.5 |
| tg32 @ d8192 | 9.09 | 9.25 | 9.32 |
+44% on prefill at 32K. Decode moves 9.09 → 9.25, which is noise, and that is structural rather than unlucky. ggml_metal_op_flash_attn_ext_use_tensor() bails on five separate conditions, and two of them exclude decode outright: it returns false if the vec path applies, and it returns false if
((ne01 + nqptg - 1)/nqptg)*ne02*ne03*dk < 8192
with nqptg 32 (16 for dk ≥ 512). One query row times a few dozen heads does not clear 8192. There is also a hard requirement that KV length is a multiple of 64 (OP_FLASH_ATTN_EXT_TENSOR_NCPSG), and the device gate still matches on MTLDevice.name containing M5, M6, A19 or A20 unless you set GGML_METAL_TENSOR_ENABLE=1.
So: this is a prefill kernel for M5-and-later, and your decode throughput is unchanged. Fine. That is not the interesting part.
The accuracy table
Same PR body, KLD against -fa off -ctk f32 -ctv f32:
| configuration | mean @ 8K | mean @ 32K | same top p @ 32K |
|---|---|---|---|
| F16 KV | 1e-6 | 1e-6 | 99.957% |
| BF16 KV, master | 1.7e-5 | 4.05e-4 | 99.2% |
| BF16 KV, this PR | 3e-6 | 5e-6 | 99.921% |
BF16 KV, PR with bfloat P·V destination | 2.1e-5 | 6.18e-4 | 99.036% |
Read rows two and four together. BF16 KV on master diverges 400× more than F16 at 32K, and the author's own ablation (keeping the probability-times-V destination in bfloat instead of float) reproduces that divergence inside the new kernel. The new kernel is accurate because it accumulates in fp32:
auto co = mm_pv.template get_destination_cooperative_tensor<decltype(tp), decltype(tv0), float>();
Now open fa_f16.metal and look at the type macros for the simdgroup path. The last triple in each is the o_t, o4_t, o8x8_t group, documented in the template as “attention accumulation types”:
#define FA_TYPES \
...
float, float4, simdgroup_float8x8
#define FA_TYPES_BF \
...
half, half4, simdgroup_half8x8
//float, float4, simdgroup_float8x8
The F16 kernel accumulates attention output in fp32. The BF16 kernel accumulates it in half, with the fp32 variant sitting right there, commented out. That is the 4.05e-4.
For anyone writing kernels: storing K/V in 16 bits is a bandwidth choice, but accumulating in 16 bits affects correctness, and here both were decided by the same macro.
What this means if you ship
The tensor path papers over it on M5/M6/A19/A20, for prefill, at the six supported head-size pairs, when KV is a multiple of 64 and the workload clears the 8192 heuristic. Outside that set (an A18, an M3, a short prompt, dk=96), BF16 K/V still runs the simdgroup kernel with a 10-bit mantissa accumulating a sum over the whole KV length.
Three concrete things:
- If you were using
-ctk bf16 -ctv bf16on pre-M5 hardware to save conversion work, switch tof16. Same 2 bytes per element, fp32 accumulation, 1e-6 KLD. - If you are on M5 or later, check for
has tensor = truein the init log before attributing a quality change to this PR. The backend compiles a dummymatmul2dkernel at init and silently disables the path if the pipeline fails, once forhalfand again forbfloat. - If you maintain your own Metal attention kernel, this is the bug to grep for.
Uncommenting the fp32 variant is a two-character change, if anyone wants to send the PR.