llama.cpp's Metal 4 fast path now ships as two metallibs, and on iPhone the second one is the whole point
Last week I wrote about a capability probe that fails soft: llama.cpp's Metal backend checks at init whether the Metal 4 tensor API is available, and when the check fails on M5/A19 hardware you get the old shader-ALU prefill path and no error. The root cause and the fix both landed in llama.cpp v0.4.0 on Friday, and the fix has a specific shape on iOS that anyone bundling llama.cpp in an app should understand.
The root cause was the shader compiler's default language version
The diagnosis is in PR #27461, opened August 20 by a contributor benchmarking on an M5 Max who noticed prefill was "curiously slow." MTLCompileOptions defaults to an older Metal language version, one that doesn't expose the tensor headers (metal_tensor, MetalPerformancePrimitives) to the shader compiler. Both tensor-API probes in ggml_metal_device_init() therefore fail to compile, and only with verbose logging turned on do you see why:
program_source:6:17: error: use of undeclared identifier 'mpp'
using namespace mpp::tensor_ops;So has_tensor ended up false on every device, and prefill ran matmuls on general-purpose shader ALUs even on silicon with dedicated matmul units. The PR does two things: request Metal 4.0 explicitly when the device reports tensor support, so unsupported hardware is untouched, and clear has_tensor when the library came from a pre-compiled metallib. That second guard matters more than it looks.
Why the second guard forces a two-library design
If the Metal library was precompiled, the runtime can't know what language version it was built with, so the safe move is to assume it can't do tensor ops. But on iOS you almost always ship precompiled: compiling MSL from source at app launch is slow, and the xcframework path exists precisely so you don't have to.
That would have made the guard a permanent "no tensor API on iPhone" rule. PR #28163, merged September 1, is the resolution. The xcframework build now supports GGML_METAL_EMBED_LIBRARY=OFF and produces two libraries:
| Library | Contents | Built when | Loaded when |
|---|---|---|---|
default.metallib | Every kernel that has always been there | Always | Always |
ggml-tensor.metallib | Metal 4 tensor-API kernels | SDK 26+, with correct SDK name and -mtargetos | Device passes the tensor probe |
The PR author notes the missing -mtargetos flag was the non-obvious part. At runtime the backend loads the default library everywhere and the tensor library only where the device passes the probe. The PR was tested with llama.swiftui on an iPhone Air (A19 Pro): both metallibs load and the tensor path shows up in the bench. On an older device without tensor support, only default.metallib loads. No numbers are published in the PR, and I don't have an A19 Pro to produce my own, so I won't invent a multiplier here. The shape should match the desktop result from the LM Studio thread: prefill moves, decode doesn't.
What this means if you ship llama.cpp in an iOS app
Your existing xcframework has no tensor path. If you built it before September 1, or you build with an SDK older than 26, there is no ggml-tensor.metallib in the bundle and every A19 Pro user runs the slow prefill path. Nothing will tell you.
Check the bundle, not the log. The init log line (has tensor = true) is still the source of truth at runtime, but the cheaper check is at build time:
$ find build-apple/llama.xcframework -name '*.metallib'
.../ios-arm64/llama.framework/default.metallib
.../ios-arm64/llama.framework/ggml-tensor.metallibIf the second file isn't there, your SDK gate failed.
Deployment target and SDK are now separate decisions. You can keep a low deployment target for the default library and still get the Metal 4 library compiled against SDK 26. That's what the -mtargetos handling in the PR is for. Before this change, the two were coupled and the coupling silently picked the old path.
Benchmark prompt processing on a real A19 Pro. Same advice as last week, now with a concrete reason it applies to phones: run llama-bench -p 512 -n 64 in-app on an iPhone 17 Pro or iPhone Air and compare with GGML_METAL_TENSOR_DISABLE=1. If the pp number doesn't change, you're not on the fast path.
The design lesson
The rule this implies: when a runtime probe can't distinguish "hardware can't" from "binary was built wrong," don't build a smarter probe. Split the artifact so the build system encodes the capability. If the SDK can produce tensor kernels, a second library exists; if not, it doesn't, and the runtime's job shrinks to "is the file there and does the device accept it."
That's the same shape as the MLX side's device-targeted runtime packs, arrived at from the other direction.
Apple's event is Wednesday, and if the A20 Pro ships with second-generation Neural Accelerators the tensor path is going to be the default assumption for every new iPhone. Worth spending the afternoon making sure your bundle has two metallibs in it.