llama.cpp's MoE expert cache: read the eligibility predicate before the benchmark table
New flag, merged October 7: --moe-cache-mib N (or LLAMA_ARG_MOE_CACHE_MIB). It keeps an LRU of MoE experts in a GPU buffer and uploads only the misses, for models whose experts don't fit in VRAM. PR #29887 is a port of the qvac-fabric cache: 14 files, +761/−18, with 550 lines of it in a new src/llama-moe-cache.cpp.
The table is striking. Qwen3.8-Flash-Next Q4_0, 93.7 GiB total with 65.4 GiB of experts, EPYC 7742 host, PCIe 4.0 x16:
| GPU | config | tg t/s | speed-up | hit rate |
|---|---|---|---|---|
| RTX 4090 | --fit (master) | 25.0 | 1.00× | - |
| RTX 4090 | --fit --moe-cache-mib 6544 | 39.4 | 1.57× | 72% |
| RTX 5090 | --fit (master) | 30.8 | 1.00× | - |
| RTX 5090 | -cmoe --moe-cache-mib 19000 | 67.8 | 2.20× | 89% |
The author suggests budgeting about 10% of total expert size. Those are real numbers and the mechanism is sound: expert selection is sparse and temporally correlated, so an LRU over a PCIe link is exactly the right trick.
Now the part that decides whether any of it applies to you.
The eligibility predicate
A layer is cacheable only if all of its expert tensors satisfy:
static bool llama_moe_cache_is_host_weight(const ggml_tensor * t) {
return t->buffer != nullptr &&
ggml_backend_buffer_get_usage(t->buffer) == GGML_BACKEND_BUFFER_USAGE_WEIGHTS &&
ggml_backend_buffer_is_host(t->buffer);
}
ggml_backend_buffer_is_host. On the Metal backend, from ggml/src/ggml-metal/ggml-metal.cpp, all three buffer types (shared, private, and mapped) implement it as:
static bool ggml_backend_metal_buffer_type_shared_is_host(ggml_backend_buffer_type_t buft) {
return false;
GGML_UNUSED(buft);
}
So nothing llama.cpp has loaded into a Metal buffer is ever a candidate, no matter how much unified memory you have or how little of it is free. If you pass --moe-cache-mib on a Mac with a normal load, you get this and nothing else:
no layer has all of its experts in host memory, MoE cache is disabled
That is not a bug. “Host” in ggml means CPU-backend memory, and a Metal buffer is a device buffer that happens to be CPU-addressable. But it means the flag's reachability on Apple Silicon is: put the experts in CPU buffers on purpose (-cmoe, --n-cpu-moe, or a buft override), then turn the cache on.
What the cache actually buys on unified memory
Do that, and the transaction changes completely. On a 4090 the cache converts a PCIe round trip into a local read: tens of gigabytes per second of bus traffic, replaced by VRAM bandwidth. On an M-series Mac there is no bus to amortize. The experts in the CPU buffer and the cache slots in the Metal buffer are in the same physical DRAM, at the same bandwidth. What you actually get is:
- expert matmuls for cached experts move from the CPU backend to the GPU,
- at the cost of a RAM-to-RAM copy on each miss,
- and a second resident copy of every cached expert, out of the same memory you were short of.
Whether that nets out positive is a question about your CPU-versus-GPU quantized matmul throughput on that machine, not about PCIe. It might be a solid win on an M4 Max with bandwidth headroom; it might be a wash on a base M4 where the copy and the extra residency cost more than the CPU path did. Nobody has published a Metal number, and the PR doesn't claim one.
Other limits
Batch size. static constexpr int64_t max_batch = 32; means ubatches over 32 tokens skip the cache entirely and take the regular offload path. The feature is a decode optimization by construction. The multi-GPU follow-up #30112 makes that explicit on 2×4090:
| config | decode t/s | vs master | prompt t/s | vs master | hit rate |
|---|---|---|---|---|---|
--fit (master) | 34.78 | 1.00× | 440.7 | 1.00× | - |
--fit --moe-cache-mib 6544 | 60.38 | 1.74× | 402.0 | 0.91× | 85.9% |
-cmoe --moe-cache-mib 15000 | 64.39 | 1.85× | 321.5 | 0.73× | 95.8% |
Buying 74–85% on decode and giving back 9–27% on prefill is a fine trade for chat and a bad one for bulk summarization. The PR headline is the decode column.
Topology. #29887 throws outright on more than one device or on pipeline parallelism; #30112 added multi-GPU a day later with a per-GPU budget. Tensor parallelism still throws.
Measuring it yourself
The implementation logs what you need, so use that rather than guessing. At load you get the cache size and the slot count as a percentage of total experts per group; at the end of the run:
llama_moe_cache: <name>: hits = N, misses = M, hit rate = X%, uploaded = Y MiB
Hit rate without a t/s comparison tells you nothing. A 90% hit rate on a machine where the miss was cheap is 90% of no benefit. Run your own prompt mix at your own -ub, with and without the flag, and look at decode and prefill. On a Mac, also watch resident memory, since the cache adds to it.