1918 live tensors at the end of prefill: ExecuTorch's MLX backend never released a temp slot
PR #23109 landed overnight. In short, ExecutionState::tensors was never pruned, so a chain ended up holding every intermediate it had ever produced. Whether that actually costs you memory depends on how the model was exported, which surprised me more than the bug itself.
Here is the measurement, iPhone 16, gemma4-e2b int4, 2392-character prompt:
| live slots | live MB | peak phys_footprint | outcome | |
|---|---|---|---|---|
| stock | 1918 | 3102.9 MB | 3401 MB | killed by jetsam in prefill |
| this PR | 3 | 1.2 MB | 1379 MB | completes normally |
A 52-character prompt completes on both builds. That sentence is the whole reason this survived into shipped releases: the failure is invisible to any smoke test whose prompt is short, because the retained intermediates are sized by sequence length.
Where the memory goes
This is not a refcount bug. Every tensor in that table is exactly as live as the runtime believes it to be. The runtime just believes something the graph stopped meaning a long time ago.
ExecuTorch's AOT memory planner assigns each intermediate a slot. Whisper's plan reuses a handful of slots across the whole program, so nothing accumulates. The gemma4-e2b prefill plan assigns roughly one slot per instruction, and with 3243 instructions you finish holding 1918 of them. Same runtime, same bug, and one model shows nothing while the other dies.
The short version: an AOT memory plan says when each tensor is dead, and the runtime has to actually free it at that point. Until this PR, nothing did.
The fix walks each chain, works out the last use of every temp, and releases the slot there. The eligibility test is one comparison:
// ids run Constant -> Input -> Output -> MutableBuffer -> Temp
id.idx >= mutable_buffer_end
Anything at or past mutable_buffer_end is neither a method output nor a caller-visible buffer, so releasing it cannot be observed. Chains containing SCAN or IF opt out entirely, because for_each_tid does not descend into nested chains and a conservative pass that cannot see all the uses must not free anything.
What came up in review
A reviewer asked whether MLX's own memory controls would be a better lever, and the answer came back as a table. M3 Pro, Qwen3-0.6B int4 exported from the branch, 2047-token prefill, medians of four interleaved rounds of five runs:
| arm | ms | peak MB |
|---|---|---|
| none | 632 | 829 |
| eval threshold 512 MB | 640 | 233 |
| eval threshold 128 MB | 654 | 97 |
set_memory_limit +512 MB | 635 | 519 |
set_memory_limit +256 MB | 635 | 263 |
set_memory_limit +128 MB | 634 | 135 |
set_cache_limit(0) | 638 | 829 |
Outputs identical across every arm. Single-token decode is 6.3 ms everywhere except set_cache_limit(0), which costs 7.1.
Read the last row first. set_cache_limit(0) is the knob everyone reaches for when an MLX process is too fat, and it moved peak by zero, because MLX's cache holds memory that is already free, and this problem is memory that is still allocated. It made decode 13% slower for nothing.
mx.set_memory_limit bounds peak about as well as the eval threshold does and costs nothing measurable in latency: 829 MB down to 135 MB for 2 ms on a 2047-token prefill. Its two drawbacks are that it is process-global and takes an absolute byte count, which is awkward the moment you have two models loaded in one app.
What I'd do
The nuance the thread surfaces and the PR body doesn't: recent MLX exports already reuse temp slots aggressively, so a graph exported this month may not produce the 1918-slot plan at all. The author's target was ET 1.4.1. So this is primarily a fix for .pte files you exported and shipped months ago. If you ship on-device models, that's probably most of yours.
Three things worth doing this week:
- Log live slot count and
phys_footprintat the end of prefill on your longest realistic prompt, on the oldest device you support. Not your test prompt. - Re-export against current MLX and diff the slot count. If it drops, your fix is free.
- If it doesn't, set
mx.set_memory_limitrather than reaching forset_cache_limit(0).
Keep in mind jetsam kills don't come with a stack trace. What you'll see is the crash-free session rate slipping on 6 GB devices and support tickets saying the app closes on long documents.