Local Codex on a 16 GB M5: gpt-oss-20b decoded at 0.3 tokens per second

#codex #openai #gpt-oss #apple-silicon #on-device

Codex has a local mode: codex --oss sends requests to a model running on your own machine instead of to OpenAI. The CLI reference supports two backends for it, Ollama and LM Studio, picked with --local-provider or with oss_provider in ~/.codex/config.toml. I set it up on my 16 GB M5 MacBook with gpt-oss-20b, since OpenAI and Ollama both say it runs in 16 GB of memory. It does run. Decode came out at 0.2 to 0.4 tokens per second, with 2% of RAM free and about 13 GB in swap.

You can predict that result from two values you can look up before downloading anything. This post goes through both, then the config I ended up with.

What I measured

Ollama 0.40.13 from Homebrew on macOS 26.6.2, server started with OLLAMA_CONTEXT_LENGTH=32768, GPU wired limit left at the default. Four generations across two server sessions, two cold and two warm, 64 output tokens each.

measurementresult
ollama ps PROCESSOR23%/77% CPU/GPU
cold load32–41 s
prefill, 76–78 token prompt2.5–3.1 tok/s
decode0.2–0.4 tok/s
memory while loaded2–4% free, 11.9–13.1 GiB of swap in use

Roughly a quarter of the model ended up on the CPU side, and there wasn't enough physical memory left to hold it either, so the machine was paging for the whole run. Warm runs weren't reliably faster than cold ones. At these speeds Codex would spend many minutes just reading its own instructions and tool definitions before it touched any code.

Why it doesn't fit: Metal's working set

On Apple Silicon the GPU doesn't get all of unified memory. Metal reports a per-device limit, recommendedMaxWorkingSetSize, and ggml (the library under llama.cpp and Ollama's runner) uses it as the GPU's total memory. You can read yours with a few lines of Swift. Save this as ws.swift and run swift ws.swift:

import Metal
let d = MTLCreateSystemDefaultDevice()!
print(d.recommendedMaxWorkingSetSize)
// 12713115648  ->  11.84 GiB

On my machine it's 11.84 GiB, and Ollama's startup log agrees: total="11.8 GiB".

The gpt-oss-20b weights are 12.82 GiB (original/model.safetensors in OpenAI's repo is 13,761,300,984 bytes; Ollama lists the model at 14 GB). Only the MoE experts are MXFP4. Attention, embeddings, the router and the LM head are kept at higher precision, so there's no easy extra quantization step left. That puts the weights about 1 GiB over budget before any KV cache. ggml's source has a comment noting you can allocate past the recommended size, and you can, but on a 16 GB machine the rest of the system pays for it.

The default context is 4,096 tokens

The other interesting line in Ollama's startup log was default_num_ctx=4096. Per Ollama's context-length docs, the default depends on VRAM: 4k under 24 GiB, 32k from 24 to 48 GiB, 256k above that. Since the GPU only gets a fraction of RAM, a 24 GB Mac lands in the 4k tier as well. The same page recommends at least 64000 tokens for agents and coding tools. That makes sense once you count what an agent keeps in context: system prompt, tool schemas, AGENTS.md, plus every file it opens and every command output it reads back. One medium-sized source file can use up 4k by itself.

This can't be fixed from the Codex side. Codex talks to providers over the Responses API (wire_api = "responses" is the only value the config reference allows), and Ollama's OpenAI-compatibility page says the OpenAI API has no way to set context size. It has to be configured on the Ollama server.

How much the context costs

This part is good news. gpt-oss-20b alternates sliding-window layers (128-token window) with full-attention layers, 12 of each, with 8 KV heads of dimension 64. At f16 that works out to 2 KiB per token per layer. Only the full-attention layers grow with context, so it's 24 KiB per token overall. The sliding layers stay at 18 MiB regardless (Ollama's runner gives them 768 cells). For comparison, MiniCPM5-2B costs 43 KB per token.

contextfull-attention KV (12 layers, f16)weights + KVover the 11.84 GiB budget
4,096 (Ollama default, <24 GiB)0.09 GiB12.91 GiB+1.07 GiB
32,7680.75 GiB13.57 GiB+1.73 GiB
64,000 (Ollama's floor for coding tools)1.46 GiB14.28 GiB+2.44 GiB
131,072 (model maximum)3.00 GiB15.82 GiB+3.98 GiB

The runner log for my 32k run showed 768.00 MiB of KV for the 12 full-attention layers, matching the table. Going from 4k to 64k costs less than 1.5 GiB, so raising the context is affordable. Most of the problem is the weights.

Config

The context window has to be set on the Ollama server, repeated in the Codex config, and small enough to fit your GPU budget. On a 16 GB machine I'd use 32768. With more memory, go to 64000 or higher.

OLLAMA_CONTEXT_LENGTH=32768 ollama serve
ollama ps    # CONTEXT is what actually got applied; PROCESSOR shows any CPU split

Codex can't ask Ollama what the window is, so give it the same number. That way it compacts its history before Ollama starts truncating the prompt. The provider goes in the main config:

# ~/.codex/config.toml
oss_provider = "ollama"

and the rest in a profile. Codex loads $CODEX_HOME/<name>.config.toml on top of the base config when you pass -p <name>:

# ~/.codex/local.config.toml
model = "gpt-oss:20b"
model_context_window = 32768
model_auto_compact_token_limit = 28000   # leave room for the reply
sandbox_mode = "workspace-write"
approval_policy = "on-request"
codex --oss -p local
codex exec --oss -p local "explain why test_kv_restore fails"

Two things in the docs that are easy to miss. codex exec --oss doesn't prompt you to pick a provider like the interactive CLI does. It exits with an error if none is configured. And Codex ignores model_provider and model_providers in a project-level .codex/config.toml, so provider settings belong in your user config.

Options on 16 GB

  • Raise the GPU wired limit. sudo sysctl iogpu.wired_limit_mb=14336 gives the GPU 14 GiB. At 32k context the model needs about 13.6 GiB, which leaves under half a gigabyte for compute buffers and only 2 GiB for the rest of macOS. 64k (14.3 GiB) won't fit. The setting resets on reboot. Re-run the Swift snippet afterwards to check the limit actually changed. I haven't tried this one yet.
  • Live with the CPU split. That's what the numbers above are.
  • Use a machine with more memory. With 24 GB or more, 64k context should fit in the default working set (check with the snippet). You still need OLLAMA_CONTEXT_LENGTH, since those machines also get the 4k default.

Where local Codex makes sense

On this laptop the hosted models are the practical choice for anything substantial. Local mode is for cases where the network is the problem: flights, air-gapped repos, or code that isn't allowed to leave the machine. Even then I'd keep it to read-only work like reviews and explanations. Since it's set up as a profile, switching is just -p local.

If you try this on a different Mac, the Swift snippet and ollama ps will tell you in about a minute whether it fits.