DeepSeek-V4-Flash is a 284B-parameter mixture-of-experts model, about 13B active per token. My Mac has 128GB of unified memory. Getting the two to run together took three real kernel panics before I found a config that holds. The answer was the opposite of what I expected: run the whole 102GB 3-bit quant on pure CPU, not the GPU.

Here are the numbers I wanted when I was searching. The IQ3_XXS quant is about 102GB on disk. -ngl 0 puts zero layers on the GPU. With --no-repack it decodes at ~6.4 tok/s, prefills at ~7 tok/s, sits at ~93GB RSS, and wired memory never leaves the 4-6GB range. Slow, but stable, and the highest-quality DeepSeek that fits in 128GB.

Wired memory: GPU offload versus CPU-onlyOn a 128 GB machine, GPU offload pins about 103 GB of wired memory, into the zone that starves macOS and triggers a kernel panic. Pure-CPU mmap holds about 5 GB.starves macOS128 GB ceilingGPU offload103 GBCPU-only mmap5 GB
Wired memory at steady state. GPU offload pins ~103 GB and starves macOS. Pure-CPU mmap holds ~5 GB. Same model, same machine.

Slow tokens are fine. A kernel panic is not.

Why not just quantize smaller

The whole point was reasoning quality. The 3-bit IQ3_XXS is the best-quality DeepSeek quant that fits 128GB at all, and I was willing to eat slow tokens to keep it. Quantizing down to something comfortable was off the table by design. The question I cared about was how to run a 102GB model on a 128GB machine without the OS falling over.

Panic #1: the naive GPU load

The obvious move on Apple Silicon is to offload everything to Metal with -ngl 99. On smaller models that is free performance. On a 102GB file it takes the machine down.

llama.cpp mmaps the model, and any Metal offload wires the whole residency set into physical memory (the behavior behind llama.cpp issue #24510). About 103GB of wired memory on a 128GB machine starves macOS of the pageable memory it needs to function. The watchdog timer macOS uses to detect a wedged system fires, and the machine kernel-panics and reboots. That takes down the whole OS, not just the llama.cpp server.

Panic #2: the fixes that should have worked

I tried the documented escape hatches. --cpu-moe to keep the expert tensors on CPU. GGML_METAL_NO_RESIDENCY=1 to ask Metal not to wire the set. Partial offload with -ngl 20 instead of 99, on the theory that a fraction of the layers would not wire the whole file.

Same panic every time. As long as any Metal offload is active, enough of that 102GB gets wired to cross the line. The partial-offload theory is wrong here: the mmap wiring does not scale down with the layer count you offload. That cost me a second reboot, chasing "a smaller -ngl will be safe."

Panic #3: a different runtime, same class of bug

While checking whether a different backend handled memory better, a sibling MLX runtime running a large model hit its own panic, an IOGPUMemory underflow instead of a wired-memory starvation. Different mechanism, same family: a big model plus a GPU memory allocator on a machine near its ceiling, with the OS left holding the bag. Three panics across two backends was enough to convince me the GPU path itself was the problem, not any particular flag.

The fix: pure CPU, pageable mmap

-ngl 0 puts every layer on the CPU backend. The reason that helps has nothing to do with speed. The CPU backend leaves the mmap'd model pageable instead of wiring it, so wired memory stays around 4-6GB and the OS keeps the rest as demand-paged file pages it can reclaim under pressure. Run tight on RAM and it pages a little and gets slow. It does not panic. I gave up GPU throughput and bought a machine that stays up.

One more flag mattered. --no-repack avoids a CPU MoE kernel assert on the 3D expert tensor. With it, the model loads and serves cleanly on an OpenAI-compatible endpoint.

llama-server -m model.gguf -ngl 0 --no-repack -fa auto \
  --host 127.0.0.1 --port 8000 --jinja
Two allocation strategies for the same modelLoading the 284B model splits into two paths: GPU offload wires the full tensor and panics the kernel three times; CPU-only mmap stays pageable and runs stably at 6.4 tokens per second.load 284B model102 GB, 3-bitGPU offloadwires full tensor · ~103 GBkernel panic ×3across 2 backendsCPU-only mmappageable · ~5 GB wiredstable6.4 tok/s
The same load, two allocation strategies. Only one keeps the OS alive.

The watchdog

Two reboots made me want a guard that kills a bad launch before it takes the machine down, instead of trusting myself to never set the wrong flag again. The launcher backgrounds a small watchdog that polls memory and aborts the load if it heads toward the panic condition. It kills the load if wired memory climbs past 60GB, which means someone set GPU layers on the big file again, or if swap balloons past a 16GB delta, which means the pageable footprint has passed RAM and is spiraling. It reads vm_stat and sysctl vm.swapusage, no dependencies. It is the same idea as a dead-man's-switch on a data pipeline: watch for the bad state directly instead of trusting that nothing tripped. The one thing you cannot do is raise that ceiling to make a GPU load fit, because a GPU load that needs a higher ceiling is the load that panics.

I packaged the launcher and watchdog as a small MIT-licensed repo, ds4-safe-launch. It works for any big MoE GGUF on Apple Silicon, not just DeepSeek.

On a Mac, "offload everything to the GPU" is the right default right up until the model gets close to your total memory, and then it is the fastest way to reboot the machine. Pure-CPU inference is slow, and it stays up, which for a 102GB model is the whole game. I should have written the watchdog after panic #1, not panic #3.