Dr Plan L ∞/21M ♽

llama.cpp merged something big for Localmaxxers whose AI models don't fit in VRAM. 👀🔥 (Strata has this too) The upgrade is pure software and no expensive or hard-to-find new GPU needed. PR #30966 merged November 7. Here's what they've added. When you run a huge Mixture-of-Experts (MoE) model that doesn't fit in GPU memory, llama.cpp can cache frequently used experts in VRAM. But what happens when the model needs an expert that ISN'T in the cache? Previously, the MoE cache path would upload that expert's weights from system RAM into GPU VRAM for execution. That means transferring potentially large amounts of data across PCIe. Now llama.cpp has another option. Run some of those missing experts directly on the CPU! This is important especially when running models on eGPU (low bandwith thuderbolt) No need to transfer their full weights into VRAM first. Okay, so what? The developer tested Qwen3.8-Flash-Next IQ4_NL with approximately 6.3GiB allocated to the GPU expert cache. ⚡ GPU uploads only → 24.2 tps 🚀 CPU handles 95% of cache misses → 47.6 tps That's 96.7% FASTER decoding! And an RTX 3090 user tested Qwen3.8-Flash-Next UD-IQ4_XS with a 5.5GiB GPU cache. 🐌 GPU uploads only → 19.5 tps 🚀 CPU handles 95% of misses → 31.8 tps Approximately 63% faster on the same RTX 3090! Now here's the clever part. The developers also added an automatic tuning script. It measures and evaluates combinations of: 💾 GPU expert-cache size 🧠 CPU execution share ⚡ Expert-cache hit rates 🔄 CPU/GPU transfer performance And recommends settings for your hardware. Because more CPU offloading isn't always faster. With a larger ~19GiB GPU cache, the developer measured these ⚡ GPU uploads only → 65.5 tps 🚀 50% CPU miss share → 66.4 tps 🐌 95% CPU miss share → 54.1 tps See what happened? The best strategy changes depending on how much VRAM is available and how fast the CPU, GPU and memory can move and process data. And to be clear, 95% CPU miss share does not necessarily mean the CPU runs 95% of the model. It means the CPU handles approximately 95% of eligible GPU cache misses, while cached experts still execute on the GPU. Your CPU and system RAM aren't just places to park model weights now. More regular RAM is better. They can actively help execute the model when moving those weights would be slower. And now this capability is merged into upstream llama.cpp. Free. Open source. No new GPU required. 🔥 Link in ALT.