Pinned Tweet
I tested Qwen3.8-27B Q6_K on a machine with 2 x 5060 Ti 16Gb. Here are quick results:
• **37.6 t/s** generation — after ~2x speedup via MTP spec decode, draft tensors now ship in the main GGUF
• **150k context** on 32 GB of VRAM, still 19.4 t/s at full depth
• And the plot twist: in our first controlled judge eval it **beat ChatGPT-5.6 Luna and DeepSeek V4 Flash 0731** — a local 27B making the best-calibrated call
Depth sweep, OOM war story, the tuning knobs that all lost, and the eval 🧵
🤖 Made with AI
```bash
llama-server -m Qwen3.8-27B-Q6_K.gguf \
-ngl 99 -fa 1 -c 153600 -ctk q8_0 -ctv q8_0 \
--parallel 1 --jinja --spec-type draft-mtp \
--tensor-split 69,61
# speed profile: drop -ctk/-ctv, -c 98304
```
Footnotes from the trenches:
• Monitors on the dGPU cost ~0.4-0.5 GB. Replugging into the iGPU buys ~15k more context — NOT more speed (split ratio measured: noise)
• llama-cli now ignores `-no-cnv` and hangs scripted runs at a `>` prompt. Benchmark through llama-server
• vs Thread 6's 3.6 era: IQ4_XS hit 51 t/s. Q6_K trades that for quality headroom — see post 6 for what that buys
Model: unsloth/Qwen3.8-27B-GGUF (Q6_K)
Engine: llama.cpp b10200, CUDA 12.8
HW: 2x RTX 5060 Ti 16 GB · Ryzen 9 7900X · 128 GB DDR5
Next up: the moment an IQ4-class quant of 3.8 drops, we chase 50 t/s at this quality. Same GPUs.
2
163
Qwen3.6-27B-MTP at ~61 tok/s. 100k context.
On two *used* RTX 3080 Tis — not the RTX 3090 everyone benchmarks (24GB, but split across 2 cards on PCIe 3.0 x8/x8, no NVLink).
Running llama.cpp's new MTP speculative decoding. The deep-context bottleneck? Nobody's talking about it. 🧵
Final recommended llama.cpp config
```bash
./llama-server \
-m Qwen3.6-27B-Q4_K_M.gguf \
-ngl 99 --split-mode layer --tensor-split 58,42 \
-c 102400 \
-fa on -ctk q4_0 -ctv q4_0 \
--spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75 \
-ub 256 \
-np 1 --jinja -t 8 \
--host 127.0.0.1 --port 8080
```
The flags that carry the findings:
- `-ub 256` — *the* result (5–6): halves the per-prefill compute-graph reserve (~1.8GB), frees ~1.4GB on the choking card. This, not KV-quant, unlocks deep ctx.
- `--tensor-split 58,42` — weight off GPU1 because MTP pins the draft head + growing draft-KV there (7); whole-layer quantized, ~170MiB/step.
- `-ctk/-ctv q4_0` — sane default, but *not* the deep-ctx fix (5).
- `--spec-type draft-mtp --spec-draft-n-max 3` — native MTP, no separate draft model (2).
- `-c 102400` — 100k, the honest ceiling (8); config D ran 0→94k clean. (Config E's `-c 153600` / `-ts 54,46` was only to trace the throughput curve, not for serving.)
# Power cap is NOT a llama.cpp flag — set it first (250W = efficiency knee, ~96% perf / −17% power):
sudo nvidia-smi -pl 250 # both cards
Power scaling — the surprise. Same config, only the cap changed:
- **250W = ~96% of 300W throughput, for 17% less power.** Basically free.
- 200W is where it falls off a cliff: −15% (short ctx) → −23% (98k)
- accept rate fully power-independent (deterministic)
~250W is the efficiency sweet spot on these cards.
Takeaway: 2×3080 Ti is a legit 24GB-class box for 27B + MTP if you (1) tune `-ub`, not KV-quant, for deep context, (2) accept ~100k as the honest ctx ceiling, (3) cap at **~250W** — the efficiency knee: ~96% of 300W throughput for 17% less power.
3
1,174
The VRAM mystery: deep context kept OOMing ~94–114k — but *only on GPU1*. Obvious suspect: KV cache. So we quantized the MTP draft KV f16→q8_0. Result: bit-identical accept rates, +58MiB freed. Zero help. KV was never the problem.
Real culprit: the compute-graph reserve — a ~1.8GB scratch buffer sized to the *physical batch* (`-ub`, default 512), allocated per prefill. Halving it (`-ub 256`) freed ~1.4GB on the choking card and unlocked deep context. The wall was batch-sizing, not KV.
Why GPU1 specifically? Self-speculative MTP puts the draft head + its (ctx-growing) draft KV entirely on one card — there's no `--spec-tensor-split`. At depth it re-saturates that card no matter how you tune `-ts`. Structural to split-VRAM MTP. Found this documented nowhere.
8
1,301
Throughput (Q4_K_M, q4_0 KV, MTP n=3, 300W cap, 24GB split 2 ways):
- short ctx: ~61 tok/s
- 32k: ~47 tok/s
- 65k: ~41 tok/s
- ~98k: ~36 tok/s
MTP accept rate 0.59–0.72 depending on depth. Consumer cards, no NVLink.
7
1,492
The rig: 2× RTX 3080 Ti (12GB ea, 24GB total), i7-7700K, Z270, PCIe 3.0 x8/x8, no NVLink → layer-split, not tensor-parallel. Q4_K_M (~17GB), q4_0 KV, MTP n=3. Both cards power-capped at 300W (from 400W stock) — deliberate for thermals/efficiency, ~5% cost, and it sets up a power-scaling test later. All numbers below @300W.
🤖 Made with AI
7
1,926
Why this is new: llama.cpp just gained native **MTP speculative decoding** (`--spec-type draft-mtp`), and unsloth shipped `Qwen3.6-27B-MTP-GGUF` with the MTP head intact. MTP = the model drafts its *own* tokens — no separate draft model. Almost zero real benchmarks for this exist yet, let alone on split-VRAM consumer rigs.
11
2,153
I installed Hermes Agent on Windows, and set it up with GPT 5.5. I gave it one of @above_spec's amazing twitter threads on Qwen3.6 35B A3B model running on 8GB of VRAM and told it to get it working annnnd... @NousResearch
Quick update on the 35B / 8GB setup.
Switched to IQ4_K_R4 — higher quality quant, without losing much speed — getting ~49tok/s through model's full native 262k context.
And VRAM usage is low enough to keep a browser with multiple tabs open the whole time. 🧵
Qwen3.6 35B A3B model. 55+ tokens/sec. $300 GPU.
No, this isn't a server card. It's an RTX 4060 Ti 8GB.
Previously I posted that I 41 t/s on this gpu and that post blew up and went viral. I went back and made it 34% faster.
And now the speed doesn't drop with context depth at all.
New benchmarks + what changed 🧵
Here is the iq4_k_r4 model for this recipe, so you don't have to quantize yourself: huggingface.co/abovespec/Qwe…
Two configs, same model:
**ncmoe=99** — all 41 expert blocks on CPU:
• ~49 t/s flat from 0 → **262k tokens** (full native ctx)
• 4.9 GB VRAM at max depth → **3.2 GB free**
• Chrome, a few tabs, other light GPU tasks: no problem
**ncmoe=32** — 9 expert blocks on GPU:
• ~55 t/s — +10% faster
• Tops out at ~147k ctx (VRAM ceiling)
• 7.0 GB VRAM at max depth → **1.2 GB free**
• Still enough headroom for a browser, just tighter
The speed gap is real but small. The context gap is not.
For daily use, ncmoe=99 wins.
Full 262k context. Higher quality. 3 GB of VRAM sitting idle while you work.
If you're benchmarking or doing short bursts, ncmoe=32 for the extra 10%. But as a background assistant that's always on? ncmoe=99 is the reliable pick.
RAM note: expert weights (~19 GB) stream from system RAM via mmap. You only need **32 GB RAM** to run this comfortably. DDR5 preferred — the CPU expert throughput is bandwidth-bound, so faster RAM = faster tokens.
```bash
llama-server \
--model Qwen3.6-35B-A3B-IQ4_K_R4.gguf \
-ngl 99 --n-cpu-moe 99 -fa 1 \
-ctk q4_0 -ctv q4_0 \
-c 262144 -t 12
```
Model: abovespec/Qwen3.6-35B-A3B-IQ4_K_R4-GGUF
Engine: ik_llama.cpp (IQ4_K_R4 won't load in mainline)
HW: RTX 4060 Ti 8 GB · Ryzen 9 7900X · 93 GB DDR5
Next up: can we squeeze in a small drafter model for spec decoding? That's a separate thread.
If you have successfully done spec decoding with 35b a3b please teach me!
3
17
3,061
















