Roberto19965157 (@Raubertard) profile picture
Joined December 2025
Photo by Roberto19965157 (@Raubertard)Photo by Roberto19965157 (@Raubertard)Photo by Roberto19965157 (@Raubertard)Photo by Roberto19965157 (@Raubertard)Photo by Roberto19965157 (@Raubertard)Photo by Roberto19965157 (@Raubertard)Photo by Roberto19965157 (@Raubertard)Photo by Roberto19965157 (@Raubertard)Photo by Roberto19965157 (@Raubertard)Photo by Roberto19965157 (@Raubertard)
If you don't have a RTX 3090, you can rent one for 13 cents per hour which is almost the price of electricity you would need to run it at home, here is my recipe to run it at high quality and context ⤵️
Another Qwen3.8-27B test results. If you're on 3090/4090 or 24GB vram gpu's, then this one is for you. TLDR/ TLDW: Unsloth UD-IQ4_XS is honestly not bad than Q4_K_M at all and should be used if you need more context and speed. This time i tested two 4-bit quant to see what differences are there. Qwen3.8-27B-Q4_K_M: 22.1 GB Qwen3.8-27B-UD-IQ4_XS: 17.7 GB Context breakdown Tested runs with - ngl 999, kv q8_0, spec_kv q8_0, np 1, auto context at 22GB VRAM usage. With MTP: - UD-IQ4_XS: 106k - Q4_K_M: 42k Without MTP: - UD-IQ4_XS: 209k - Q4_K_M: 142k UD-IQ4_XS gives you more context and speed than Q4_K_M does, all the benefits that comes with being smaller. However, i wanted to find out how significant the quality deterioration is. To test this, i conducted my Three.js eval tests to compare the two. And results were surprising, Not only were the results closer to Q4_K_M, sometimes the result form IQ4_XS were visually better as well. It's not too much, you can be the judge of which is better, Overall for a smaller size, extra context and faster generation speed, i think IQ4_XS is the better solution. Here's the video with some new evals you might have not seen, still just a fraction of what i'm building.
1
2
492
You can use my template and filter on 3090s using this link: cloud.vast.ai/?creator_id=29… (not affiliated or anything).
1
178
Click the rent button, once ready you can click the "Open" button on your instance. You'll see the model downloading in the logs and once finished you can use llama cpp ui in the "applications" tab. Connect open code to your instance using a ssh tunnel and you are ready to go!
149
Here is my setup for Opus-level intelligence on a 16gb consumer GPU at 200k context window, using turboquant, qwopus and MoE offloading, a thread (I'm still not premium) ⬇️
1
1
4
1,012
For the engine, I'm using llama cpp turboquant fork github.com/TheTom/llama-cpp-… with this settings (here on windows using powershell). n-cpu-moe 10 means we let 10 experts on CPU and we have 30 experts on GPU, leaving enough room for 200k context.
1
2
136
400 tokens for prefill and 50-60 for decode. Don't pick the MTP variants unless you can fit all the experts (so basically drop the n-cpu-moe param without any OOM). Here is the result, 50 tokens on an egpu (you should get way more)
1
1
106
I made this work on a 6gb GPU (see here with my settings: teapawt.net/Raubertard/status/2051…) and I found a way to make it even faster! The trick is to use speculative decoding => more in comments (i'm not premium)
"You need a 24 GB GPU for serious local LLMs in 2026." Everyone repeats this. It's not true anymore. Just ran a 35B-parameter model on an RTX 4060 Ti 8 GB: • 41 tok/s at 16k context • 24 tok/s at 200k context Recipe + benchmarks below 🧵
1
284
This trick doesn't work with turboquant fork so I used Q8_0 for kv/cache quant. Full settings here ( batch size 4096 is important with speculative decoding, will use a bit more ram)
1
112
As a personal opinion, i'll keep the version without speculative decoding since turboquant fork gives me more room for a bigger K/V cache but as soon as it works on the fork or if llama cpp merge turboquant, it will be my default setup!
92
I can confirm this thing is real. I'm running opus 4.5 level intelligence on a 6gb vram GPU with 256k context at 20 tokens per second this is unreal. MoE architectures are the best thing ever made
"You need a 24 GB GPU for serious local LLMs in 2026." Everyone repeats this. It's not true anymore. Just ran a 35B-parameter model on an RTX 4060 Ti 8 GB: • 41 tok/s at 16k context • 24 tok/s at 200k context Recipe + benchmarks below 🧵
1
1
17
1,213