optimization enthusiast @trymirai . || Fast vs Slow Thinking
Inside Kernels
Joined January 2018
- Tweets353
- Following563
- Followers268
- Likes434
Pinned Tweet
Glad I can finally share another project we have been cooking! We built Weaver to make speculative decoding practical( and SOTA😉) for Qwen3.5-27B… we’re ready to share the paper, draft model, and SGLang reference implementation.
It will be in our own inference engine soon but in the meantime test and play with it on your own!
Paper: arxiv.org/abs/2607.06763
Model: huggingface.co/trymirai/weav…
Code: github.com/trymirai/sglang
We are releasing our speculative decoding approach, together with the draft model for Qwen3.5-27B and the reference implementation in SGLang.
Paper: arxiv.org/abs/2607.06763
Model: huggingface.co/trymirai/weav…
SGLang implementation: github.com/trymirai/sglang
ryan mathieu retweeted
Mirai made Qwen3.8 Flash-Next 2-bit, I made it UNCENSORED, and now it does 91-103 tok/s on ONE 3090 with the experts in system RAM. 41 GiB of RAM, 262K context, vision.
I ported their Strata patch to Strata 0.2 and put the weights on HF, so you can run it today.
@trymirai gave me early access and said I can publish my own versions. The hard work is theirs: the quant, the Strata format, the kernels and the converter. The S files in my repo are their files byte for byte. I moved the patch from Strata 0.1.43 to 0.2.0, converted their 32 GB S-small with their converter and added the refusal vector and the MTP draft layer.
On my fork, S on 1 card at 300W:
4K: 91-102 tok/s
32K: 99-101 tok/s
131K: 100-103 tok/s
Prefill is the weak part, 0.9-1.6K tok/s on 1 card depending on prompt length.
Earlier I ran it on 0.1.43 with Mirai's patch, same rig. There it did 80-88 tok/s on 1 card, and the IQ3_S GGUF I use daily did 74-84 and needed 53 GiB of RAM. On 2 cards S did 104-113 with 42 GiB, IQ3_S 95-108 with 80 GiB. I gave it a 217K token document with 3 hidden codes on 2 cards, it read it in 86 seconds and found all 3. S-small needs only 33 GiB of RAM and did 88-98 on 1 card, 100-118 on 2.
The refusal vector I made from Huihui's abliteration works on it without changes: 2 refusals out of 146 harmful prompts, the stock model refused 141. The quality numbers are Mirai's and I didnt check them, KL 0.040 to bf16 and 0.303 for the IQ2_XS GGUF.
On 2 cards their first patch just repeated one token forever. The rotation constants were uploaded only to the first GPU, so the second card had zeros. 7 lines fixed it, Mirai merged the fix and put my name in their README.
Upstream Strata cant read these files yet. The port is a draft PR to Niko, until it lands build my fork:
huggingface.co/alesha-pro/Qw…
github.com/alesha-pro/Strata
PR: github.com/Niko1221/Strata/p…
Original quant by Mirai: huggingface.co/trymirai
ryan mathieu retweeted
882 tok/s prefill on M5 Max for Qwen3.8-27B, with uzu.
Compared to Splash:
prefill: 1.83x faster
time to first token: 1.81x faster
generation: 1.05x faster
Measured on MT-Bench, MATH-500 and HumanEval.
ryan mathieu retweeted
This inference engine runs LLMs 4x faster than llama.cpp:
(Qwen3.5-9B at 92 tokens/s on 16GB Mac)
I ran Qwen3.5 9B on a base M5 MacBook Pro using three local inference engines, one at a time.
→ llama.cpp reached 22.0 tokens/s
→ MLX reached 25.1 tokens/s
→ Uzu reached 92.1 tokens/s
The video shows all three runs side by side.
The first two results do not mean llama.cpp or MLX are poorly optimized. Both are approaching a physical limit of ordinary local decoding.
Generating each new token normally requires another pass through the model. That pass streams roughly 5.2 GB of weights through memory.
A base M5 provides about 153 GB/s of memory bandwidth.
At one token per pass, the theoretical ceiling is roughly 153 ÷ 5.2, or 29 tokens/s, however fast the GPU computes.
Uzu gets past that ceiling by optimizing speculative decoding for Apple Silicon.
Speculative decoding uses a smaller drafting system to propose several future tokens. The full 9B model then checks those proposals together instead of generating every token through a separate pass.
Uzu pushes this further in two ways.
→ It drafts up to 16 positions and uses Weaver to turn independent guesses into coherent sequences.
→ It verifies several possible sequences as a tree, then commits state only for the path the full model accepts.
The 9B model remains in control of the final output. Uzu simply gives it more useful work to approve in each pass.
In my run, every pass of the 9B model produced 7.5 output tokens on average.
Ordinary decoding produces exactly 1.
That is how Uzu reached 92.1 tokens/s on hardware whose one-token-per-pass ceiling is roughly 29.
The rest of the engine is designed around this path too. Uzu ships its own quantized checkpoints, and on M5 chips its Metal kernels run verification through the GPU’s accelerated int8 matrix path.
It is an open-source inference engine from Mirai, with bindings for Rust, Swift, Python, and TypeScript.
The repository is here: github.com/trymirai/uzu
(don't forget to star 🌟)
I wrote the full breakdown of why local inference is memory-bound and how Uzu gets past that limit, with reproducible results and code.
The article is quoted below.
ryan mathieu retweeted
A 27B model in 12 GB of VRAM. Full 262K context. 70 tok/s. And now it is UNCENSORED.
I ran it on my own RTX 3090, held to 12 GB. This is Mirai S Qwen3.8-27B, 2.4 bits per weight, an 11 GB file.
With my llama.cpp port the same card did 82K context at 38 tok/s inside 12 GB. Then @CaryPalmerr took the port and did the hard part. Same card, same file, his serve:
- context: 82K -> 262,144 with q8 KV
- decode: 38 -> 69.9 tok/s
- VRAM peak: 11.7 GB
- prefill: 1,124 tok/s
He keeps part of the KV cache in VRAM and the rest in pinned RAM, and he drafts with MTP at every depth. He checked every kernel against my fork token for token and wrote down every experiment, the failed ones too. On his real 12 GB RTX 4070 it does 76.6.
My part is small. I added abliteration. You cant edit these weights, they are compressed codes, so I measured the refusal direction on the quantized model and the engine removes it while the model runs. A 1.3 MB file and 1 flag:
- harmful prompts refused: 64 of 64 -> 0 of 64
- held-out set: 78 of 82 -> 0 of 82
- decode: 69.9 -> 69.7 tok/s
Counted by a regex, and with thinking on 1 of 24 still refuses. Deep in the context it gets slower on my card: 27.9 tok/s at 60K.
So thats 12 GB of VRAM, 262K context, 70 tok/s, abliterated. Open weights, open code, runs on your desk.
Cary did a genius amount of work and almost nobody has seen it. Go follow @CaryPalmerr right now, I mean it.
serve: github.com/professorpalmer/m…
weights + vector: huggingface.co/alesha-pro/Qw…
ryan mathieu retweeted
Qwen3.8-27B running at just 8.45 GB and hitting 60 tok/s on a single RTX 3090 is the kind of local AI result that makes you rethink what a 27B model actually needs.
I tested the new 2.4-bit Qwen3.8-27B quant from trymirai on a real coding task.
The prompt was basically:
Build a voxel pagoda garden.
No pile of separate files.
No huge framework.
Just a single HTML file using Three.js.
The agent setup was:
• Pi Agent
• Medium reasoning
• 10 turns
• 32.5K output tokens
• About 12 minutes
• One prompt to start the project
And it produced the pagoda.
That matters more to me than a model fitting into a certain file size. A heavily compressed model is only useful if it can still maintain context, reason through changes and actually finish a task.
The performance on my setup was also surprisingly strong.
1× RTX 3090
60 tok/s with MTP
44 tok/s without MTP
~1,190 tok/s prefill on a 7K-token prompt
159K context
The GPU was running around 290W.
Then I moved to two 3090s with pipeline parallelism:
2× RTX 3090
66 tok/s decode
~1,890 tok/s prefill
Full 262K context
So the trade-off here is pretty interesting.
The model is slightly larger than some of the more aggressively compressed Qwen3.8-27B releases, but the goal isn’t simply to win a file-size contest.
You want the compressed model to remain useful.
And in this test, the 2.4-bit model was able to handle a multi-turn Three.js build, generate 32.5K tokens and maintain the project through ten turns.
That’s the real promise of aggressive quantization.
Take a model that normally requires much more memory, compress it heavily, put it on hardware that people already own, then see whether it can still perform actual work.
27B parameters.
8.45 GB.
60 tok/s on one 3090.
262K context with two.
That’s a pretty serious local setup for a model this size.
ryan mathieu retweeted
qwen on 16gb mac mini is crazy
Mirai Labs (@trymirai) compressed Qwen3.8-27B to 2.4 bits. I ported it to llama.cpp and now it runs on a 12 GB card with 128K context.
With q4 KV cache the 128K window takes 11.3 GB of VRAM, with q8 KV you get 74K in 11.1 GB. Decode is 40 tok/s, still 34-35 tok/s at 62K deep, and prefill around 1000 tok/s. I measured all of it on 1 RTX 3090 at 300 W, capped to what fits in 12 GB.
No requant anywhere. The GGUF carries Mirai's compressed codes bit for bit, and greedy output matches their own vLLM plugin.
Everything in the video was built by this same model as an agent in pi, 11 builds from 1 prompt each, a few needed a follow up and 1 is a blooper. That run was vLLM on my 3090s. Today the same weights run in llama.cpp.
Big thread, all links and recipes 👇
ryan mathieu retweeted
Mirai Labs (@trymirai) compressed Qwen3.8-27B to 2.4 bits. I ported it to llama.cpp and now it runs on a 12 GB card with 128K context.
With q4 KV cache the 128K window takes 11.3 GB of VRAM, with q8 KV you get 74K in 11.1 GB. Decode is 40 tok/s, still 34-35 tok/s at 62K deep, and prefill around 1000 tok/s. I measured all of it on 1 RTX 3090 at 300 W, capped to what fits in 12 GB.
No requant anywhere. The GGUF carries Mirai's compressed codes bit for bit, and greedy output matches their own vLLM plugin.
Everything in the video was built by this same model as an agent in pi, 11 builds from 1 prompt each, a few needed a follow up and 1 is a blooper. That run was vLLM on my 3090s. Today the same weights run in llama.cpp.
Big thread, all links and recipes 👇
ryan mathieu retweeted
27B model in 8.45 GB. 60 tok/s on a single 3090. And it built this voxel pagoda from a single prompt.
Qwen3.8-27B in 2.4 bits from @trymirai is on a completely different level compared to the mess from @PrismML. Yeah, it's slightly larger in size, but these guys don't claim 98% accuracy either, unlike some others.
Pagoda: 1 prompt, medium reasoning, pi agent, 10 turns, 32.5K output tokens, 12 minutes. A single HTML file on three.js.
Speed on my rig, GPU at 290 W, vLLM:
1x 3090: 60 tok/s with MTP, 44 without, prefill ~1190 tok/s on a 7K prompt, 159K context
2x 3090 (pipeline parallel): 66 tok/s, prefill ~1890 tok/s, full 262K context
Huge thanks to @norpadon and @gapDEEPry for the opportunity to test this model before release and help the team with testing.
We made the preview publicly available: huggingface.co/trymirai/Qwen…
Qwen3.8 27B that fits in 8.5Gb
Works with an experimental build of uzu (our inference engine) on Apple Silicon, and in vllm on Nvidia
See the model card for instructions
ryan mathieu retweeted
We made the preview publicly available: huggingface.co/trymirai/Qwen…
Qwen3.8 27B that fits in 8.5Gb
Works with an experimental build of uzu (our inference engine) on Apple Silicon, and in vllm on Nvidia
See the model card for instructions
ryan mathieu retweeted
We may have cooked a 2-bit version of Qwen3.8 27B which *actually works*, is not deep fried to benchmaxx, and fits in 8.5GB
We have not yet finished evaluating it, and it'll take some time to properly ship it, but if someone wants to try and share feedback please dm me
ryan mathieu retweeted
The latest version got significantly improved performance, especially on prefill
We made a 27B model run at over 100t/s on a MacBook*
We did this by co-designing the quantization and speculative decoding around the latest Apple M5 chips, to take maximal advantage of Apple's neural accelerators
trymirai.com/local-models/al…
*For now on M5+ chips only
ryan mathieu retweeted
Today we are releasing our speculative decoding implementation in our inference engine uzu.
Initially for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon.
On Apple M5-series chips, we outperform MTPLX (MLX + speculative decoding) by almost 2x, and llama.cpp by over 3x at comparable quantization levels, with the strongest gains achieved on mathematical reasoning and coding tasks.
Run the model:
Mirai-M: trymirai.com/local-models/al…
Mirai-L: trymirai.com/local-models/al…
Explore the benchmarks:
trymirai.com/metrics
Learn more about our speculative decoding implementation:
trymirai.com/blog/speculativ…
Our draft model, quantized checkpoint format, verification algorithm, and GPU kernels are co-designed from the ground up around the latest Apple M5 chips to take maximum advantage of GPU Neural Accelerators.
Unlike popular speculative decoding architectures such as model-native MTP, which produce small draft chains of 3-4 tokens at a time, we use extremely aggressive speculative budgets of 16-32 tokens. This enables us to use Neural Accelerator-backed GEMM kernels, achieving maximum utilization of hardware arithmetic throughput.
ryan mathieu retweeted
Today we are releasing our speculative decoding implementation in Uzu. Biggest release since inception of our lab.
Initially, for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon.
ALT Best performing local ai stack
ryan mathieu retweeted
what if they used this style for their iPod launch?
with contentyard.io, you don't have to imagine.
ryan mathieu retweeted
wow looks cool
We are releasing our speculative decoding approach, together with the draft model for Qwen3.5-27B and the reference implementation in SGLang.
Paper: arxiv.org/abs/2607.06763
Model: huggingface.co/trymirai/weav…
SGLang implementation: github.com/trymirai/sglang
ryan mathieu retweeted
We are releasing our speculative decoding approach, together with the draft model for Qwen3.5-27B and the reference implementation in SGLang.
Paper: arxiv.org/abs/2607.06763
Model: huggingface.co/trymirai/weav…
SGLang implementation: github.com/trymirai/sglang
ryan mathieu retweeted
When we invested in Mirai Labs (@trymirai) earlier this year, they had a long-ish term plan to offer full-stack on-device inference infrastructure and the underlying models. I thought it would take them a year...
In less than four months, their co-designed quant+inference stack for Apple Silicon shows 40–60% more tokens/sec vs llama.cpp and MLX at the same quality level.
This week they published technical posts on quantization (trymirai.com/blog/quantizati…) and sparse buffers (trymirai.com/blog/sparse-buf…) which I encourage app builders and inference consumers to read.
Awesome stuff from @Darkolorin, @dimashvets, and the Mirai team.
ryan mathieu retweeted
Really proud of this release from the team :)
We just shipped our first quantized Qwen3.5 checkpoints, the quantization format was co-designed with our inference engine from scratch instead of trying to make an off the shelf format fast. Our 8-bit models decode faster than llama.cpp 4-bit on Apple silicon, and they're virtually identical to the original!