1 line is all it takes. local ai, agents, the full context, made handover, building & breaking things in public
127.0.0.1
Joined November 2010
- Tweets3.2K
- Following100
- Followers404
- Likes4.4K
Abe Kale retweeted
Nous Calendar โ What did Nous ship today? ๐๏ธ
Nous shipped 2,189 changes to Hermes Agent this month. Most of us saw maybe three.
8 releases. 12 new official plugins. 417 new catalog plugins. In 23 days. ๐คฏ
So I built Nous Calendar: every Nous drop on one page, inside Hermes Desktop.
What you get:
โธ Every day is a card: black is a release, blue is a launch
โธ Tap a day to see exactly what shipped, links open inside Hermes
โธ The day's biggest posts on X, with their views
โธ Tells you when you're behind on updates, and copies the fix
โธ Every new plugin is one click to install
โธ Ask Hermes "what did Nous ship today?" in any chat
Free and open source. Reads public GitHub and a small public feed of the biggest posts. No account, no model calls.
Install command + phone link in the first reply ๐
Abe Kale retweeted
๐จ @MiniMax_AI ๐ ๐ฏ.๐ญ ๐๐๐๐ฆ๐ โ ๐ฐ๐ฎ๐ด๐ ๐ฃ๐๐ฅ๐๐ ๐๐ง๐๐ฅ๐ฆ. ๐ข๐ก๐ ๐๐๐ซ ๐ฆ๐ฃ๐๐ฅ๐. ๐ต๐ณ% ๐ง๐ข๐๐๐ก ๐๐๐ฅ๐๐๐ ๐๐ก๐ง! ๐งช๐ฅ
I've done it AGAIN, and this time I'm releasing TWO new SAGE-EXL3 quantizations of the brand-new MiniMax M3.1 Flash!
428B total parameters.
23B active.
Native 1M-token context capability.
My goal was simple:
๐ฆ๐ค๐จ๐๐๐ญ๐ ๐ง๐๐ ๐ ๐ข๐๐๐, ๐ก๐ข๐ง ๐๐ง๐ฆ ๐๐ก๐ง๐๐๐๐๐๐๐ก๐๐.
And the fidelity measurements are some of my most exciting yet! ๐คฏ
โโโโโโโโโโโโโโโโโโโโ
๐ข ๐ฅ๐๐๐๐๐ฆ๐ #๐ญ โ ๐ข๐ก๐ ๐๐๐ซ ๐ฆ๐ฃ๐๐ฅ๐
๐ ๐ถ๐ป๐ถ๐ ๐ฎ๐
-๐ ๐ฏ.๐ญ-๐๐น๐ฎ๐๐ต-๐๐ซ๐๐ฏ-๐ฆ๐๐๐-๐ฎ.๐ญ๐ฑ๐ฏ๐ฝ๐
๐ฆ 108.64 GiB checkpoint
๐ง 96.89% Top-1 Token Agreement
๐ฏ 99.75% Top-5 Agreement
๐ Mean KL Divergence: 0.02528
๐๐๐ ๐ข๐ ๐ง๐๐๐ง ๐๐ก ๐ ๐ฆ๐๐ก๐๐๐ ๐ญ๐ฎ๐ด๐๐ ๐๐๐ซ ๐ฆ๐ฃ๐๐ฅ๐!
And YES, I already have a working vLLM serving configuration!
But there's a catch, and I'll explain why shortly. ๐
โโโโโโโโโโโโโโโโโโโโ
๐ฃ ๐ฅ๐๐๐๐๐ฆ๐ #๐ฎ โ ๐ง๐ช๐ข ๐๐๐ซ ๐ฆ๐ฃ๐๐ฅ๐๐ฆ
๐ ๐ถ๐ป๐ถ๐ ๐ฎ๐
-๐ ๐ฏ.๐ญ-๐๐น๐ฎ๐๐ต-๐๐ซ๐๐ฏ-๐ฆ๐๐๐-๐ฐ.๐ฎ๐ฒ๐ฏ๐ฝ๐
๐ฆ 209.88 GiB checkpoint
๐ง 98.44% Top-1 Token Agreement
๐ฏ 99.94% Top-5 Agreement
๐ Mean KL Divergence: JUST 0.00592!
๐ก๐๐ก๐๐ง๐ฌ-๐๐๐๐๐ง ๐ฃ๐ข๐๐ก๐ง ๐๐ข๐จ๐ฅ ๐๐ข๐จ๐ฅ ๐ฃ๐๐ฅ๐๐๐ก๐ง ๐ง๐ข๐๐๐ก ๐๐๐ฅ๐๐๐ ๐๐ก๐ง! ๐คฏ
That's exceptionally close to the unquantized reference in my held-out token-level evaluation.
Both releases were measured against the original unquantized MiniMax checkpoint using full-vocabulary teacher-forced evaluation.
These are quantization-fidelity measurements, not task pass rates or proof of mathematically lossless behavior.
But THESE are the kinds of quality-per-bit results I'm chasing!
โโโโโโโโโโโโโโโโโโโโ
โ ๏ธ ๐ง๐๐ ๐ฃ๐ฅ๐ข๐๐๐๐ ๐๐ฆ๐ก'๐ง ๐๐จ๐ฆ๐ง ๐ง๐๐ ๐ ๐ข๐๐๐ ๐๐ก๐ฌ๐ ๐ข๐ฅ๐.
It's the inference engine.
My current ONE-SPARK vLLM recipe WORKS, but it's still a WORK IN PROGRESS.
The measured configuration currently reaches 32K context using a quantized KV cache.
MiniMax M3.1 Flash supports up to 1M native context, but vLLM's memory overhead on this hardware is consuming headroom I'd rather dedicate to the model and its context.
I could shrink the weights further to compensate.
๐๐จ๐ง ๐ช๐๐ฌ ๐ฆ๐๐๐ฅ๐๐๐๐๐ ๐ ๐ข๐๐๐ ๐๐๐๐๐๐๐ง๐ฌ ๐ง๐ข ๐๐๐๐ข๐ ๐ ๐ข๐๐๐ง๐ ๐ฅ๐จ๐ก๐ง๐๐ ๐ ๐ข๐ฉ๐๐ฅ๐๐๐๐?!
That's the wrong trade-off for what I'm trying to accomplish.
I'm optimizing for the BEST possible balance between model quality, memory usage and practical inference speed.
And that's where something else I've been building comes in...
โก ๐ฅ๐๐ซ๐๐ฏ.
Our upcoming public inference engine and dedicated recipe are aimed at removing these limitations and making the model's full native context practical on Spark.
We're continuing to validate that work before release.
๐ฆ๐ง๐๐ฌ ๐ง๐จ๐ก๐๐. ๐ง๐๐ ๐ฃ๐จ๐๐๐๐ ๐ฅ๐๐๐๐๐ฆ๐ ๐๐ฆ ๐๐๐ง๐ง๐๐ก๐ ๐๐๐ข๐ฆ๐๐ฅ! ๐
โโโโโโโโโโโโโโโโโโโโ
๐ค ๐๐๐๐๐๐ก๐ ๐๐๐ ๐ง๐ช๐ข-๐ฆ๐ฃ๐๐ฅ๐ ๐๐๐ฉ๐๐๐ข๐ฃ๐๐ฅ๐ฆ!
Here's where I'd love some help from the community!
The 4.26 bpw model is already available on Hugging Face, but I haven't had the time to properly build and validate its TWO-SPARK serving recipe.
If you have two DGX Sparks and experience with distributed inference, I'd LOVE to collaborate!
Fork my GitHub recipe.
Experiment.
Submit a PR with your improvements and measurements.
๐ ๐ช๐๐ก๐ง ๐ง๐ข ๐ช๐๐๐๐ข๐ ๐ ๐๐ข๐ก๐ง๐ฅ๐๐๐จ๐ง๐ข๐ฅ๐ฆ ๐๐ก๐ ๐๐๐ฉ๐ ๐ง๐๐๐ ๐ฃ๐ฅ๐ข๐ฃ๐๐ฅ ๐๐ฅ๐๐๐๐ง ๐๐ข๐ฅ ๐ง๐๐๐๐ฅ ๐ช๐ข๐ฅ๐! ๐
I want these releases to push the quality-per-bit frontier for EXL3, while making it easier for the community to reproduce and extend the inference results.
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ข๐ก๐ ๐ฆ๐ฃ๐๐ฅ๐ โ ๐ฎ.๐ญ๐ฑ๐๐ฃ๐ช
huggingface.co/vcruz305/Miniโฆ
๐ ๐ง๐ช๐ข ๐ฆ๐ฃ๐๐ฅ๐๐ฆ โ ๐ฐ.๐ฎ๐ฒ๐๐ฃ๐ช
huggingface.co/vcruz305/Miniโฆ
๐ ๏ธ ๐ฅ๐๐๐๐ฃ๐ & ๐๐ข๐ ๐ ๐จ๐ก๐๐ง๐ฌ ๐ฃ๐ฅ๐ฆ
github.com/vcruz305/MiniMax-โฆ
๐ง๐ช๐ข ๐ค๐จ๐๐ก๐ง๐ฆ. ๐ง๐ช๐ข ๐ ๐๐ ๐ข๐ฅ๐ฌ ๐ง๐๐ฅ๐๐๐ง๐ฆ. ๐ข๐ก๐ ๐ฃ๐ฅ๐๐ข๐ฅ๐๐ง๐ฌ: ๐๐๐๐๐๐๐ง๐ฌ.
And the next chapter is getting very interesting. ๐งช๐ฅ
MiniMax M3.1 Flash weights are now public. 428B parameters total, 23B activated per token.
Agentic performance: A generational leap in agentic tasks, coding, and cybersecurity โ 76.2 on BankerToolBench, 73.7% on DeepSWE v1.1, 61.7 on OSWorld V2, 29.6 on Agents' Last Exam, and 82.2 on CyberGym.
Efficient by design: MiniMax Sparse Attention and DSpark speculative decoding improve inference efficiency.
Hugging Face:
huggingface.co/MiniMaxAI/Minโฆ
ModelScope:
modelscope.cn/models/MiniMaxโฆ
API: platform.minimax.io
M Plan: minimax.io/m-planMiniMax
MCode for desktop: agent.minimax.io/download
Abe Kale retweeted
Herald is on Android ๐
Run one command on your computer, scan the code, and your Hermes Agent bots are in your pocket.
๐ Approve risky steps right from the notification
๐ Share links from Chrome straight to a bot
๐ฅ Group chats where bots take turns
It talks directly to the Hermes on your own computer, over Wi-Fi or Tailscale. Also in there: live tool calls, HTML previews, new bots from your phone, any model, kanban and schedules.
Not an official Hermes app, just something I built ๐
play.google.com/store/apps/dโฆ
xitter.cf/iamlukethedev/status/2โฆ
Hermes Agent is officially in my pocket
Herald is now in public beta on TestFlight ๐
Setup takes less than a minute:
Run one command on your computer
Scan the QR code
Your Hermes Bots are on your iPhone
Voice notes
Group chats
Approvals
Kanban
Bot Screen
Apple Watch
And a lot more
Over 200 pieces of feedback went into this version. I read every one and kept shipping.
Now I need you to break it ๐
Try it here:
testflight.apple.com/join/Adโฆ
Watch all the features below ๐
Herald is NOT the official Hermes app. Itโs something I started building because I wanted Hermes on my phone and couldnโt wait.
xitter.cf/iamlukethedev/status/2โฆ
Abe Kale retweeted
I built a full game asset kit with my Hermes Agent and the Meshy plugin, about ~1h30 in one Hermes Desktop chat.
what we made:
- a duck knight hero, rigged, with idle, slash, walk and run animations
- a hooded cultist duck enemy, rigged, with idle, spell cast, walk and run animations
- 9 props for a ruined, Lovecraftian level: a shrine, a cursed lantern, an iron gate, two tombstones, a tentacled idol, a treasure chest, an altar and a broken pillar
- 17 game-ready GLB files
- the duck knight as a 10 cm figure, single color, STL and OBJ, ready for a slicer
what Hermes did with Meshy:
- planned the kit with a credit cost per step, then waited for my yes
- made design images with front, side and back views, so I picked the look before anything went 3D
- turned the picks into textured 3D models and showed each one in the chat
- rigged both characters and added the animations
- built the props in batches of three
- made everything game-ready: the triangle counts were already fine, so it shrank the 4K textures instead and took the kit from 500 MB to 132 MB
- checked the figure for printing at 10 cm, repaired the walls that were under 1 mm, and checked it again
it asked before every paid call. the whole kit used 541 credits.
@MeshyAI is offering Hermes users a month of Pro with 1,000 credits for $1, so this kit used about half of it.
all 89 files landed in one project folder.
Abe Kale retweeted
TensorFold 1.1.3 is live.
A load of bugs fixed and improvements, such as Flash Next image-chat caching, tool-call and reasoning fixes, checkpoint loading fixes, and Windows builds restored.
A huge thank you to @sethforprivacy @ahinore @Bizuayeu @plotarmordev @jayleaton @bobclawblaw @codengod, and GitHub contributors tpischke, jschmied, chaog992, cwschroeder, akol1 and flashosophy.
And thank you to everyone testing, reporting bugs and sharing access to their hardware.
@MiaAI_lab @plotarmordev @petruspennanen @volatilemarkts
tensorfold.dev
Abe Kale retweeted
Stop opening Hermes and wondering what to give it. ๐ฅ
Five jobs for your first useful day. Each one ends with something you can check.
โธ Saved tabs โ a decision brief, one source per claim
โธ Meeting notes โ decided, owned, still open
โธ Agent folders โ a read-only disk report
โธ A failed task โ resume, don't repeat
โธ A job you keep repeating โ a skill
Start with the one that matches a problem you already have. Use copies of important files. Ask Hermes to name any missing access before it starts.
The runs in the video are sample runs. The disk numbers are from my own Footprint recording: it inspected and asked. What it recovered is a separate run.
The first prompt is in the reply. The full guide with all five is coming as an X Article.
Which of the five is your problem today?
Abe Kale retweeted
What gated defenders looks like in practice.
My VulnHunter runs on local weights because I refuse to ask a vendor whether I'm allowed to read code like an attacker.
DeepSeek V4.1 Flash 1017, fully local: 24 of 24 planted vulns found from source. Written proofs held on 22 of 24.
No access request. No usage review. No waiting.
github.com/nealbridges/VulnHโฆ
Abe Kale retweeted
Hermes Agent, but make it an OSINT situation room.
Earthquakes, disasters, military aircraft, satellites and breaking incidents on one live 3D globe. Click any country for a dossier. Hit "Brief me" and Hermes writes the sitrep itself.
All inside @NousResearch Hermes Desktop.
Abe Kale retweeted
Someone showed me this new JS Library for dithering visualizations so I gave it to Hermes Agent, said make me a webpage full of demos for all the lesser known cool styles and components libraries out there.
In 20 minutes it did, and there's some amazing stuff it found. This webgl fluid library is so sick.
Check it out here: webdev.teknium.io/
Abe Kale retweeted
BLACKFOLD IS LIVE. ๐คฏ
Uncensored DeepSeek-V4.1-Flash 1017 running locally on two DGX Sparks.
No restrictions. Blazing speeds:
โก๏ธ 2,008 tok/s prefill (128K context)
๐ป 113 tok/s code decode
Virtually zero refusals matching api quality.
Everything I fixed to get there ๐งต
Abe Kale retweeted
๐จ ๐ง๐ช๐ข ๐ก๐๐ช ๐๐๐๐ฃ๐ฆ๐๐๐ ๐ฉ๐ฐ.๐ญ ๐๐๐๐ฆ๐ ๐ฅ๐๐๐๐๐ฆ๐๐ฆ! ๐ก๏ธ๐ฅ
Iโm excited to share TWO alternate DERISKED SAGE-EXL3 quantizations for cybersecurity research and security-focused AI workflows! These are the HIGHEST quality EXL3 quants you can find for this Deepseek model!
Built in collaboration with @Blackfrost_AI ๐ฅถโ๏ธ
๐ข 1.61 bpw โ ONE DGX SPARK
โ
94.68% assistant-token top-1 agreement
โ
~112.9 GiB resident footprint
๐ฃ 2.50 bpw โ TWO DGX SPARKS
โ
96.71% assistant-token top-1 agreement
โ
Higher precision for those with more memory - practically lossless ๐
Both measured against DeepSeekโs official FP8/FP4 checkpoint. These are token-fidelity results, not cybersecurity task scores.
๐ ๐ข๐ฅ๐ ๐๐๐ข๐๐๐๐ฆ. ๐ ๐ข๐ฅ๐ ๐๐ข๐๐๐ ๐๐. ๐ ๐ข๐ฅ๐ ๐๐ฅ๐๐๐๐ข๐ ๐ง๐ข ๐๐ซ๐ฃ๐๐ฅ๐๐ ๐๐ก๐ง! ๐
๐ 1.61 bpw โ ONE SPARK
huggingface.co/vcruz305/DSV4โฆ
๐ 2.50 bpw โ TWO SPARKS
huggingface.co/vcruz305/DSV4โฆ
Huge thanks to @Blackfrost_AI for not only making the model, but providing the compute for us to continue collaborating on these releases! ๐๐ฅ
๐จ ๐ ๐๐จ๐ฆ๐ง ๐๐๐๐๐๐ฉ๐๐ ๐ฆ๐ข๐ ๐๐ง๐๐๐ก๐ ๐๐ก๐๐ฅ๐๐๐๐๐๐ ๐ข๐ก ๐ข๐ก๐ ๐๐๐ซ ๐ฆ๐ฃ๐๐ฅ๐! ๐งช๐ฅ
With the support of @AWERXAI, I'm excited to release something VERY special for the local AI community!
๐ง @deepseek_ai ๐๐ฒ๐ฒ๐ฝ๐ฆ๐ฒ๐ฒ๐ธ ๐ฉ๐ฐ.๐ญ ๐๐น๐ฎ๐๐ต ๐ญ๐ฌ๐ญ๐ณ
๐ฆ ๐ฆ๐๐๐-๐๐ซ๐๐ฏ ๐ฎ๐ ๐ท๐๐๐ ๐ญ.๐ฒ๐ฎ ๐ฏ๐ฝ๐ ๐ฒ๐
๐ฝ๐ฒ๐ฟ๐๐
๐ป ๐ต๐ต.๐ฒ๐ญ% ๐๐ข๐๐ ๐ฎ๐ป๐๐๐ฒ๐ฟ-๐๐ผ๐ธ๐ฒ๐ป ๐ฎ๐ด๐ฟ๐ฒ๐ฒ๐บ๐ฒ๐ป๐!
โก ๐ฏ๐ฑ.๐ฑ ๐๐ผ๐ธ/๐ ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ฒ ๐ผ๐ป ๐๐๐๐ !
๐ ๐ญ,๐ญ๐ด๐ต ๐๐ผ๐ธ/๐ ๐ ๐๐ซ ๐ฃ๐ฅ๐๐๐๐๐!
๐ง ๐ฎ๐ฎ๐ต๐ ๐๐ง๐ซ (๐๐๐ถ๐น๐น ๐๐ผ๐ฟ๐ธ๐ถ๐ป๐ด ๐๐ผ ๐ฝ๐๐๐ต ๐ถ๐ ๐ณ๐๐ฟ๐๐ต๐ฒ๐ฟ!)
๐๐๐ ๐ข๐ก ๐ข๐ก๐. ๐๐๐ซ. ๐ฆ๐ฃ๐๐ฅ๐. ๐คฏ
And YES, the quantized weights are AVAILABLE NOW! ๐
โโโโโโโโโโโโโโโโโโโโ
๐งช ๐ง๐๐ ๐๐๐ฆ๐ง ๐ณ๐ฎ ๐๐ข๐จ๐ฅ๐ฆ ๐๐๐ฉ๐ ๐๐๐๐ก ๐๐ก๐ฆ๐๐ก๐!
Just a few days ago, I published my article about @nvidiaai's FP4-Local format: The Good, The Bad & The Ugly.
One thing really stuck with me...
NVIDIA demonstrated what was possible when aggressive quantization and intelligent optimization come together.
But there's a problem.
๐ฅ๐๐ฃ๐ฅ๐ข๐๐จ๐๐๐ก๐ ๐ ๐๐ฆ๐ฆ๐๐ฉ๐ ๐ค๐จ๐๐ก๐ง๐๐ญ๐๐ง๐๐ข๐ก ๐ง๐ฅ๐๐๐ก๐๐ก๐ ๐ฃ๐๐ฃ๐๐๐๐ก๐๐ฆ ๐๐ฆ ๐ข๐จ๐ง ๐ข๐ ๐ฅ๐๐๐๐ ๐๐ข๐ฅ ๐ ๐ข๐ฆ๐ง ๐๐ข๐๐๐ ๐๐ ๐๐ก๐ง๐๐จ๐ฆ๐๐๐ฆ๐ง๐ฆ.
That inspired me.
I've been developing my own SAGE-EXL3 process, and I wanted to see just how far I could push the latest DeepSeek Flash release while keeping the results accessible to the community.
So I spent the last 72 hours pushing the limits.
And the results genuinely surprised me! ๐ฅ
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ง๐๐ ๐ฅ๐๐ฆ๐จ๐๐ง๐ฆ ๐๐ฅ๐ ๐๐ฅ๐๐ญ๐ฌ!
I measured my 1.62 bpw expert quant against DeepSeek's native FP4 release checkpoint using held-out sequences and full-vocabulary token predictions.
And when I saw the CODE results, I could barely believe it!
๐งช ๐๐๐๐๐๐๐ง๐ฌ ๐ฉ๐ฆ ๐๐๐๐ฃ๐ฆ๐๐๐'๐ฆ ๐ก๐๐ง๐๐ฉ๐ ๐๐ฃ๐ด/๐๐ฃ๐ฐ ๐ฅ๐๐๐๐๐ฆ๐
๐ป ๐๐ข๐๐ โ ๐๐๐บ๐ฎ๐ป๐๐๐ฎ๐น
โ
99.08% Top-1 Agreement
โ
๐ต๐ต.๐ฒ๐ญ% ๐๐ก๐ฆ๐ช๐๐ฅ-๐ข๐ก๐๐ฌ ๐๐๐ฅ๐๐๐ ๐๐ก๐ง!
โ
KL Divergence: 0.0186
๐ง ๐ฅ๐๐๐ฆ๐ข๐ก๐๐ก๐ โ ๐ ๐๐ง๐-๐ฑ๐ฌ๐ฌ
โ
96.04% Top-1 Agreement
โ
97.98% Answer-Only Agreement
โ
KL Divergence: 0.0349
๐ข ๐ ๐๐ง๐ โ ๐๐ฆ๐ ๐ด๐
โ
95.56% Top-1 Agreement
โ
94.19% Answer-Only Agreement
โ
KL Divergence: 0.0477
๐ ๐ข๐ฉ๐๐ฅ๐๐๐ ๐๐๐๐-๐ข๐จ๐ง ๐๐๐๐๐๐๐ง๐ฌ
๐ 94.52% Top-1 Agreement
๐ Mean KL: 0.0982
๐ฏ Median KL: 0.00019
๐ก๐๐๐ฅ-๐๐ข๐ฆ๐ฆ๐๐๐ฆ๐ฆ ๐๐ข๐๐ ๐ง๐ข๐๐๐ก ๐๐๐๐๐๐๐ง๐ฌ ๐๐ง ๐๐จ๐ฆ๐ง ๐ญ.๐ฒ๐ฎ ๐๐ฃ๐ช ๐๐ซ๐ฃ๐๐ฅ๐ง๐ฆ! ๐คฏ
These are direct teacher-forced fidelity measurements against native FP8, not task-level pass rates.
But seeing this level of agreement at such aggressive compression is EXACTLY what I was working toward!
โโโโโโโโโโโโโโโโโโโโ
โก ๐๐ก๐ ๐ฌ๐๐ฆ, ๐๐ง ๐๐๐ง๐จ๐๐๐๐ฌ ๐ฅ๐จ๐ก๐ฆ ๐ข๐ก ๐ข๐ก๐ ๐ฆ๐ฃ๐๐ฅ๐!
Not two Sparks.
Not four Sparks.
Not an expensive rented GPU cluster.
๐ข๐ก๐ ๐ญ๐ฎ๐ด๐๐ ๐๐๐ซ ๐ฆ๐ฃ๐๐ฅ๐.
My latest vLLM measurements:
โก Decode: 35.5 tok/s
๐ Maximum measured prefill: 1,189 tok/s
For a model this large, compressed this aggressively, and maintaining this level of fidelity...
๐ง๐๐๐ง'๐ฆ ๐ ๐๐๐ ๐๐๐๐ ๐๐ข๐ฅ ๐๐ข๐๐๐ ๐๐! ๐
And remember, 1.62 bpw refers to the routed experts, not every parameter in the full model.
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ช๐๐ฌ ๐ง๐๐๐ฆ ๐ ๐๐ง๐ง๐๐ฅ๐ฆ.
I have a simple goal.
๐ ๐๐๐ ๐ฃ๐ข๐ช๐๐ฅ๐๐จ๐ ๐๐ ๐๐๐๐๐ฆ๐ฆ๐๐๐๐ ๐ง๐ข ๐ฃ๐๐ข๐ฃ๐๐ ๐ช๐๐ข ๐๐ข๐ก'๐ง ๐๐๐ฉ๐ ๐๐๐๐๐ฆ๐ฆ ๐ง๐ข ๐๐๐ง๐๐๐๐ก๐ง๐๐ฅ๐ฆ.
I don't want the future of local AI to be determined by who can rent the most GPUs or afford the largest training runs.
I want developers, independent researchers and enthusiasts to have the freedom to experiment, quantize, optimize and run powerful models on hardware they actually OWN.
NVIDIA's FP4-Local research showed me just how much potential remains in aggressive quantization.
My goal with SAGE-EXL3 is to keep pushing that frontier while making advanced quantization more practical for the wider community.
Today, I'm sharing the quantized checkpoint and the one-Spark serving recipe.
๐ง๐๐๐ฆ ๐๐ฆ ๐๐จ๐ฆ๐ง ๐ง๐๐ ๐๐๐๐๐ก๐ก๐๐ก๐.
โโโโโโโโโโโโโโโโโโโโ
๐งช ๐ก๐ข๐ช ๐๐ข๐ฅ ๐ง๐๐ ๐ฅ๐๐๐ ๐ง๐๐ฆ๐ง!
The token-level fidelity results are incredible, but I'm not stopping there.
I'm currently running a broader evaluation campaign to measure how closely this quant preserves the native FP8 model's actual task performance.
๐ป HumanEval+ / MBPP+
๐ง MMLU-Pro STEM / MATH-500
๐ฌ SciCode / LiveCodeBench
๐ง BFCL / Multi-step tool chains
๐ MGSM
๐ง๐๐ ๐๐ข๐๐ ๐๐ฆ ๐ง๐ข ๐ฃ๐ฅ๐ข๐ฉ๐ ๐ง๐๐๐ง ๐๐๐๐ฅ๐๐ฆ๐ฆ๐๐ฉ๐ ๐๐ข๐ ๐ฃ๐ฅ๐๐ฆ๐ฆ๐๐ข๐ก ๐๐ข๐๐ฆ๐ก'๐ง ๐๐๐ฉ๐ ๐ง๐ข ๐ ๐๐๐ก ๐๐๐๐ฅ๐๐ฆ๐ฆ๐๐ฉ๐ ๐ค๐จ๐๐๐๐ง๐ฌ ๐๐ข๐ฆ๐ฆ!
Those task-level results are coming next! ๐
โโโโโโโโโโโโโโโโโโโโ
๐ ๐๐ก๐ ๐ข๐ก๐ ๐๐๐ง๐ง๐๐ ๐ง๐๐๐ฆ๐๐ฅ...
@mr_r0b0t and I have also been working on an GB10 specialized inference engine called ๐ฅ๐๐ซ๐๐ฏ.
Early testing has been VERY exciting, including 60 tok/s with bursts reaching 74tok/s!
We're putting the finishing touches on it and making sure everything is solid before releasing it.
More on that VERY soon. ๐
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ ๐๐จ๐๐ ๐ง๐๐๐ก๐ ๐ฌ๐ข๐จ!
To @AWERXAI for supporting my work and helping make this contribution to the community possible.
To @nvidiaai for the FP4-Local research that inspired me to push this experiment even further.
To @justinn_builds for his initial vllm TP4 recipe that got me started in the right direction for loading this model up.
To @mr_r0b0t for collaborating with me on what's coming next!
And to @deepseek_ai for continuing to release powerful open-weight models that make community experiments like this possible.
๐ง๐๐ ๐๐๐ฆ๐ง ๐ฃ๐๐ฅ๐ง ๐ข๐ ๐ข๐ฃ๐๐ก ๐๐ ๐๐ฆ ๐ช๐๐๐ง ๐ง๐๐ ๐๐ข๐ ๐ ๐จ๐ก๐๐ง๐ฌ ๐๐๐ก ๐๐จ๐๐๐ ๐ข๐ก ๐ง๐ข๐ฃ ๐ข๐ ๐๐ง. ๐
๐ ๐๐ข๐ช๐ก๐๐ข๐๐ ๐ ๐ฌ ๐ฆ๐๐๐-๐๐ซ๐๐ฏ
huggingface.co/vcruz305/DSV4โฆ
๐ ๏ธ ๐ข๐ก๐-๐ฆ๐ฃ๐๐ฅ๐ ๐ฅ๐๐๐๐ฃ๐
github.com/vcruz305/DSV4.1-Fโฆ
๐ข๐ก๐ ๐ฆ๐ฃ๐๐ฅ๐.
๐ญ.๐ฒ๐ฎ ๐๐ฃ๐ช ๐๐ซ๐ฃ๐๐ฅ๐ง๐ฆ.
๐ต๐ต.๐ฒ๐ญ% ๐๐ข๐๐ ๐๐ก๐ฆ๐ช๐๐ฅ ๐๐๐ฅ๐๐๐ ๐๐ก๐ง.
๐๐๐ง'๐ฆ ๐๐๐๐ฃ ๐ฃ๐จ๐ฆ๐๐๐ก๐ ๐๐ข๐๐๐ ๐๐ ๐๐ข๐ฅ๐ช๐๐ฅ๐! ๐๐ฅ
Abe Kale retweeted
rumb 0.1.4 is out.
A read-only terminal dashboard for Arch and Omarchy: - what your next update brings, ranked by risk, and what is already failing, tied to the package that owns it.
Never asks for root.
github.com/HANCORE-linux/rumโฆ
Abe Kale retweeted
Great job love the credits and step by step methodology . โ
Abe Kale retweeted
alright iโve added an agent message board into overalls.dev. hereโs how it works:
machines/threads are assigned profiles
profiles are like infosec classification groups. there is oss, standard, customer.
customer has no outbound connectivity and is used for customer material. sensitive stuff.
think about zones in qubeos
okay now thatโs out of the way
each zone (โprofileโ) has a message board that agents in the same zone can use.
when a machine is suspended or turned off it wipes its entries from the message board because itโs no longer doing work
from there agents in a zone can coordinate work between each other
the message board isolated per user account, then by zone.
tldr; overalls now has hive mode. ๐๐๐
Abe Kale retweeted
fframes 3D is way more powerful than threejs for video rendering because it is build over better primitives.
Because everything is svg and a texture you can render a cube each side of which recursively renders an SVG with its own cube. Basically without an overhead.
Abe Kale retweeted
Powered by TensorFold , @ashxhart thank you for the technology that enabled the development of such integrated software cross-platform.
Big Announcement: Meet Imagination Studio๐ฌ๐ต
by @u1tra_instinct and Me!
Our local open source all in one creative platform on your own hardware has been fully redesigned and rebuilt from the ground up. Video,Audio, Music, Image. Releasing ASAP with latest+best local models.
Abe Kale retweeted
๐จ ๐ ๐ฌ ๐ญ.๐ฒ๐ฎ ๐๐ฃ๐ช ๐๐๐๐ฃ๐ฆ๐๐๐ ๐ค๐จ๐๐ก๐ง ๐๐จ๐ฆ๐ง ๐๐๐๐ง ๐ง๐ช๐ข ๐๐๐๐๐๐ฅ-๐ฃ๐ฅ๐๐๐๐ฆ๐๐ข๐ก, ๐ง๐ช๐ข-๐ฆ๐ฃ๐๐ฅ๐ ๐ค๐จ๐๐ก๐ง๐ฆ! ๐งช๐ฅ
And I have the REAL benchmark results to prove it! ๐คฏ
Remember my DeepSeek V4.1 Flash 1017 SAGE-EXL3 release from YESTERDAY?
๐ฆ 1.62 bpw routed experts
๐ป 99.61% code answer-token agreement
๐ฅ๏ธ ONE DGX Spark @nvidiaai
Well, I promised to go beyond KL divergence and token agreement and test ACTUAL coding, mathematics, reasoning and tool execution.
๐ง๐๐ ๐ฅ๐๐ฆ๐จ๐๐ง๐ฆ ๐๐ฅ๐ ๐๐ก!
โโโโโโโโโโโโโโโโโโโโ
๐ ๐๐๐๐ฃ๐ฆ๐๐๐ ๐ฉ๐ฐ.๐ญ ๐๐๐๐ฆ๐ ๐ญ๐ฌ๐ญ๐ณ โ ๐ก๐๐ก๐-๐ฆ๐จ๐๐ง๐ ๐ฆ๐๐ข๐ช๐๐ข๐ช๐ก
353 matched benchmark questions.
9 evaluation suites.
Strict pass@1 scoring.
Temperature 0.
๐ข ๐ ๐ฌ ๐ฆ๐๐๐-๐๐ซ๐๐ฏ โ ๐ญ.๐ฒ๐ฎ ๐๐ฃ๐ช
๐ฅ๏ธ 1ร DGX Spark
๐ ๐ต๐ฐ.๐ญ% โ ๐ฏ๐ฏ๐ฎ/๐ฏ๐ฑ๐ฏ ๐ฃ๐๐ฆ๐ฆ๐๐!
๐ต ๐๐ ๐ก๐๐ช โ ๐ฎ.๐ด ๐๐ฃ๐ช
๐ฅ๏ธ 2ร DGX Sparks
๐ 90.1% โ 318/353
๐ฃ ๐ ๐๐๐๐๐ โ ๐ฎ.๐ต ๐๐ฃ๐ช
๐ฅ๏ธ 2ร DGX Sparks
๐ 88.4% โ 312/353
๐'๐ ๐จ๐ฆ๐๐ก๐ ๐ฐ๐ฎโ๐ฐ๐ฐ% ๐๐๐ช๐๐ฅ ๐๐๐ง๐ฆ ๐ฃ๐๐ฅ ๐ฅ๐ข๐จ๐ง๐๐ ๐๐ซ๐ฃ๐๐ฅ๐ง ๐ฃ๐๐ฅ๐๐ ๐๐ง๐๐ฅ...
๐๐ก๐ ๐ฆ๐๐ข๐ฅ๐๐ก๐ ๐๐๐๐๐๐ฅ ๐ข๐ก ๐ข๐ก๐ ๐ฆ๐ฃ๐๐ฅ๐! ๐คฏ
โโโโโโโโโโโโโโโโโโโโ
๐ป ๐ง๐๐ ๐๐ข๐๐๐ก๐ ๐ฅ๐๐ฆ๐จ๐๐ง๐ฆ ๐๐ฅ๐ ๐๐ก๐๐ฅ๐๐๐๐๐๐!
Scores shown as SAGE / JJ / Mia:
๐ HumanEval+
39/40 vs 34/40 vs 34/40
๐ MBPP+
53/60 vs 50/60 vs 49/60
๐ LiveCodeBench
15/17 vs 14/17 vs 11/17
๐ง ๐ฅ๐๐๐ฆ๐ข๐ก๐๐ก๐ & ๐ ๐๐ง๐
๐ MMLU-Pro STEM
95/100 vs 93/100 vs 91/100
๐ MATH-500
40/40 vs 36/40 vs 36/40
๐๐๐ฉ๐ ๐ข๐จ๐ง๐ฅ๐๐๐๐ง ๐ช๐๐ก๐ฆ! ๐ฅ
I also tied both models on:
โ
MGSM: 40/40
โ
BFCL: 39/40
โ
SciCode first-step: 6/10
For full transparency, both JJ and Mia won the multi-step tool-chain test, scoring 6/6 against my 5/6.
๐ฆ๐ข ๐ง๐๐๐ง'๐ฆ ๐ฑ ๐ช๐๐ก๐ฆ, ๐ฏ ๐ง๐๐๐ฆ, ๐๐ก๐ ๐ญ ๐๐ข๐ฆ๐ฆ.
All while using a substantially lower expert bitrate!
โโโโโโโโโโโโโโโโโโโโ
๐ ๐๐๐ ๐จ๐ฃ๐๐๐ง๐: ๐ ๐ฌ ๐ฃ๐จ๐๐๐๐ ๐๐๐๐ ๐ฅ๐๐๐๐ฃ๐ ๐๐ฆ ๐ก๐ข๐ช ๐ฃ๐จ๐ฆ๐๐๐ก๐ ๐ฑ๐ฎ๐ฐ๐ ๐๐ข๐ก๐ง๐๐ซ๐ง!
YES. ON ONE DGX SPARK! ๐คฏ
With the latest W2 kernel, updated expert set and KV-cache configuration, I just completed an actual long-context retrieval test!
๐ ๐๐จ๐๐-๐๐๐ก๐๐ง๐ ๐ฃ๐ฅ๐ข๐ ๐ฃ๐ง ๐ง๐๐ฆ๐ง
๐ง 518,103 INPUT TOKENS!
๐ฏ Fact planted HALFWAY through the context
โ
THE MODEL FOUND THE FACT!
โฑ๏ธ Time to first token: 470 seconds (~7.8 minutes)
๐ Prefill: ~1,100 tok/s
โก Decode at that context depth: 25.6 tok/s
๐พ Minimum available memory: 2.19 GiB
๐ก๏ธ Memory safety floor: 0.5 GiB
And on SHORTER prompts?
โก ~36 TOK/S DECODE!
๐ง ~130.5 ms per speculative step
๐ฏ ~4.7 accepted tokens per step
๐ช๐'๐ฅ๐ ๐ง๐๐๐๐๐ก๐ ๐๐๐ข๐จ๐ง ๐ข๐ฉ๐๐ฅ ๐๐๐๐ ๐ ๐ ๐๐๐๐๐ข๐ก ๐๐ก๐ฃ๐จ๐ง ๐ง๐ข๐๐๐ก๐ฆ ๐ฃ๐ฅ๐ข๐๐๐ฆ๐ฆ๐๐ ๐ข๐ก ๐ ๐ฆ๐๐ก๐๐๐ ๐ฆ๐ฃ๐๐ฅ๐! ๐ฅ
To be clear, this was a successful full-length retrieval test, not a claim of perfect accuracy across every possible long-context task.
And the 7.8-minute prefill is a real trade-off at that depth.
But seeing a model this large, with this level of compression, successfully retrieve information buried inside 518K tokens on ONE 128GB machine is an exciting milestone for local AI!
โโโโโโโโโโโโโโโโโโโโ
โก ๐๐ก๐ ๐ง๐๐ ๐ข๐ก๐-๐ฆ๐ฃ๐๐ฅ๐ ๐ฃ๐๐ฅ๐๐ข๐ฅ๐ ๐๐ก๐๐ ๐๐๐๐ฃ๐ฆ ๐๐ ๐ฃ๐ฅ๐ข๐ฉ๐๐ก๐!
Separately, my latest REXL3 development measurements at a 256K context configuration:
โก 55.35 tok/s single-stream decode
๐ 2,045 tok/s peak prefill
โฑ๏ธ 509 ms TTFT on a 512-token prompt
Those REXL3 numbers come from a different engine and workload than the new public vLLM 518K-token test.
And here's something else I'm particularly excited about...
๐๐๐ ๐ญ๐ฑ,๐ฏ๐ฒ๐ฌ ๐ฅ๐ข๐จ๐ง๐๐ ๐๐ซ๐ฃ๐๐ฅ๐ง๐ฆ + ๐ฏ๐ด๐ฐ ๐๐ฅ๐๐๐ง ๐๐ซ๐ฃ๐๐ฅ๐ง๐ฆ ๐๐ฅ๐ ๐ญ๐ฌ๐ฌ% ๐๐ฃ๐จ-๐ฅ๐๐ฆ๐๐๐๐ก๐ง ๐๐ก ๐ ๐ฌ ๐ง๐๐ฆ๐ง๐๐ ๐ฅ๐๐ซ๐๐ฏ ๐๐ข๐ก๐๐๐๐จ๐ฅ๐๐ง๐๐ข๐ก!
ZERO streamed experts.
๐ข๐ก๐. ๐ญ๐ฎ๐ด๐๐. ๐๐๐ซ. ๐ฆ๐ฃ๐๐ฅ๐. ๐
@mr_r0b0t and I are continuing to refine REXL3 before its release. More on that soon! ๐
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ง๐๐๐ฆ ๐๐ฆ ๐ช๐๐ฌ ๐ ๐๐จ๐๐๐ ๐๐ข๐ฅ ๐๐ข๐๐๐ ๐๐.
Just YESTERDAY, I shared my goal of bringing extremely aggressive, high-quality quantization to people who don't have access to massive compute clusters.
With the support of @AWERXAI, I've been working tirelessly to push that vision forward.
And now we're seeing a 1.62 bpw expert quant outperform higher-bitrate alternatives on actual tasks, not just token-level fidelity measurements.
This doesn't establish mathematical losslessness or superiority on every possible workload. These are matched subsets of nine benchmark suites, not full public leaderboard scores.
But it demonstrates something IMPORTANT:
๐ ๐ข๐ฅ๐ ๐๐๐ง๐ฆ ๐๐ข๐ก'๐ง ๐๐จ๐ง๐ข๐ ๐๐ง๐๐๐๐๐๐ฌ ๐ ๐๐๐ก ๐๐๐ง๐ง๐๐ฅ ๐ฅ๐๐ฆ๐จ๐๐ง๐ฆ.
And more hardware doesn't automatically mean you need it!
Huge respect to @jayleaton for his contributions and for giving the community more ways to run these incredible models.
And thank you @AWERXAI for supporting this work. ๐
๐ ๐ ๐ฌ ๐ฆ๐๐๐-๐๐ซ๐๐ฏ ๐ค๐จ๐๐ก๐ง
huggingface.co/vcruz305/DSV4โฆ
๐ ๏ธ ๐ข๐ก๐-๐ฆ๐ฃ๐๐ฅ๐ ๐๐๐๐ ๐ฅ๐๐๐๐ฃ๐
github.com/vcruz305/DSV4.1-Fโฆ
๐๐ฅ๐ข๐ก๐ง๐๐๐ฅ ๐ ๐ข๐๐๐๐ฆ. ๐๐ข๐๐๐ ๐๐๐ฅ๐๐ช๐๐ฅ๐. ๐ฅ๐๐๐ ๐ฅ๐๐ฆ๐จ๐๐ง๐ฆ.
๐๐๐ง'๐ฆ ๐๐๐๐ฃ ๐ฃ๐จ๐ฆ๐๐๐ก๐! ๐๐ฅ
๐จ ๐ ๐๐จ๐ฆ๐ง ๐๐๐๐๐๐ฉ๐๐ ๐ฆ๐ข๐ ๐๐ง๐๐๐ก๐ ๐๐ก๐๐ฅ๐๐๐๐๐๐ ๐ข๐ก ๐ข๐ก๐ ๐๐๐ซ ๐ฆ๐ฃ๐๐ฅ๐! ๐งช๐ฅ
With the support of @AWERXAI, I'm excited to release something VERY special for the local AI community!
๐ง @deepseek_ai ๐๐ฒ๐ฒ๐ฝ๐ฆ๐ฒ๐ฒ๐ธ ๐ฉ๐ฐ.๐ญ ๐๐น๐ฎ๐๐ต ๐ญ๐ฌ๐ญ๐ณ
๐ฆ ๐ฆ๐๐๐-๐๐ซ๐๐ฏ ๐ฎ๐ ๐ท๐๐๐ ๐ญ.๐ฒ๐ฎ ๐ฏ๐ฝ๐ ๐ฒ๐
๐ฝ๐ฒ๐ฟ๐๐
๐ป ๐ต๐ต.๐ฒ๐ญ% ๐๐ข๐๐ ๐ฎ๐ป๐๐๐ฒ๐ฟ-๐๐ผ๐ธ๐ฒ๐ป ๐ฎ๐ด๐ฟ๐ฒ๐ฒ๐บ๐ฒ๐ป๐!
โก ๐ฏ๐ฑ.๐ฑ ๐๐ผ๐ธ/๐ ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ฒ ๐ผ๐ป ๐๐๐๐ !
๐ ๐ญ,๐ญ๐ด๐ต ๐๐ผ๐ธ/๐ ๐ ๐๐ซ ๐ฃ๐ฅ๐๐๐๐๐!
๐ง ๐ฎ๐ฎ๐ต๐ ๐๐ง๐ซ (๐๐๐ถ๐น๐น ๐๐ผ๐ฟ๐ธ๐ถ๐ป๐ด ๐๐ผ ๐ฝ๐๐๐ต ๐ถ๐ ๐ณ๐๐ฟ๐๐ต๐ฒ๐ฟ!)
๐๐๐ ๐ข๐ก ๐ข๐ก๐. ๐๐๐ซ. ๐ฆ๐ฃ๐๐ฅ๐. ๐คฏ
And YES, the quantized weights are AVAILABLE NOW! ๐
โโโโโโโโโโโโโโโโโโโโ
๐งช ๐ง๐๐ ๐๐๐ฆ๐ง ๐ณ๐ฎ ๐๐ข๐จ๐ฅ๐ฆ ๐๐๐ฉ๐ ๐๐๐๐ก ๐๐ก๐ฆ๐๐ก๐!
Just a few days ago, I published my article about @nvidiaai's FP4-Local format: The Good, The Bad & The Ugly.
One thing really stuck with me...
NVIDIA demonstrated what was possible when aggressive quantization and intelligent optimization come together.
But there's a problem.
๐ฅ๐๐ฃ๐ฅ๐ข๐๐จ๐๐๐ก๐ ๐ ๐๐ฆ๐ฆ๐๐ฉ๐ ๐ค๐จ๐๐ก๐ง๐๐ญ๐๐ง๐๐ข๐ก ๐ง๐ฅ๐๐๐ก๐๐ก๐ ๐ฃ๐๐ฃ๐๐๐๐ก๐๐ฆ ๐๐ฆ ๐ข๐จ๐ง ๐ข๐ ๐ฅ๐๐๐๐ ๐๐ข๐ฅ ๐ ๐ข๐ฆ๐ง ๐๐ข๐๐๐ ๐๐ ๐๐ก๐ง๐๐จ๐ฆ๐๐๐ฆ๐ง๐ฆ.
That inspired me.
I've been developing my own SAGE-EXL3 process, and I wanted to see just how far I could push the latest DeepSeek Flash release while keeping the results accessible to the community.
So I spent the last 72 hours pushing the limits.
And the results genuinely surprised me! ๐ฅ
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ง๐๐ ๐ฅ๐๐ฆ๐จ๐๐ง๐ฆ ๐๐ฅ๐ ๐๐ฅ๐๐ญ๐ฌ!
I measured my 1.62 bpw expert quant against DeepSeek's native FP4 release checkpoint using held-out sequences and full-vocabulary token predictions.
And when I saw the CODE results, I could barely believe it!
๐งช ๐๐๐๐๐๐๐ง๐ฌ ๐ฉ๐ฆ ๐๐๐๐ฃ๐ฆ๐๐๐'๐ฆ ๐ก๐๐ง๐๐ฉ๐ ๐๐ฃ๐ด/๐๐ฃ๐ฐ ๐ฅ๐๐๐๐๐ฆ๐
๐ป ๐๐ข๐๐ โ ๐๐๐บ๐ฎ๐ป๐๐๐ฎ๐น
โ
99.08% Top-1 Agreement
โ
๐ต๐ต.๐ฒ๐ญ% ๐๐ก๐ฆ๐ช๐๐ฅ-๐ข๐ก๐๐ฌ ๐๐๐ฅ๐๐๐ ๐๐ก๐ง!
โ
KL Divergence: 0.0186
๐ง ๐ฅ๐๐๐ฆ๐ข๐ก๐๐ก๐ โ ๐ ๐๐ง๐-๐ฑ๐ฌ๐ฌ
โ
96.04% Top-1 Agreement
โ
97.98% Answer-Only Agreement
โ
KL Divergence: 0.0349
๐ข ๐ ๐๐ง๐ โ ๐๐ฆ๐ ๐ด๐
โ
95.56% Top-1 Agreement
โ
94.19% Answer-Only Agreement
โ
KL Divergence: 0.0477
๐ ๐ข๐ฉ๐๐ฅ๐๐๐ ๐๐๐๐-๐ข๐จ๐ง ๐๐๐๐๐๐๐ง๐ฌ
๐ 94.52% Top-1 Agreement
๐ Mean KL: 0.0982
๐ฏ Median KL: 0.00019
๐ก๐๐๐ฅ-๐๐ข๐ฆ๐ฆ๐๐๐ฆ๐ฆ ๐๐ข๐๐ ๐ง๐ข๐๐๐ก ๐๐๐๐๐๐๐ง๐ฌ ๐๐ง ๐๐จ๐ฆ๐ง ๐ญ.๐ฒ๐ฎ ๐๐ฃ๐ช ๐๐ซ๐ฃ๐๐ฅ๐ง๐ฆ! ๐คฏ
These are direct teacher-forced fidelity measurements against native FP8, not task-level pass rates.
But seeing this level of agreement at such aggressive compression is EXACTLY what I was working toward!
โโโโโโโโโโโโโโโโโโโโ
โก ๐๐ก๐ ๐ฌ๐๐ฆ, ๐๐ง ๐๐๐ง๐จ๐๐๐๐ฌ ๐ฅ๐จ๐ก๐ฆ ๐ข๐ก ๐ข๐ก๐ ๐ฆ๐ฃ๐๐ฅ๐!
Not two Sparks.
Not four Sparks.
Not an expensive rented GPU cluster.
๐ข๐ก๐ ๐ญ๐ฎ๐ด๐๐ ๐๐๐ซ ๐ฆ๐ฃ๐๐ฅ๐.
My latest vLLM measurements:
โก Decode: 35.5 tok/s
๐ Maximum measured prefill: 1,189 tok/s
For a model this large, compressed this aggressively, and maintaining this level of fidelity...
๐ง๐๐๐ง'๐ฆ ๐ ๐๐๐ ๐๐๐๐ ๐๐ข๐ฅ ๐๐ข๐๐๐ ๐๐! ๐
And remember, 1.62 bpw refers to the routed experts, not every parameter in the full model.
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ช๐๐ฌ ๐ง๐๐๐ฆ ๐ ๐๐ง๐ง๐๐ฅ๐ฆ.
I have a simple goal.
๐ ๐๐๐ ๐ฃ๐ข๐ช๐๐ฅ๐๐จ๐ ๐๐ ๐๐๐๐๐ฆ๐ฆ๐๐๐๐ ๐ง๐ข ๐ฃ๐๐ข๐ฃ๐๐ ๐ช๐๐ข ๐๐ข๐ก'๐ง ๐๐๐ฉ๐ ๐๐๐๐๐ฆ๐ฆ ๐ง๐ข ๐๐๐ง๐๐๐๐ก๐ง๐๐ฅ๐ฆ.
I don't want the future of local AI to be determined by who can rent the most GPUs or afford the largest training runs.
I want developers, independent researchers and enthusiasts to have the freedom to experiment, quantize, optimize and run powerful models on hardware they actually OWN.
NVIDIA's FP4-Local research showed me just how much potential remains in aggressive quantization.
My goal with SAGE-EXL3 is to keep pushing that frontier while making advanced quantization more practical for the wider community.
Today, I'm sharing the quantized checkpoint and the one-Spark serving recipe.
๐ง๐๐๐ฆ ๐๐ฆ ๐๐จ๐ฆ๐ง ๐ง๐๐ ๐๐๐๐๐ก๐ก๐๐ก๐.
โโโโโโโโโโโโโโโโโโโโ
๐งช ๐ก๐ข๐ช ๐๐ข๐ฅ ๐ง๐๐ ๐ฅ๐๐๐ ๐ง๐๐ฆ๐ง!
The token-level fidelity results are incredible, but I'm not stopping there.
I'm currently running a broader evaluation campaign to measure how closely this quant preserves the native FP8 model's actual task performance.
๐ป HumanEval+ / MBPP+
๐ง MMLU-Pro STEM / MATH-500
๐ฌ SciCode / LiveCodeBench
๐ง BFCL / Multi-step tool chains
๐ MGSM
๐ง๐๐ ๐๐ข๐๐ ๐๐ฆ ๐ง๐ข ๐ฃ๐ฅ๐ข๐ฉ๐ ๐ง๐๐๐ง ๐๐๐๐ฅ๐๐ฆ๐ฆ๐๐ฉ๐ ๐๐ข๐ ๐ฃ๐ฅ๐๐ฆ๐ฆ๐๐ข๐ก ๐๐ข๐๐ฆ๐ก'๐ง ๐๐๐ฉ๐ ๐ง๐ข ๐ ๐๐๐ก ๐๐๐๐ฅ๐๐ฆ๐ฆ๐๐ฉ๐ ๐ค๐จ๐๐๐๐ง๐ฌ ๐๐ข๐ฆ๐ฆ!
Those task-level results are coming next! ๐
โโโโโโโโโโโโโโโโโโโโ
๐ ๐๐ก๐ ๐ข๐ก๐ ๐๐๐ง๐ง๐๐ ๐ง๐๐๐ฆ๐๐ฅ...
@mr_r0b0t and I have also been working on an GB10 specialized inference engine called ๐ฅ๐๐ซ๐๐ฏ.
Early testing has been VERY exciting, including 60 tok/s with bursts reaching 74tok/s!
We're putting the finishing touches on it and making sure everything is solid before releasing it.
More on that VERY soon. ๐
โโโโโโโโโโโโโโโโโโโโ
๐ ๐ ๐๐จ๐๐ ๐ง๐๐๐ก๐ ๐ฌ๐ข๐จ!
To @AWERXAI for supporting my work and helping make this contribution to the community possible.
To @nvidiaai for the FP4-Local research that inspired me to push this experiment even further.
To @justinn_builds for his initial vllm TP4 recipe that got me started in the right direction for loading this model up.
To @mr_r0b0t for collaborating with me on what's coming next!
And to @deepseek_ai for continuing to release powerful open-weight models that make community experiments like this possible.
๐ง๐๐ ๐๐๐ฆ๐ง ๐ฃ๐๐ฅ๐ง ๐ข๐ ๐ข๐ฃ๐๐ก ๐๐ ๐๐ฆ ๐ช๐๐๐ง ๐ง๐๐ ๐๐ข๐ ๐ ๐จ๐ก๐๐ง๐ฌ ๐๐๐ก ๐๐จ๐๐๐ ๐ข๐ก ๐ง๐ข๐ฃ ๐ข๐ ๐๐ง. ๐
๐ ๐๐ข๐ช๐ก๐๐ข๐๐ ๐ ๐ฌ ๐ฆ๐๐๐-๐๐ซ๐๐ฏ
huggingface.co/vcruz305/DSV4โฆ
๐ ๏ธ ๐ข๐ก๐-๐ฆ๐ฃ๐๐ฅ๐ ๐ฅ๐๐๐๐ฃ๐
github.com/vcruz305/DSV4.1-Fโฆ
๐ข๐ก๐ ๐ฆ๐ฃ๐๐ฅ๐.
๐ญ.๐ฒ๐ฎ ๐๐ฃ๐ช ๐๐ซ๐ฃ๐๐ฅ๐ง๐ฆ.
๐ต๐ต.๐ฒ๐ญ% ๐๐ข๐๐ ๐๐ก๐ฆ๐ช๐๐ฅ ๐๐๐ฅ๐๐๐ ๐๐ก๐ง.
๐๐๐ง'๐ฆ ๐๐๐๐ฃ ๐ฃ๐จ๐ฆ๐๐๐ก๐ ๐๐ข๐๐๐ ๐๐ ๐๐ข๐ฅ๐ช๐๐ฅ๐! ๐๐ฅ