Cruz

🚨 𝗜 𝗝𝗨𝗦𝗧 𝗔𝗖𝗛𝗜𝗘𝗩𝗘𝗗 𝗦𝗢𝗠𝗘𝗧𝗛𝗜𝗡𝗚 𝗜𝗡𝗖𝗥𝗘𝗗𝗜𝗕𝗟𝗘 𝗢𝗡 𝗢𝗡𝗘 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞! 🧪🔥 With the support of @AWERXAI, I'm excited to release something VERY special for the local AI community! 🧠 @deepseek_ai 𝗗𝗲𝗲𝗽𝗦𝗲𝗲𝗸 𝗩𝟰.𝟭 𝗙𝗹𝗮𝘀𝗵 𝟭𝟬𝟭𝟳 📦 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯 𝗮𝘁 𝗷𝘂𝘀𝘁 𝟭.𝟲𝟮 𝗯𝗽𝘄 𝗲𝘅𝗽𝗲𝗿𝘁𝘀 💻 𝟵𝟵.𝟲𝟭% 𝗖𝗢𝗗𝗘 𝗮𝗻𝘀𝘄𝗲𝗿-𝘁𝗼𝗸𝗲𝗻 𝗮𝗴𝗿𝗲𝗲𝗺𝗲𝗻𝘁! ⚡ 𝟯𝟱.𝟱 𝘁𝗼𝗸/𝘀 𝗱𝗲𝗰𝗼𝗱𝗲 𝗼𝗻 𝘃𝗟𝗟𝗠! 🚀 𝟭,𝟭𝟴𝟵 𝘁𝗼𝗸/𝘀 𝗠𝗔𝗫 𝗣𝗥𝗘𝗙𝗜𝗟𝗟! 🧠 𝟮𝟮𝟵𝗞 𝗖𝗧𝗫 (𝘀𝘁𝗶𝗹𝗹 𝘄𝗼𝗿𝗸𝗶𝗻𝗴 𝘁𝗼 𝗽𝘂𝘀𝗵 𝗶𝘁 𝗳𝘂𝗿𝘁𝗵𝗲𝗿!) 𝗔𝗟𝗟 𝗢𝗡 𝗢𝗡𝗘. 𝗗𝗚𝗫. 𝗦𝗣𝗔𝗥𝗞. 🤯 And YES, the quantized weights are AVAILABLE NOW! 💚 ━━━━━━━━━━━━━━━━━━━━ 🧪 𝗧𝗛𝗘 𝗟𝗔𝗦𝗧 𝟳𝟮 𝗛𝗢𝗨𝗥𝗦 𝗛𝗔𝗩𝗘 𝗕𝗘𝗘𝗡 𝗜𝗡𝗦𝗔𝗡𝗘! Just a few days ago, I published my article about @nvidiaai's FP4-Local format: The Good, The Bad & The Ugly. One thing really stuck with me... NVIDIA demonstrated what was possible when aggressive quantization and intelligent optimization come together. But there's a problem. 𝗥𝗘𝗣𝗥𝗢𝗗𝗨𝗖𝗜𝗡𝗚 𝗠𝗔𝗦𝗦𝗜𝗩𝗘 𝗤𝗨𝗔𝗡𝗧𝗜𝗭𝗔𝗧𝗜𝗢𝗡 𝗧𝗥𝗔𝗜𝗡𝗜𝗡𝗚 𝗣𝗜𝗣𝗘𝗟𝗜𝗡𝗘𝗦 𝗜𝗦 𝗢𝗨𝗧 𝗢𝗙 𝗥𝗘𝗔𝗖𝗛 𝗙𝗢𝗥 𝗠𝗢𝗦𝗧 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜 𝗘𝗡𝗧𝗛𝗨𝗦𝗜𝗔𝗦𝗧𝗦. That inspired me. I've been developing my own SAGE-EXL3 process, and I wanted to see just how far I could push the latest DeepSeek Flash release while keeping the results accessible to the community. So I spent the last 72 hours pushing the limits. And the results genuinely surprised me! 🔥 ━━━━━━━━━━━━━━━━━━━━ 🏆 𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 𝗔𝗥𝗘 𝗖𝗥𝗔𝗭𝗬! I measured my 1.62 bpw expert quant against DeepSeek's native FP4 release checkpoint using held-out sequences and full-vocabulary token predictions. And when I saw the CODE results, I could barely believe it! 🧪 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗩𝗦 𝗗𝗘𝗘𝗣𝗦𝗘𝗘𝗞'𝗦 𝗡𝗔𝗧𝗜𝗩𝗘 𝗙𝗣𝟴/𝗙𝗣𝟰 𝗥𝗘𝗟𝗘𝗔𝗦𝗘 💻 𝗖𝗢𝗗𝗘 — 𝗛𝘂𝗺𝗮𝗻𝗘𝘃𝗮𝗹 ✅ 99.08% Top-1 Agreement ✅ 𝟵𝟵.𝟲𝟭% 𝗔𝗡𝗦𝗪𝗘𝗥-𝗢𝗡𝗟𝗬 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧! ✅ KL Divergence: 0.0186 🧠 𝗥𝗘𝗔𝗦𝗢𝗡𝗜𝗡𝗚 — 𝗠𝗔𝗧𝗛-𝟱𝟬𝟬 ✅ 96.04% Top-1 Agreement ✅ 97.98% Answer-Only Agreement ✅ KL Divergence: 0.0349 🔢 𝗠𝗔𝗧𝗛 — 𝗚𝗦𝗠𝟴𝗞 ✅ 95.56% Top-1 Agreement ✅ 94.19% Answer-Only Agreement ✅ KL Divergence: 0.0477 📊 𝗢𝗩𝗘𝗥𝗔𝗟𝗟 𝗛𝗘𝗟𝗗-𝗢𝗨𝗧 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 🏆 94.52% Top-1 Agreement 📉 Mean KL: 0.0982 🎯 Median KL: 0.00019 𝗡𝗘𝗔𝗥-𝗟𝗢𝗦𝗦𝗟𝗘𝗦𝗦 𝗖𝗢𝗗𝗘 𝗧𝗢𝗞𝗘𝗡 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗔𝗧 𝗝𝗨𝗦𝗧 𝟭.𝟲𝟮 𝗕𝗣𝗪 𝗘𝗫𝗣𝗘𝗥𝗧𝗦! 🤯 These are direct teacher-forced fidelity measurements against native FP8, not task-level pass rates. But seeing this level of agreement at such aggressive compression is EXACTLY what I was working toward! ━━━━━━━━━━━━━━━━━━━━ ⚡ 𝗔𝗡𝗗 𝗬𝗘𝗦, 𝗜𝗧 𝗔𝗖𝗧𝗨𝗔𝗟𝗟𝗬 𝗥𝗨𝗡𝗦 𝗢𝗡 𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞! Not two Sparks. Not four Sparks. Not an expensive rented GPU cluster. 𝗢𝗡𝗘 𝟭𝟮𝟴𝗚𝗕 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞. My latest vLLM measurements: ⚡ Decode: 35.5 tok/s 🚀 Maximum measured prefill: 1,189 tok/s For a model this large, compressed this aggressively, and maintaining this level of fidelity... 𝗧𝗛𝗔𝗧'𝗦 𝗔 𝗕𝗜𝗚 𝗗𝗘𝗔𝗟 𝗙𝗢𝗥 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜! 💚 And remember, 1.62 bpw refers to the routed experts, not every parameter in the full model. ━━━━━━━━━━━━━━━━━━━━ 💚 𝗪𝗛𝗬 𝗧𝗛𝗜𝗦 𝗠𝗔𝗧𝗧𝗘𝗥𝗦. I have a simple goal. 𝗠𝗔𝗞𝗘 𝗣𝗢𝗪𝗘𝗥𝗙𝗨𝗟 𝗔𝗜 𝗔𝗖𝗖𝗘𝗦𝗦𝗜𝗕𝗟𝗘 𝗧𝗢 𝗣𝗘𝗢𝗣𝗟𝗘 𝗪𝗛𝗢 𝗗𝗢𝗡'𝗧 𝗛𝗔𝗩𝗘 𝗔𝗖𝗖𝗘𝗦𝗦 𝗧𝗢 𝗗𝗔𝗧𝗔𝗖𝗘𝗡𝗧𝗘𝗥𝗦. I don't want the future of local AI to be determined by who can rent the most GPUs or afford the largest training runs. I want developers, independent researchers and enthusiasts to have the freedom to experiment, quantize, optimize and run powerful models on hardware they actually OWN. NVIDIA's FP4-Local research showed me just how much potential remains in aggressive quantization. My goal with SAGE-EXL3 is to keep pushing that frontier while making advanced quantization more practical for the wider community. Today, I'm sharing the quantized checkpoint and the one-Spark serving recipe. 𝗧𝗛𝗜𝗦 𝗜𝗦 𝗝𝗨𝗦𝗧 𝗧𝗛𝗘 𝗕𝗘𝗚𝗜𝗡𝗡𝗜𝗡𝗚. ━━━━━━━━━━━━━━━━━━━━ 🧪 𝗡𝗢𝗪 𝗙𝗢𝗥 𝗧𝗛𝗘 𝗥𝗘𝗔𝗟 𝗧𝗘𝗦𝗧! The token-level fidelity results are incredible, but I'm not stopping there. I'm currently running a broader evaluation campaign to measure how closely this quant preserves the native FP8 model's actual task performance. 💻 HumanEval+ / MBPP+ 🧠 MMLU-Pro STEM / MATH-500 🔬 SciCode / LiveCodeBench 🔧 BFCL / Multi-step tool chains 🌍 MGSM 𝗧𝗛𝗘 𝗚𝗢𝗔𝗟 𝗜𝗦 𝗧𝗢 𝗣𝗥𝗢𝗩𝗘 𝗧𝗛𝗔𝗧 𝗔𝗚𝗚𝗥𝗘𝗦𝗦𝗜𝗩𝗘 𝗖𝗢𝗠𝗣𝗥𝗘𝗦𝗦𝗜𝗢𝗡 𝗗𝗢𝗘𝗦𝗡'𝗧 𝗛𝗔𝗩𝗘 𝗧𝗢 𝗠𝗘𝗔𝗡 𝗔𝗚𝗚𝗥𝗘𝗦𝗦𝗜𝗩𝗘 𝗤𝗨𝗔𝗟𝗜𝗧𝗬 𝗟𝗢𝗦𝗦! Those task-level results are coming next! 👀 ━━━━━━━━━━━━━━━━━━━━ 👀 𝗔𝗡𝗗 𝗢𝗡𝗘 𝗟𝗜𝗧𝗧𝗟𝗘 𝗧𝗘𝗔𝗦𝗘𝗥... @mr_r0b0t and I have also been working on an GB10 specialized inference engine called 𝗥𝗘𝗫𝗟𝟯. Early testing has been VERY exciting, including 60 tok/s with bursts reaching 74tok/s! We're putting the finishing touches on it and making sure everything is solid before releasing it. More on that VERY soon. 😏 ━━━━━━━━━━━━━━━━━━━━ 🙏 𝗔 𝗛𝗨𝗚𝗘 𝗧𝗛𝗔𝗡𝗞 𝗬𝗢𝗨! To @AWERXAI for supporting my work and helping make this contribution to the community possible. To @nvidiaai for the FP4-Local research that inspired me to push this experiment even further. To @justinn_builds for his initial vllm TP4 recipe that got me started in the right direction for loading this model up. To @mr_r0b0t for collaborating with me on what's coming next! And to @deepseek_ai for continuing to release powerful open-weight models that make community experiments like this possible. 𝗧𝗛𝗘 𝗕𝗘𝗦𝗧 𝗣𝗔𝗥𝗧 𝗢𝗙 𝗢𝗣𝗘𝗡 𝗔𝗜 𝗜𝗦 𝗪𝗛𝗔𝗧 𝗧𝗛𝗘 𝗖𝗢𝗠𝗠𝗨𝗡𝗜𝗧𝗬 𝗖𝗔𝗡 𝗕𝗨𝗜𝗟𝗗 𝗢𝗡 𝗧𝗢𝗣 𝗢𝗙 𝗜𝗧. 💚 🔗 𝗗𝗢𝗪𝗡𝗟𝗢𝗔𝗗 𝗠𝗬 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯 huggingface.co/vcruz305/DSV4… 🛠️ 𝗢𝗡𝗘-𝗦𝗣𝗔𝗥𝗞 𝗥𝗘𝗖𝗜𝗣𝗘 github.com/vcruz305/DSV4.1-F… 𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞. 𝟭.𝟲𝟮 𝗕𝗣𝗪 𝗘𝗫𝗣𝗘𝗥𝗧𝗦. 𝟵𝟵.𝟲𝟭% 𝗖𝗢𝗗𝗘 𝗔𝗡𝗦𝗪𝗘𝗥 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧. 𝗟𝗘𝗧'𝗦 𝗞𝗘𝗘𝗣 𝗣𝗨𝗦𝗛𝗜𝗡𝗚 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜 𝗙𝗢𝗥𝗪𝗔𝗥𝗗! 🚀🔥

A lot of people in this community were genuinely shocked when they learned this: I do not own a DGX Spark. Much of the Spark-specific work I’ve released, including day-zero EXL3 and GGUF quants, vllm-exl3 development, custom kernels, SixCat evaluations, and reproducible serving recipes, has only been possible because incredible people in this community have loaned me access to their own hardware and clusters. I’m deeply grateful for that support. But borrowed access is temporary and unpredictable. I often have a new model, fix, or benchmark ready, then have to wait until someone else’s machine is available before I can validate and publish it properly. So I’m raising **$5,000 toward a DGX Spark of my own.** 🙏 𝗪𝗛𝗔𝗧 𝗔 𝗗𝗘𝗗𝗜𝗖𝗔𝗧𝗘𝗗 𝗦𝗣𝗔𝗥𝗞 𝗪𝗢𝗨𝗟𝗗 𝗘𝗡𝗔𝗕𝗟𝗘 → More true day-zero quantized model releases → Immediate hardware validation instead of projected results → Continuous vllm-exl3 and ExLlamaV3 kernel development → SixCat quality, speed, concurrency, and long-context benchmarks → Reproducible one-Spark recipes the community can run themselves → Faster debugging when new models or inference engines break It would turn a lot of: “I built it, but I need someone with a Spark to test it” into: “I built it, tested it, documented it, and released the recipe.” 𝗠𝗬 𝗣𝗨𝗕𝗟𝗜𝗖 𝗪𝗢𝗥𝗞 𝗦𝗧𝗔𝗬𝗦 𝗣𝗨𝗕𝗟𝗜𝗖 This is not an access fee, preorder, or paywall. The models, recipes, benchmarks, fixes, and tools will remain public. This is completely optional support for the hardware behind that work. I’m planning to place the order within 48 hours and will personally cover whatever the fundraiser does not. For complete transparency, I’ll publish: → The purchase receipt with private details removed → Photos of the actual DGX Spark → Total community contributions → Platform and payment fees → Final purchase cost → The amount I contributed myself To everyone who has loaned me hardware, funded compute, tested a build, submitted a PR, shared my work, or simply encouraged me: thank you. I genuinely could not have pushed this work as far as I have without this community. A contribution would mean a lot. A repost helps just as much. ❤️ ko-fi.com/victorangelcruz/go…

Qwen3.8-Flash-Next now reaches ~43 tok/s after a 122,902-token prompt on ONE DGX Spark. ⚡🚀 MTP k=2 won my draft-depth sweep, with +42.5% mean decode over no draft. The PLE table stays fully on-device. I promised the deeper MTP tests. Here are the results, and now you can explore them in an interactive benchmark page too. 𝗧𝗪𝗢 𝗗𝗥𝗔𝗙𝗧 𝗧𝗢𝗞𝗘𝗡𝗦 𝗪𝗢𝗡 Mean single-request decode with 32K context configured: MTP k=2: 39.21 tok/s MTP k=3: 36.42 tok/s MTP k=1: 35.18 tok/s No draft: 27.51 tok/s k=2 also produced the fastest individual sweep run: 41.34 tok/s. Four runs each for no draft, k=1 and k=2. Seven for k=3. Decode excludes time to first token. Here, k means speculative draft depth, not quantization bits. k=3 produced more tokens per step, but the extra drafting work did not pay off in throughput. k=2 is my current pick for this setup. 𝗧𝗛𝗘 𝟭𝟮𝟯𝗞-𝗧𝗢𝗞𝗘𝗡 𝗣𝗥𝗢𝗠𝗣𝗧 𝗧𝗘𝗦𝗧 I then ran a separate long-prompt comparison: Actual input: 122,902 tokens Configured context: 262,144 Requested output: 128 tokens One request at a time MTP k=2: ~43 tok/s No draft: 26.2 tok/s Time to first token: 110.6 seconds with MTP 107.0 seconds without it The win here is generation speed, not faster prefill. To keep the scope clear: 256K was the configured limit. This was a real ~123K input, not a completely filled 256K window or a full k sweep at that depth. 𝗣𝗟𝗘 𝗦𝗧𝗔𝗬𝗦 𝗢𝗡 𝗧𝗛𝗘 𝗦𝗣𝗔𝗥𝗞 Whole model on-device: 78.57 GiB Packed 5-bit PLE table: 30.4 GiB, included in that total No NVMe PLE offload in this build. This is still turboderp’s 3.05bpw_h5_ng5 EXL3 pack, served through my vllm-exl3 integration. My work here is the serving integration and testing. These are preliminary performance measurements, not a quality evaluation or a claim of bit-exact full-output parity. 𝗘𝗫𝗣𝗟𝗢𝗥𝗘 𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 The benchmark page has the individual sweep values, long-prompt comparison, and measurement scope. You can play the animation, export the charts, or download the HTML and data to render them yourself. No Spark needed to view the results. Credit to @turboderp_ / ExLlamaV3 for the pack and kernels, @vllm_project for the serving engine, and @Alibaba_Qwen @QwenDevs for the model. Recipe + reproduction: github.com/vcruz305/Qwen3.8-… Interactive benchmark: vcruz305.github.io/Qwen3.8-F…

Qwen3.8-Flash-Next hits 36.4 tok/s with built-in MTP on ONE DGX Spark. ⚡ Turboderp’s 3.05 bpw EXL3 pack, served through my vllm-exl3 plugin. The full model occupies 78.6 GiB on device, including the packed n-gram table. 𝗧𝗛𝗘 𝗡-𝗚𝗥𝗔𝗠 𝗧𝗔𝗕𝗟𝗘 𝗦𝗧𝗔𝗬𝗦 𝗣𝗔𝗖𝗞𝗘𝗗 30.4 GiB at 5 bits, with rows reconstructed on demand using ExLlamaV3’s kernel. That avoids the ~102 GB BF16 table allocation that was killing construction in my vLLM path before the model could even load. The table has 320 million rows × 160 dimensions. Materializing all of it in BF16 was the wrong approach for this box. The row-reconstruction check is bit-exact against ExLlamaV3’s own reference. That checks the implementation, not parity with a BF16 model. 𝗠𝗧𝗣 𝗔𝗗𝗗𝗦 𝗠𝗢𝗥𝗘 𝗧𝗛𝗥𝗢𝗨𝗚𝗛𝗣𝗨𝗧 Built-in MTP, k=1: 33.8–36.4 tok/s No draft: 27.2–28.0 tok/s Measured improvement: +27–30% Mean acceptance length: 1.86 out of 2 These are single-request decode rates, excluding time to first token. No-draft TTFT was 185 ms in the 128-token test. The same pack’s built-in MTP head supplies the draft. 𝗧𝗛𝗘 𝗢𝗡𝗘-𝗦𝗣𝗔𝗥𝗞 𝗦𝗘𝗧𝗨𝗣 Hardware: GB10, 128 GB unified memory Pack: 3.05bpw_h5_ng5, about 80 GB on disk Model on device: 78.6 GiB 512 experts, top-10 routing, 48 layers Configured context: 32K GPU memory utilization setting: 0.80 KV pool with MTP: 275,636 tokens KV pool without draft: 385,570 tokens Those pool sizes are serving capacity, not a claim of a 385K context window. Startup is about 12–13 minutes, including roughly 9.5 minutes loading weights from NVMe. 𝗧𝗛𝗘 𝗦𝗘𝗥𝗩𝗜𝗡𝗚 𝗜𝗡𝗧𝗘𝗚𝗥𝗔𝗧𝗜𝗢𝗡 Nine load/serve fixes covered shard IDs, codebook markers, per-expert loading, padded matrix geometry, fused-MoE flags, compile-safe dense and lookup ops, the safetensors index, and small vLLM plumbing changes. DeepSeek-V4-Flash also passes a regression run on the same patched plugin. Greedy output stays coherent. In four prompt checks, MTP matched the no-draft text byte-for-byte on one and diverged late on three. Full output parity is not established yet. 𝗡𝗘𝗫𝗧 𝗜𝗡 𝗧𝗛𝗘 𝗦𝗪𝗘𝗘𝗣 Pending measurements: [k=2: __ tok/s, acceptance __] [k=3: __ tok/s, acceptance __] [Kernel profile: __] Credit to @turboderp_ / ExLlamaV3 for the pack and kernels, @vllm_project for the serving engine, @Alibaba_Qwen @QwenDevs for the model and @MarkusEicher70 for the GB10 access! My contribution here is the integration that lets this native EXL3 pack load and serve through vLLM on one Spark. Plugin: github.com/vcruz305/vllm-exl… Recipe: github.com/vcruz305/Qwen3.8-…