Baidu's Unlimited OCR 2 parses a hundred-page document in one pass, and at that scale each decoding step costs 7.0x less than with full attention.
The problem is dull but expensive. A parser that transcribes page 90 while still attending to every visual token from pages 1 to 89 pays for the whole history at every token it emits. Context grows, per-step cost grows with it, and long PDFs turn into a memory gamble.
SR-SWA, sparse reference sliding window attention, caps the active reference tokens at a fixed capacity; the card puts the working set at 512 tokens. The model looks only at what it is currently transcribing, the way a person copies a book without re-reading earlier chapters. Per-step cost stops tracking document length.
Their numbers at hundred-page scale: 7.0x faster per step than full attention, 5.6x faster than R-SWA. They also shipped MarathonDocBench, a new benchmark for hundred-page parsing, and claim SOTA on it plus MPDocBench. Weights are open on Hugging Face and ModelScope, repo is MIT, serving goes through SGLang with a custom wheel.
Caveats: self-reported numbers, measured per decoding step, not end-to-end wall clock on your box. Nobody outside Baidu has reproduced them yet. Still, OCR is exactly the workload where long context used to be quadratic and therefore useless, so fixed-cost attention there is worth watching.
Source: huggingface.co/baidu/Unlimit…
#LocalLLM #OCR #OpenSource #Inference #LLM
A 3-bit open model just tied Claude Sonnet 5 on a single RTX 5090.
akitaonrails wrapped up LLM Benchmark v4: almost 70 runs, almost 60 models, one Rails app built over 7 sprints with 14 security and correctness defects injected between them. Fix a defect unprompted and you get full weight, fix it only after a hint and you get 40%, miss it and you get zero.
The local result: Qwen3.8 Flash-Next at IQ3_S on an RTX 5090, served by the Strata engine, scored 91.0 at 159.2 tok/s for $0 per run. Claude Sonnet 5 scored 91.0 in the cloud, GPT 6 sol 91.0 as well.
Server side is a bit higher: the same model in NVFP4 on one B300 scored 92.5 at 195 tok/s, and DeepSeek V4.1 Flash and GLM 5.3 Flash matched it. That is 7.5 points off the 100 ceiling.
Caveats worth reading. The yardstick measures code review vigilance, not who builds your system better. Seven models hit 100, so the top of the scale stops separating. A repeated GPT 5.5 run scored 91 against its official 100, so small gaps are noise. One author, one harness, logs on GitHub.
What still separates open from closed is consistency: open models produce D-tier runs that no top closed model does. Also notable, the same DeepSeek serves 32 comfortable sessions on a B300 pod and only 8 on an H200 pod.
Which one would you put on your own GPU: 159 tok/s at $0, or the cloud's consistency?
Source: akitaonrails.com/en/2026/10/…
#LocalLLM #OpenSource #LLM #Inference #AI
Coming soon: My new recipe 1.2.7. with TensorFold
🚀We're building an adaptive KV-cache system that learns from real workloads to make smarter decisions about which prefixes to retain as your work changes.
The goal: reuse more useful context and spend less time repeating prefill.
Follow along.
@ashxhart
#LocalLLM #DGXSpark #DeepSeek #TensorFold
🤖 Made with AI
A 70 W workstation GPU gained 67% more decode throughput without a single new part.
Michal Piszczek took an RTX PRO 4000 Blackwell SFF (24 GB GDDR7, 432 GB/s, factory cap 70 W, no auxiliary power connector) and raised the runtime power limit to 120 W with a modified VBIOS staged in memory through patched open kernel modules. No EEPROM flash, no shunt mod.
Qwen3.8 27B, NVFP4 plus DFlash2, 8k input tokens, 512 output tokens, ECC off:
35.04 to 58.53 tok/s, +67.05%, at 120 W with memory +20% and core +180 MHz. Best prefill on the same grid rose about 52%.
Power alone did not explain it. With zero clock offsets, 70 to 120 W gave +54.65% on the 27B and only +13.48% on Qwen3.6 35B A3B in Q4_K_S. That MoE reached 115.56 tok/s (+36.01%) only with memory +33.3%, and it was flat from 106 to 120 W: 96.21 to 96.43 tok/s. One GPU, two different curves.
Caveats the author states himself: 120 W is above the 75 W PCIe slot budget on a card with no aux connector, connector and VRM temperatures were never measured, and each grid cell holds 1 to 3 requests out of 158 total. Throughput per watt fell in every paired 106 to 120 W comparison.
And 67% more tokens per second is about 40% less generation time at fixed output length, not a 67% faster answer.
The reusable part is the method: one lever at a time, prefill and decode measured separately.
Source: piszczek.pl/blog/rtx-pro-400…
#LocalLLM #Inference #GPU #Quantization #AI
Four DGX Sparks, one 45-layer MoE, 128 tok/s decode on code. KindlingAI's exlr8 build of GLM-5.3-Flash is a 4-bit trellis quantization that now beats NVFP4 on decode speed at equal quality.
Every routed and shared expert is stored at three trellis widths, picked at load time. K2: 78.2 GB of experts at 2.0 bpw, fits one 128 GB box at TP=1. K3: 117.3 GB at 3.0 bpw, TP=2. K4: 156.4 GB at 4.0 bpw, TP=3 or 4, the validated default. The model is 288 routed experts, top-8 routing plus one shared expert per MoE layer.
Measured on GB10 boxes, clocks locked at 2200 MHz, mean of 16 runs per workload with DFlash2 speculative decoding:
- TP=1 (K2): 53.0 tok/s code, 29.5 prose
- TP=2 (K3): 84.2 / 50.1
- TP=3 (K4): 105.8 / 57.8
- TP=4 (K4): 128.0 / 72.5, prefill 4,393 tok/s at 8K
KV cache at TP=4: a 38 GiB pin holds 6.46M tokens on DGX OS.
Against the NVFP4 build on the same four boxes: decode 3-8% faster, NLL within 0.0034 of BF16 on held-out sets, GSM8K 98% on both, KV pool about 15% larger. The honest cost is prefill, 7.7% slower at 32K and 128K.
Caveats: K2 and K3 were never scored against BF16, so measure them on your own workload. And the loader must round every trellis codebook value to FP8 E4M3 (round to nearest even, saturating) before applying the side vectors. A stock EXL3 decoder that skips it reads measurably worse weights from the same bytes. All numbers are KindlingAI's own vLLM recipe, not an independent benchmark.
Source: huggingface.co/kindlingai/gl…
#LocalLLM #Quantization #DGXSpark #vLLM #OpenSource
Serving a 3T-parameter model across 59 wafers, with weights that never move.
Cerebras published how they actually run trillion-parameter frontier models on wafer-scale silicon, using Kimi K3 (open weights, ~3T params, 69 linear-attention plus 24 dense layers) as the worked example. The arithmetic is refreshingly concrete:
- 3T parameters at 4 bits is 1.5 TB of raw weights. No single accelerator holds that, so the model gets split.
- One layer at 4-bit is about 16 GB. A WSE-3T wafer carries 44 GB of on-chip SRAM, so a whole layer fits on one chip.
- Dense layers need KV cache beside them (8-bit, 256k assumed request size): one dense layer per wafer, 24 wafers.
- Linear-attention layers keep a fixed-size state instead of a KV cache: two per wafer, 69 layers, 35 wafers.
- Total 59 wafers, roughly 2.6 TB of on-chip memory holding the weights. Claimed result: 1000-2000+ tokens per second on multi-trillion models.
The interesting part is what crosses the wire. Weights stay put, only activations hop between wafers over RDMA Ethernet at about 2 microseconds per hop, and the pipeline stays full with different requests, so stage N can work on token X of request A while stage N+1 handles request B. That is why a pipeline is not a batch of one.
Two honest caveats. This is vendor documentation, not an independent benchmark. And concurrency is capped purely by how much KV cache fits next to each dense layer, so the token rate and the request count trade off directly. Cerebras also argues the metric that matters is tokens per second per megawatt at equal output speed, which nobody on the GPU side reports. Until someone publishes that comparison, read 1000-2000 tok/s as a latency story, not a cost story.
Source: cerebras.ai blog, Oct 22.
#LocalLLM #AIInfra #Inference #Cerebras #OpenSource
🚀 My TensorFold Recipe v1.2.6 — Smart Adaptive Speculation & Advanced KV Pool Optimization
The first results were from the initial run. I've now tested it again with the speculation controller trained on historical workload data, and the improvement is pretty impressive! 🔥
🧠 1. Smart Adaptive Speculation (K1–K5)
The controller doesn't learn to predict tokens themselves. Instead, it learns how many tokens to speculate based on historical acceptance rates.
It remembers which tools, operations, and workload patterns previously resulted in successful or unsuccessful speculative decoding.
Based on that history, it dynamically selects a speculation budget between K1 and K5.
The key is avoiding unnecessary speculative work. Higher speculation depths are only selected when the controller has evidence that they've worked well in similar situations.
This reduces verification overhead and wasted computation while maintaining high acceptance rates.
Current results after the controller has learned:
~84% speculative token acceptance
Over 90% acceptance during repetitive agent workflows, such as long-running coding, reviews, and recurring tool calls
⚡ 2. KV Pool Memory Management & Fragmentation
I've also significantly improved how the KV pool manages memory.
One problem I discovered was memory fragmentation. Cached contexts need efficient placement and allocation to maximize reuse and avoid unnecessary prefill operations.
Think of it a bit like defragmenting old FAT16/FAT32 hard drives. The principle is similar, although here we're dealing with RAM and KV cache allocation.
Traditional defragmentation introduces overhead, so I've disabled continuous defragmentation.
Instead, I developed a mechanism that efficiently relocates existing cached contexts immediately before prefill, creating contiguous space for incoming contexts.
For example, relocating a 300K-token cached context within RAM costs roughly 100 ms in my testing. It's simply moving data between RAM locations, with relatively little overhead compared to the potential cost of recomputing that context.
I've also improved prefix ID tracking and storage. There were some issues with how newly loaded prefixes were registered and reused, which I've addressed.
🎯 3. KV Cache Hit Rate: ~96% → 98.5–99.2%
This is probably one of the most interesting improvements for my actual Hermes workload.
Previously, I was getting around 96% cache hits, which was already good.
Now I'm seeing approximately 98.5–99.2% in early testing! 🔥
When agents repeatedly send context back to the model, nearly everything can be reused directly from RAM. Only genuinely new content, such as new instructions or reasoning, needs additional prefill processing.
And that's important because every unnecessary prefill introduces latency.
The goal isn't simply maximizing raw decode tokens per second. It's minimizing total overhead and making long-running agent workflows as efficient and responsive as possible.
I'm going to let this version run for several more hours under my real Hermes workload and analyze the detailed logs before drawing final conclusions.
But so far, v1.2.6 looks extremely promising! 😎🚀
git: github.com/LakyFx/deepseek-v…
huggingface.co/LakyFx/DeepSe…
#TensorFold #DeepSeek #DGXSpark #EXL3 #LocalLLM #AIInference
@ashxhart
Your speculative decoding win on an H100 does not follow you to a MacBook.
A NeurIPS 2026 workshop paper (On-Device Intelligence + SLMs for Agentic Systems) tested four inference optimization families on four Apple Silicon machines, single-request decode, with matched RTX 4090 runs where possible. The results invert most of the folklore.
Speculative decoding: every method they evaluated reduced throughput on Apple, even though the same techniques report 2x to 6x gains on server GPUs. The draft-verify loop pays off under batching, not under a single token stream.
KV cache pruning: up to 1.75x end-to-end throughput on Apple, while on desktop NVIDIA the same trick can cost throughput. Same technique, opposite sign.
Ported CUDA kernels: transfer only partially. Public Metal has no async copy and batch-one decode makes several serving-oriented mechanisms inapplicable, so only the transferable parts deliver, up to 1.20x decode.
Quantization: gains vary machine by machine, there is no universal multiplier.
The conclusion is the useful part. A speedup is not a portable property of a technique, it is a joint function of technique, runtime, hardware contract and workload regime.
Which means a benchmark from a batched server tells you almost nothing about your laptop. The only number that counts is the one you measured on your own hardware.
Paper: arxiv.org/abs/2610.31479
#LocalLLM #AppleSilicon #LLMInference #MachineLearning #AI
18 open-weight models, 291 graded runs, two RTX 5060 Ti cards: none replaced a 27B dense model.
Siim Tolk built a loop that finds new open-weight models on Hugging Face each morning, downloads the ones fitting his 32 GB of VRAM, and lets a coding agent run his 97-case suite overnight. Every case runs 3 times, graded by code, not by another model. His production Qwen3.8-27B passed 283 of 291 runs. Nemotron 3.5 Bolt 74B-A9B, 4th under 100B on Artificial Analysis, passed 247, and only 16 of 45 Estonian cases.
The serving setup is the useful part: IQ4_XS at 15.5 GB instead of Q5_K_M at 19.7 GB, same score and 20% faster, all layers split across both cards, 262k context in two 128k slots, q4_0 KV cache with flash attention, and the built-in MTP draft head. MTP took the model from 25 to 41 tok/s on one card for 1.2 GB of extra memory. n-gram drafting on top was 15 to 20% slower, because MTP already guessed most of the repetitive JSON.
The honest caveats: decode falls from 45 tok/s on a short prompt to 26.5 tok/s after 90k tokens, and prefill of a fresh 90k prompt takes about 95 s. Each card needs ~2.8 GB of compute buffer, which is why Gemma 4 31B QAT fit on paper and ran out of memory in five configurations once its draft model was added. vLLM generated up to 1.4x faster but needed ~30 of 32 GB and 4.5 to 5.5 minutes to start.
Tokens per second is not time per answer. His previous MoE at 137 tok/s moved the median real request from 2.3 to 8.6 seconds when he switched to the denser, more accurate 27B. And one "uncensored" build followed an instruction planted inside a document every single time, which disqualifies it for an agent that reads the web.
Suite and results: github.com/siimtolk/local-ll…
Source: siimtolk.github.io/articles/…
#LocalLLM #llamaCPP #OpenSource #AI #Agents
Fastest TP3 recipe yet for DeepSeek 4.1 Flash 1017 🚀
🤖 763B parameters. Weights + recipe are out.
Check it out 👇 links in coments.
Git: github.com/LakyFx/deepseek-v…
Weights: huggingface.co/LakyFx/DeepSe…
Recursive's Flint: 39.8B MoE with 2.5B active per token, Apache 2.0. Vendor-run 89.7% SWE-bench Pro v2, 74.2% SWE-bench Verified. The 3-bit GGUF is 18.5GB, so 24GB RAM runs it at ~131 tok/s on an M4 Pro. Coding-agent quality on a laptop.
#LocalLLM #OpenSource #AI
OpenAI just slowed its own frontier training because a model got too good at hacking.
In a post published yesterday, OpenAI says preliminary evidence shows its upcoming model Astra may meet the "critical cybersecurity capability" threshold under its Preparedness Framework. The response: a two-week pause on reinforcement learning for models headed to deployment, and the largest planned frontier RL run stays on hold.
The numbers behind the pause:
- monitoring overhead: roughly 20% of the inference compute being monitored
- alert target: within 30 minutes of concerning activity
- if the flag is not cleared as a false positive within 30 minutes, safety, security and research teams are expected to pause the run
Since August 7, every Astra inference with tools carries the extra monitoring requirement, not just RL training. A significant number of Astra workloads remain paused until they are migrated into the new sandboxes.
The honest read: this is a major lab publicly slowing scaling because a capability threshold was crossed, not because of a release schedule. It also sets a bar the rest of the field will be compared against, and OpenAI admits its own framework needs rebuilding.
Source: openai.com/index/pacing-mode…
#AI #AISafety #OpenAI #FrontierModels #CyberSecurity
A 3-bit quant of an open model on one RTX 5090 scored 91.0 on the biggest coding benchmark of the year. Claude Sonnet 5 in the cloud scored 91.0 too, at about $27 per run.
Fabio Akita closed benchmark v4 yesterday: 67 runs, roughly 60 models, past $6,000 spent. The yardstick is code review vigilance, not code writing. The model builds a Rails chat app in seven sprints while 14 security and correctness defects get injected. Fixed unprompted counts full, after a reveal 40%, never zero.
Consumer result: Qwen3.8 Flash-Next, IQ3_S at about 3.4 bits, RTX 5090 with the Strata engine. 91.0, 159.2 tok/s, $0 per run. Same model in NVFP4 on a B300: 92.5, 195 tok/s, $12 of inference. Going home cost 1.5 points and 18% of the speed.
What did not survive the move home: Ornith 1.5 35B went 81.0 on a B300 (BF16) to 64.5 on the 5090 (Q8_0), GLM 4.7 Flash 33.5 to 24.0. The bits were not the main problem, the serving stack was. Serving the same checkpoint yourself moved scores 8 points up (GLM 5.3 Flash) and 67 points down (Nex N2.5 Pro).
Caveats: 42 of 67 runs hit Tier A and seven models tied at 100, so the top is saturated. A repeated GPT 5.5 run scored 91 against its own 100, so a few points are noise. One run per configuration.
The takeaway for local inference: a 125B MoE with 6B active per token, 3-bit weights and hot experts pinned to VRAM is now enough for serious review work at 159 tok/s. Below a 5090 it collapses. Qwen 3.8 27B on Strix Halo finished at 7.7 tok/s.
Source: akitaonrails.com/en/2026/10/… (code and logs on GitHub)
#LocalLLM #LLM #AI #OpenSource #Inference
More work from three DGX Sparks our TensorFold recipe 1.2.4 is published. 🚀
DeepSeek-V4.1-Flash-1017, EXL3 ~3.51 bpw routed experts, MAX reasoning, running across 3× DGX Spark with TP3.
The focus is smart adaptive speculation: learning actual verification cost and draft acceptance, then adjusting speculative work allocation as concurrency changes.
What we improved:
More accurate cost estimation: price the verification graph actually executed and skip sampling rows the current path cannot use.
CPU/GPU overlap: prepare CPU penalty scores while the GPU works.
Controlled adaptation: limit timing-probe overhead and refresh stale cost estimates.
Fairer scheduling: give existing streams a decode turn between new admissions.
Against 1.2.2TF, our heavy agent workflow real tools, code execution, tests and repair rounds delivered:
282 tok/s sampled aggregate decode peak at C11.
16.8% more correct tasks/minute.
14.3% less total workflow time.
78/78 tasks correct in both versions.
Same hardware, pinned weights and sampler; one pass per version. Output lengths and repair paths vary. The peak is a roughly two-second rolling observation, not sustained workflow throughput.
All 14 measured recipes, C1–C12 results and build instructions are public.
Built on @ashxhart TensorFold
🔧 GitHub: recipes, source and benchmarks github.com/LakyFx/deepseek-v…
🤗 Hugging Face: weights huggingface.co/LakyFx/DeepSe…
#DeepSeek #DGXSpark #LocalLLM #LLMInference
Good JOB @ViC305 🥳 DS4.1f for 1x Spark owners
🚨 𝗠𝗬 𝟭.𝟲𝟮 𝗕𝗣𝗪 𝗗𝗘𝗘𝗣𝗦𝗘𝗘𝗞 𝗤𝗨𝗔𝗡𝗧 𝗝𝗨𝗦𝗧 𝗕𝗘𝗔𝗧 𝗧𝗪𝗢 𝗛𝗜𝗚𝗛𝗘𝗥-𝗣𝗥𝗘𝗖𝗜𝗦𝗜𝗢𝗡, 𝗧𝗪𝗢-𝗦𝗣𝗔𝗥𝗞 𝗤𝗨𝗔𝗡𝗧𝗦! 🧪🔥
And I have the REAL benchmark results to prove it! 🤯
Remember my DeepSeek V4.1 Flash 1017 SAGE-EXL3 release from YESTERDAY?
📦 1.62 bpw routed experts
💻 99.61% code answer-token agreement
🖥️ ONE DGX Spark @nvidiaai
Well, I promised to go beyond KL divergence and token agreement and test ACTUAL coding, mathematics, reasoning and tool execution.
𝗧𝗛𝗘 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 𝗔𝗥𝗘 𝗜𝗡!
━━━━━━━━━━━━━━━━━━━━
🏆 𝗗𝗘𝗘𝗣𝗦𝗘𝗘𝗞 𝗩𝟰.𝟭 𝗙𝗟𝗔𝗦𝗛 𝟭𝟬𝟭𝟳 — 𝗡𝗜𝗡𝗘-𝗦𝗨𝗜𝗧𝗘 𝗦𝗛𝗢𝗪𝗗𝗢𝗪𝗡
353 matched benchmark questions.
9 evaluation suites.
Strict pass@1 scoring.
Temperature 0.
🟢 𝗠𝗬 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯 — 𝟭.𝟲𝟮 𝗕𝗣𝗪
🖥️ 1× DGX Spark
🏆 𝟵𝟰.𝟭% — 𝟯𝟯𝟮/𝟯𝟱𝟯 𝗣𝗔𝗦𝗦𝗘𝗗!
🔵 𝗝𝗝 𝗡𝗘𝗪 — 𝟮.𝟴 𝗕𝗣𝗪
🖥️ 2× DGX Sparks
📊 90.1% — 318/353
🟣 𝗠𝗜𝗔𝗟𝗔𝗕 — 𝟮.𝟵 𝗕𝗣𝗪
🖥️ 2× DGX Sparks
📊 88.4% — 312/353
𝗜'𝗠 𝗨𝗦𝗜𝗡𝗚 𝟰𝟮–𝟰𝟰% 𝗙𝗘𝗪𝗘𝗥 𝗕𝗜𝗧𝗦 𝗣𝗘𝗥 𝗥𝗢𝗨𝗧𝗘𝗗 𝗘𝗫𝗣𝗘𝗥𝗧 𝗣𝗔𝗥𝗔𝗠𝗘𝗧𝗘𝗥...
𝗔𝗡𝗗 𝗦𝗖𝗢𝗥𝗜𝗡𝗚 𝗛𝗜𝗚𝗛𝗘𝗥 𝗢𝗡 𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞! 🤯
━━━━━━━━━━━━━━━━━━━━
💻 𝗧𝗛𝗘 𝗖𝗢𝗗𝗜𝗡𝗚 𝗥𝗘𝗦𝗨𝗟𝗧𝗦 𝗔𝗥𝗘 𝗜𝗡𝗖𝗥𝗘𝗗𝗜𝗕𝗟𝗘!
Scores shown as SAGE / JJ / Mia:
🏆 HumanEval+
39/40 vs 34/40 vs 34/40
🏆 MBPP+
53/60 vs 50/60 vs 49/60
🏆 LiveCodeBench
15/17 vs 14/17 vs 11/17
🧠 𝗥𝗘𝗔𝗦𝗢𝗡𝗜𝗡𝗚 & 𝗠𝗔𝗧𝗛
🏆 MMLU-Pro STEM
95/100 vs 93/100 vs 91/100
🏆 MATH-500
40/40 vs 36/40 vs 36/40
𝗙𝗜𝗩𝗘 𝗢𝗨𝗧𝗥𝗜𝗚𝗛𝗧 𝗪𝗜𝗡𝗦! 🔥
I also tied both models on:
✅ MGSM: 40/40
✅ BFCL: 39/40
✅ SciCode first-step: 6/10
For full transparency, both JJ and Mia won the multi-step tool-chain test, scoring 6/6 against my 5/6.
𝗦𝗢 𝗧𝗛𝗔𝗧'𝗦 𝟱 𝗪𝗜𝗡𝗦, 𝟯 𝗧𝗜𝗘𝗦, 𝗔𝗡𝗗 𝟭 𝗟𝗢𝗦𝗦.
All while using a substantially lower expert bitrate!
━━━━━━━━━━━━━━━━━━━━
🚀 𝗕𝗜𝗚 𝗨𝗣𝗗𝗔𝗧𝗘: 𝗠𝗬 𝗣𝗨𝗕𝗟𝗜𝗖 𝘃𝗟𝗟𝗠 𝗥𝗘𝗖𝗜𝗣𝗘 𝗜𝗦 𝗡𝗢𝗪 𝗣𝗨𝗦𝗛𝗜𝗡𝗚 𝟱𝟮𝟰𝗞 𝗖𝗢𝗡𝗧𝗘𝗫𝗧!
YES. ON ONE DGX SPARK! 🤯
With the latest W2 kernel, updated expert set and KV-cache configuration, I just completed an actual long-context retrieval test!
📚 𝗙𝗨𝗟𝗟-𝗟𝗘𝗡𝗚𝗧𝗛 𝗣𝗥𝗢𝗠𝗣𝗧 𝗧𝗘𝗦𝗧
🧠 518,103 INPUT TOKENS!
🎯 Fact planted HALFWAY through the context
✅ THE MODEL FOUND THE FACT!
⏱️ Time to first token: 470 seconds (~7.8 minutes)
🚀 Prefill: ~1,100 tok/s
⚡ Decode at that context depth: 25.6 tok/s
💾 Minimum available memory: 2.19 GiB
🛡️ Memory safety floor: 0.5 GiB
And on SHORTER prompts?
⚡ ~36 TOK/S DECODE!
🔧 ~130.5 ms per speculative step
🎯 ~4.7 accepted tokens per step
𝗪𝗘'𝗥𝗘 𝗧𝗔𝗟𝗞𝗜𝗡𝗚 𝗔𝗕𝗢𝗨𝗧 𝗢𝗩𝗘𝗥 𝗛𝗔𝗟𝗙 𝗔 𝗠𝗜𝗟𝗟𝗜𝗢𝗡 𝗜𝗡𝗣𝗨𝗧 𝗧𝗢𝗞𝗘𝗡𝗦 𝗣𝗥𝗢𝗖𝗘𝗦𝗦𝗘𝗗 𝗢𝗡 𝗔 𝗦𝗜𝗡𝗚𝗟𝗘 𝗦𝗣𝗔𝗥𝗞! 🔥
To be clear, this was a successful full-length retrieval test, not a claim of perfect accuracy across every possible long-context task.
And the 7.8-minute prefill is a real trade-off at that depth.
But seeing a model this large, with this level of compression, successfully retrieve information buried inside 518K tokens on ONE 128GB machine is an exciting milestone for local AI!
━━━━━━━━━━━━━━━━━━━━
⚡ 𝗔𝗡𝗗 𝗧𝗛𝗘 𝗢𝗡𝗘-𝗦𝗣𝗔𝗥𝗞 𝗣𝗘𝗥𝗙𝗢𝗥𝗠𝗔𝗡𝗖𝗘 𝗞𝗘𝗘𝗣𝗦 𝗜𝗠𝗣𝗥𝗢𝗩𝗜𝗡𝗚!
Separately, my latest REXL3 development measurements at a 256K context configuration:
⚡ 55.35 tok/s single-stream decode
🚀 2,045 tok/s peak prefill
⏱️ 509 ms TTFT on a 512-token prompt
Those REXL3 numbers come from a different engine and workload than the new public vLLM 518K-token test.
And here's something else I'm particularly excited about...
𝗔𝗟𝗟 𝟭𝟱,𝟯𝟲𝟬 𝗥𝗢𝗨𝗧𝗘𝗗 𝗘𝗫𝗣𝗘𝗥𝗧𝗦 + 𝟯𝟴𝟰 𝗗𝗥𝗔𝗙𝗧 𝗘𝗫𝗣𝗘𝗥𝗧𝗦 𝗔𝗥𝗘 𝟭𝟬𝟬% 𝗚𝗣𝗨-𝗥𝗘𝗦𝗜𝗗𝗘𝗡𝗧 𝗜𝗡 𝗠𝗬 𝗧𝗘𝗦𝗧𝗘𝗗 𝗥𝗘𝗫𝗟𝟯 𝗖𝗢𝗡𝗙𝗜𝗚𝗨𝗥𝗔𝗧𝗜𝗢𝗡!
ZERO streamed experts.
𝗢𝗡𝗘. 𝟭𝟮𝟴𝗚𝗕. 𝗗𝗚𝗫. 𝗦𝗣𝗔𝗥𝗞. 💚
@mr_r0b0t and I are continuing to refine REXL3 before its release. More on that soon! 👀
━━━━━━━━━━━━━━━━━━━━
💚 𝗧𝗛𝗜𝗦 𝗜𝗦 𝗪𝗛𝗬 𝗜 𝗕𝗨𝗜𝗟𝗗 𝗙𝗢𝗥 𝗟𝗢𝗖𝗔𝗟 𝗔𝗜.
Just YESTERDAY, I shared my goal of bringing extremely aggressive, high-quality quantization to people who don't have access to massive compute clusters.
With the support of @AWERXAI, I've been working tirelessly to push that vision forward.
And now we're seeing a 1.62 bpw expert quant outperform higher-bitrate alternatives on actual tasks, not just token-level fidelity measurements.
This doesn't establish mathematical losslessness or superiority on every possible workload. These are matched subsets of nine benchmark suites, not full public leaderboard scores.
But it demonstrates something IMPORTANT:
𝗠𝗢𝗥𝗘 𝗕𝗜𝗧𝗦 𝗗𝗢𝗡'𝗧 𝗔𝗨𝗧𝗢𝗠𝗔𝗧𝗜𝗖𝗔𝗟𝗟𝗬 𝗠𝗘𝗔𝗡 𝗕𝗘𝗧𝗧𝗘𝗥 𝗥𝗘𝗦𝗨𝗟𝗧𝗦.
And more hardware doesn't automatically mean you need it!
Huge respect to @jayleaton for his contributions and for giving the community more ways to run these incredible models.
And thank you @AWERXAI for supporting this work. 🙏
🔗 𝗠𝗬 𝗦𝗔𝗚𝗘-𝗘𝗫𝗟𝟯 𝗤𝗨𝗔𝗡𝗧
huggingface.co/vcruz305/DSV4…
🛠️ 𝗢𝗡𝗘-𝗦𝗣𝗔𝗥𝗞 𝘃𝗟𝗟𝗠 𝗥𝗘𝗖𝗜𝗣𝗘
github.com/vcruz305/DSV4.1-F…
𝗙𝗥𝗢𝗡𝗧𝗜𝗘𝗥 𝗠𝗢𝗗𝗘𝗟𝗦. 𝗟𝗢𝗖𝗔𝗟 𝗛𝗔𝗥𝗗𝗪𝗔𝗥𝗘. 𝗥𝗘𝗔𝗟 𝗥𝗘𝗦𝗨𝗟𝗧𝗦.
𝗟𝗘𝗧'𝗦 𝗞𝗘𝗘𝗣 𝗣𝗨𝗦𝗛𝗜𝗡𝗚! 🚀🔥
