
CEO & Founder | PhD | Startup Advisor | @Columbia | Author Generative Software Engineering https://xitter.cf/t.co/9oqvHuTX5f | ๐ Follow for AI & Vibe Coding Tips ๐
PNW
Joined July 2023
- Tweets17.8K
- Following552
- Followers12.3K
- Likes5.3K
Pinned Tweet
Qwen3.5-4B running CPU-ONLY at up to ~11+ tps on a Ryzen 5 laptop. No GPU. Running models in RAM and CPU is something the Qwen4 team is already thinking about.
๐ก I built this standalone .exe because businesses are already deploying local models to save money and ensure privacy. (thumb drive friendly - just drag drop and doubleclick)
My llama.cpp recipe ๐
๐ง Qwen3.5-4B Q4_K_M GGUF
โ๏ธ Ryzen 5 7540U โ 6C/12T
๐งต --threads 9
๐งต --threads-batch 12
โก --prio 2
๐ --poll 50
๐ฆ --batch-size 2048
๐ฆ --ubatch-size 512
๐ --flash-attn on
๐ง KV cache: q4_0 / q4_0
๐ง --repack
๐พ --mmap
๐ค --parallel 1
๐ซ --device none
๐ซ --gpu-layers 0
๐ซ KV/op GPU offload
๐ซ MTP OFF
Interesting result ๐ MTP=3 was slower (~10 tps in benchmarks).
Plain decode + 9 threads + Q4 KV hit ~11.4 tok/s.
That's about 10% faster just from tuning llama.cpp โ on a basic laptop CPU.
Are you getting ready for Kimi K3.1?
It may already be testing in Arena under the anonymous name Kibitz.
And there are now multiple clues pointing back to Moonshot.
๐งฌ Community tokenizer/fingerprint tests reportedly behave like Kimi
๐ฃ๏ธ In some Arena tests, Kibitz identifies itself as Kimi/Moonshot
๐ Kimi Code's configuration now contains a model-switch target named k3.1
And the anonymous naming pattern is eye catching
K2.5 โ Kiwido
K2.6 โ Kiwire
K3 โ Kivine
K3.1? โ Kibitz
K3 itself reportedly appeared anonymously in Arena roughly 30 hours before release.
None of this confirms Kibitz = Kimi K3.1.
Moonshot has not announced anything officially yet.
But K3.1 itself looks increasingly real.
Kimi Code now literally contains a migration rule pointing existing K3 sessions toward k3.1.
If Moonshot follows K3 with open weights again, K3.1 could become the next ridiculous multi-GPU / multi-node Local AI project.
Links in ALT
A new decision model tied for #1 on Artificial Analysis' Decision Index and its 8 points ahead of Jev 1.14.
SubQ Snap 1.1 is built to make the decisions inside an AI agent.
Artificial Analysis Stats
๐ฅ SubQ Snap 1.1 โ 65
๐ฅ d1-ultra โ 65
๐ฅ Elephant 9B โ 65
Jev 1.14 โ 57
And Snap has one unusual advantage, its context
๐ 384K context
Subquadratic says its Sparse Attention architecture reduces attention compute from quadratic to linear scaling with context.
Snap reads the context and returns probabilities over a fixed set of allowed decisions in a single forward pass.
Subquadratic reports these benchmarks
๐งฐ When2Call โ 96.2
๐ 81% fewer confident errors on its unanswerable-query test
๐ 384K context
๐ฐ $0.04 / 1M input tokens
๐ no output-token charge
This isn't a local model today as Snap 1.1 is currently offered through Subquadratic's API.
Layer this between the prompt and the models
Link in ALT
Nvidia released Kumo Relational, a small purpose-built AI model to predict what happens next from your relational database.๐
๐Customer churn. | Fraud. | Future demand. | Conversion. | Risk.
This is not a replacement for SQL. SQL tells you what already happened.
Kumo is designed to predict what happens next from the relationships and history already inside your database.
And it can do this without training a separate model for every task or manually engineering features.
Instead of flattening a relational database into one giant table, it works across the relationships between your existing tables.
Give it historical examples in-context, up to 10,000, and ask it to perform classification or regression.
Nvidia tested it across ๐ 41 tasks ๐๏ธ 15 databases
They report it beating supervised relational deep-learning systems and expert-engineered gradient-boosted-tree pipelines.
On SAP SALT, Nvidia says it beat large AutoGluon ensembles by 8% MRR.
You can download the weights and run it locally on CUDA.
The checkpoints are tiny
๐ง Classification โ ~120MB
๐ Regression โ ~124MB
Your model doesn't just query your private database anymore. It can try to predict what happens next from it.
๐ NVIDIA-authored code โ Apache 2.0
โ๏ธ Weights โ OpenMDW 1.1
Link in ALT
2 x ~$400 Snapdragon mini PCs. No GPU used and 64GB of RAM combined.
And they're running a 95GB Qwen3.8-Flash-Next at 20.3 tps. ๐
The model is 31GB larger than their combined RAM.
I have to build this.
The setup
๐ฅ๏ธ 2ร Lenovo IdeaCentre Mini
โ๏ธ Snapdragon X X1-26-100
๐พ 32GB LPDDR5 each
๐ฝ 512GB NVMe each
๐ connected with one USB4 cable
๐ฐ ~$800 total
Here's the model:
๐ง Qwen3.8-Flash-Next
๐ฆ IQ4_NL โ 95GB
Reported with the author's ToRc inference engine
๐ฅ Read โ 46.2 tps
โ๏ธ Generation โ 20.3 tps
๐ป Coding generation โ 19.2 tps
And the Adreno GPUs aren't being used.
ToRc uses the Snapdragon CPUs + Hexagon NPUs for some of the dense math, while the model's experts are streamed from NVMe as they're needed because the weights don't fit in RAM.
The second box also isn't simply splitting normal token generation in half.
For Flash-Next, ToRc pipelines MTP across the two machines over USB4.
โ ๏ธ ToRc isn't publicly released yet and these are the author's results, not independently reproduced benchmarks.
Link in ALT
This may be one of the more important Healthcare specialized Local AI weight releases.
Ant open-sourced the weights for Ling-3.0-flash-Sante.
The model itself isn't new as Sante has been available through APIs since September but ...
What's new is ๐ you can now bring the healthcare model home.
Stats
๐ง 124B total parameters
โก only ~5.1B active/token
๐ 256K context
๐ฉบ healthcare-specialized
๐ deep research
๐ค multi-agent collaboration
๐ง tool use
It's built on the same Ling-3.0-flash architecture that already runs in llama.cpp.
For reference, base Ling-3.0-flash GGUFs already exist around
Q4_K_M โ ~77GB
IQ3 โ ~54GB
IQ2 โ ~40โ46GB
IQ1 โ ~28โ31GB
โ ๏ธ Those are BASE Ling-3.0-flash quants. I haven't found an official Sante GGUF yet.
But now that the Sante weights are downloadable, this becomes a home model for
๐ฅ๏ธ 96/128GB workstations
๐ high-memory Apple Silicon
โก DGX/RTX Spark-class unified memory
๐ multi-GPU local systems
And healthcare is one of the places where Local AI can be useful for private documents + research + agent workflows can stay on hardware you control instead of being sent to a cloud API.
Ant also reports substantial healthcare gains in the new release, including
HealthBench Professional
45.73 โ 54.29
AFUMED ComplexReasoning
65.22 โ 76.92
Link in ALT
Wow, that's a pretty nice bump for Ling-3.1-flash.
Ant introduced the new Ling-3.1-flash-1019 checkpoint, and its Artificial Analysis score jumped
Ling-3.1-flash โ 41
Ling-3.1-flash-1019 โ 44
The highlighted gains come from its updated agentic performance
AA-Briefcase
~1400 โ 1524 Elo
AutomationBench-AA
62% โ 68%
The existing Ling-3.1-flash is already open-weight under MIT and it's big.
Stats
๐ง ~560B total
โก ~25B active/token
๐ up to 1M context
Community GGUFs already go down to
IQ1_S โ 142GB
IQ2_XXS โ 174GB
IQ3_XXS โ 256GB
Q4_K_M โ 354GB
The existing Q4 has even been loaded across 4ร DGX Sparks using llama.cpp RPC.
โ ๏ธ 1019 is a new checkpoint/update, not a new Ling architecture announcement.
Ant hasn't released the updated 1019 weights yet. It says open weights + API are coming soon.
Link in ALT
Ling-3.1-flash scores 41 on Artificial Analysisโs Intelligence Index versus its predecessor.
The 560B model I posted last week now has another reason to stay on the watchlist.
You can test it before the weights arrive on OpenCode because it has added free hosted access, and Vercelโs free trial runs through October 13.
Vercel currently serves 262K context. The advertised ceiling is 1M.
Still waiting on downloadable weights and confirmed local deployment requirements.
Save the tasks it handles well now, then rerun them on a local quant when available. How much of that capability survives?
Vercel link in ATL
Apparently, private Local AI roleplaying AI is a pretty serious use case.
ReadyArt's new Wicked-Tutu family is already pulling serious downloads.
And this is not a tiny model
Stats
๐ง 31B dense Gemma 4
๐พ Q4_K_M โ ~18.7GB
๐ฎ fits in 24GB-GPU territory
๐ up to 256K context
๐ completely local
Runs directly with
โ๏ธ llama.cpp
๐ฆ Ollama
๐ฅ๏ธ LM Studio
ReadyArt trained it specifically as a private local roleplay model, with a purpose-built dataset and thousands of scenario categories.
The earlier v0.1 GGUF has already passed 17K downloads, and they're iterating fast enough that a Final 31B build is already out.
Local AI isn't only competing with ChatGPT but it's enabling categories where privacy + customization + no cloud at all are the product.
โ ๏ธ The HF metadata says Apache 2.0, but the model card also adds personal/non-commercial terms, so I'd treat the licensing as unclear for now.
Link in ALT
This small Local AI model could replace an entire document-processing pipeline.
ATH-MaaS released OvisOCR3.
Itโs only a 0.8B-class multimodal model.
But one local model can now do all these things
๐ document โ Markdown
๐งพ document โ structured JSON
๐ OCR + bounding boxes
๐งฎ formulas โ LaTeX
๐ charts โ structured data
๐ key-value extraction
๐ผ๏ธ figure captioning
๐ flowcharts โ Mermaid
๐ cross-page table reconstruction
That last one is interesting.
Give it consecutive document pages and OvisOCR3 can determine whether a table continues onto the next page and reconstruct the complete table.
Team-reported results
OCRBench โ 88.8
OCRBench KIE โ 90.5
Cross-page table Relation F1
OvisOCR3 โ 97.07
Qwen3.8-Max โ 94.47
And itโs
๐ง built on Qwen3.5-0.8B
๐ Apache 2.0
๐ downloadable/local
โ๏ธ Transformers + vLLM + SGLang
๐ OpenAI-compatible local server through vLLM
No official GGUF that I can find yet.
Link in ALT
The company behind PUBG just trained its own 29B Local AI model from scratch.
And the architecture is interesting.
KRAFTONโs new Raon-1 mini is โฆ
๐ง 28.8B total parameters
โก only 4.3B active/token
๐พ official Q4_K_M โ 19.7GB
๐ 262K context
๐ Apache 2.0
๐ค reasoning + agent/tool use
18 of its 25 layers use fixed-state linear attention.
Only 7 layers maintain a growing KV cache.
KRAFTON says those layers use only ~8 KiB of BF16 KV per token.
So even at the full 262K context, thatโs roughly:
๐ ~2GB of growing KV cache before other runtime/state overhead.
And in KRAFTONโs reported comparisons, this new 30B-A4B model beats Qwen3.6-35B-A3B on 11 of 18 listed benchmarks.
Lots of new models coming for Qwen3.6-35B recently.
โ ๏ธ Preview release. KRAFTONโs benchmark results. Upstream llama.cpp support is still pending and KRAFTON provides its own branch today.
Link in ALT
Nvidia lets you build a 1TB Local AI cluster out of DGX Sparks.
& Nvidia has a playbook for AI clustering them.
8 ร DGX Spark
๐ง 128GB each
โ
๐ฅ 1TB unified memory across the cluster
Nvidia now officially supports manually configured clusters of up to 8 x DGX Sparks using their built-in ConnectX-7 networking.
And Nvidia says that 1TB is enough to run frontier-class models like GLM-5.3 locally.
They specifically call out
๐ง larger models
๐ longer contexts
๐ค multiple concurrent agents
๐จโ๐ป multiple developers sharing one local backend
โ๏ธ distributed development
They list models such as
โ GLM-5.3 Flash
โ DeepSeek V4 Flash
โ Qwen3.8 Flash Next
And later this month Nvidia Sync Model Launcher is supposed to make it easier.
Link in ALT
AMD and Intel agreed on something pretty unusual.
A local PyTorch AI app shouldn't have to care whether the CPU underneath it is AMD or Intel.
AMD is folding its ZenDNN AI optimizations into oneDNN, the library originally developed by Intel.
AMD + Intel have now aligned on oneDNN as the default open-source x86 acceleration path in PyTorch.
ZenDNN has been working on refining for ...
๐ง LLM inference
โก INT4 / INT8 / BF16 / FP16
๐งฉ fused MoE operations
๐ vLLM CPU acceleration
๐ง optimized attention/matmul
Now that work is being brought into the common oneDNN ecosystem AMD + Intel are also jointly developing ACE, new x86 AI matrix instructions.
The plan is for future AMD/Intel AI instructions โ oneDNN โ PyTorch
So local PyTorch software gets one x86 target instead of separate vendor paths.
โ ๏ธ This does NOT make llama.cpp/GGUF faster, and AMD/Intel haven't published new performance numbers from the merger.
๐ฏBut for CPU Local AI, the two companies that spent decades fighting over x86 are now cooperating on the AI software layer.
Link in ALT
Here we go! Purpose built AI generated LLMs!
Then AI helped optimize the code that runs it.
Recursive released Flint, a new coding/agent model designed specifically for fast local inference.
Beats Qwen3.5/3.6/3.8 in some coding benchmarks!
Stats โฆ
๐ง 39.8B total parameters
โก only ~2.5B active/token
๐ 262K documented context
๐ค reasoning + tool use
๐ Apache 2.0
And there are official GGUFs:
๐ข NVIDIA NVFP4 โ 22.9GB
๐ Apple INT4 โ 26.0GB
Recursive recommends 48GB-class memory for these local builds, so think:
๐ 48GB+ Apple Silicon
๐ข 48GB+ NVIDIA
โก 64GB RTX Spark
Recursive reports its optimized runtime at
๐ M4 Pro โ >125 TPs
๐ข 64GB RTX Spark โ >150 TPs
And WIREDโs Will Knight actually ran Flint locally on a 48GB Mac and reported it was fast and successfully handled the coding jobs he tried.
Side Note: Recursive built Flint using its Eureka Machine, an automated AI research system that helped
๐งฉ design the architecture
๐งช develop training recipes
โ๏ธ optimize infrastructure
๐ป write hardware-specific inference kernels
Nvidia says those specialized kernels improved Flintโs RTX Spark generation performance by 55% wo changing the model weights.
Recursive-reported coding results
SWE-bench Verified โ 74.2
SWE-bench Pro v2 โ 89.7
Terminal-Bench 2.1 โ 54.9
LiveCodeBench v6 โ 80.3
So this isnโt merely AI writing more AI code.
Link in ALT
A single 24GB GPU can now run a 35B scientific research agent that spawns sub-agents and checks its own evidence.
Sage-A1.5 is 35B / ~3B active, and its official Q4_K_M is just 21.7GB.
Here are the new Sage-A1.5 model stats
๐ง 35B MoE / ~3B active
๐พ official Q4_K_M โ 21.7GB
๐ฎ fits a single 24GB GPU at moderate context
๐ 256K native context
๐๏ธ multimodal
๐ Apache 2.0
And it's specifically trained as a long-horizon research agent.
So it is designed to
๐ use tools/search/code
๐ค spawn parallel sub-agents
โ
inspect its own evidence
๐ change direction when research stalls
No separate critic model required for its verification mode.
These are the full-precision benchmark results
PinchBench
Sage-A1.5 โ 90.0
Kimi K3 โ 88.2
GLM-5.3 โ 86.6
GPT-5.6 Sol โ 87.3
FrontierScience-Olympiad
Sage โ 81.0
Kimi โ 69.0
GLM โ 67.0
GPT โ 73.0
SWE-Pro
Sage โ 61.2
Kimi โ 63.3
GLM โ 64.6
GPT โ 64.6
โ ๏ธ Those benchmark scores are from the BF16 checkpoint, not the Q4.
But the same model now has an official 21.7GB GGUF with llama.cpp / Ollama / LM Studio support.
Link in ALT
This might be my new favorite all-purpose local AI model. Fits nicely into 16GB of VRAM for an RTX
5060/5070TI.
The new SparkX2.6-4B beats Qwen3.5-9B in 80% of shared benchmarks and 97% of Gemma4-12Bs.
Stats
๐พ official Q4_K_M โ 3.54GB
๐ง only 4B parameters
๐ native 1M context
๐ญ none / low / high thinking budgets
๐ค trained for local agents
๐ Apache 2.0
It already runs with:
โ๏ธ llama.cpp
๐ฆ Ollama
๐ฅ๏ธ LM Studio
๐ MLX
๐ vLLM / SGLang
In the team's reported benchmark suite Averages:
Spark-X2.6-4B โ 63.3
Spark-X2.5-4B โ 52.1
Qwen3.5-9B โ 49.1
So they're claiming a 4B model beats their tested 9B comparison overall.
For normal local-agent contexts, though, a ~3.5GB reasoning/agent model with this runtime support is very interesting.
Link in ALT
Underdog released Bark 3.0, and it's only a 163MB download. Runs locally w/Pytorch+CUDA or MLX
On their Open ASR Leaderboard evaluation, Bark wins
๐ถ Bark 3.0 โ 4.94% WER โ 163MB
๐ข NVIDIA Parakeet โ 4.94% โ 2.51GB
๐ Whisper large-v3 โ 6.04% โ 9.26GB
So Bark matches Parakeet's average accuracy while being ๐ ~15ร smaller
And beats Whisper while being ๐ ~57ร smaller
On recorded meetings check out these stats
Bark โ 7.82% WER
Parakeet โ 9.42%
Whisper large-v3 โ 13.63%
And this isn't CUDA-server-only.
๐ fully local
๐ฑ phone-class
๐ MLX + Core ML
๐ง Apple GPU / ANE
๐ช CPU + CUDA paths
๐ CC-BY-4.0
I've checked and no GGUF. It uses its own compressed Bark format plus MLX/Core ML/Transformers runtimes.
Bark 3.0 Plus is only 180MB and improves average WER to 4.76%.
โ ๏ธ English only
Link in ALT
DeepSeek V4.1 Flash 1017 is now the #1 open model on Agent Arena leaderboard.
Hosted
๐ #1 open Agent Arena
๐ฐ ~$0.19 median/task
๐ MIT weights
Localmaxxers are already running the same 1017 checkpoint on 2ร RTX 3090s + 128GB DDR4.
And getting
โก ~30 tps decode
๐ 301โ470 tps prefill
This is a monster model w/552B MoE backbone
๐พ + ~196B Engram memory
๐ 1M native context
The local build squeezes the experts to ~2.03 bpw and uses exact routing, with the big Engram tables on storage.
โ ๏ธ Those 2x3090s use an aggressive quant. It does not mean the local 2-bit version retains the exact Arena score.
But that's still kind of nuts.
Link to the 2x 3090's test in the ALT
The RTX Spark laptop configuration I thought would signal real demand is the expensive one SKU.
Microsoft Surface Laptop Ultra w/๐พ 128GB unified memory, full RTX Spark N1X, $5,899.99.
Microsoft currently lists it ๐ Out of Stock
ASUS also said demand for its RTX Spark systems โgreatly exceededโ expectations and its initial planned volume had already been fully preordered by channel partners.
MSI said its first N1X batch was nearly sold out and was asking Nvidia for more supply.
People may be buying RTX Spark for the thing normal Windows laptops can't do. 1) 128GB unified memory 2) CUDA
Today feels like an inflection point for Windows Local AI.
RTX Spark laptops are officially available and not just Surface laptops.
Nvidia now has RTX Spark Windows laptops shipping from these guys too
๐ป ASUS
๐ป Dell
๐ป HP
๐ป Lenovo
๐ป Microsoft
๐ป MSI
And at the high end you can now put this in a reglar laptop
๐ง up to 128GB unified memory
โก up to 1 PFLOP FP4
๐ฎ Blackwell RTX GPU
๐ง CUDA
๐ช Windows 11 ARM
That really isn't something we've had before.
128GB-class unified memory + CUDA in a Windows laptop.
Day-one agent support includes
๐ค Hermes
๐ฆ OpenClaw
๐ง WorkBuddy
๐ค Pi
Artificial Analysis found an RTX Spark laptop came within roughly 3โ8% of DGX Spark on its tested agent workloads.
While 128GB lets it run models that simply won't fit in a 32GB RTX 5090.
Prices start around $2,599, although that buys a lower-memory configuration.
๐ฐ Surface Laptop Ultra 128GB โ $5,899
๐ฐ ASUS ProArt P16 128GB โ $6,999.99
๐ Link in ALT
Tencent has released its own official GGUFs of Hy-Memory-30B-A3B.
And they didn't just throw up some quants.
They benchmarked them too
๐พ Q4_K_M โ 18.24GB โ 56.49%
๐พ Q6_K โ 24.71GB โ 58.78%
๐พ Q8_0 โ 31.99GB โ 57.63%
All tested with llama.cpp on the same LoCoMo memory benchmark setup.
๐ Q6 actually scored highest.
And the Q4 means this specialized long-term memory writer is now officially an ~18GB local model.
Tencent even documents running it directly with llama.cpp:
llama serve -hf tencent/Hy-Memory-30B-A3B-GGUF
This is the sort of Local AI architecture I want to experiment with.
Link in ALT
Holy ๐ฉ. Tencent released a Local AI model dedicated to deciding what your AI agent should remember.
This might become my favorite memory companion local AI model to pair with Qwen3.8-27B
And community Q4 builds are already around ~17GB. ๐
Hy-Memory-30B-A3B is not meant to be a chatbot but rather a dedicated long-term memory writer.
It
๐ง reads new conversations / agent activity
๐ฏ decides what is actually worth remembering
๐ turns evidence into precise persistent memories
๐ updates memories when facts change
๐๏ธ avoids storing irrelevant information
๐พ maintains the persistent memory bank
Why?
Without persistent memory, your agent either:
๐ง forgets useful information when the session/context ends
๐ keeps stuffing old conversations back into context
A memory system gives your agent a compact long-term store that survives across sessions.
Hy-Memory is the model deciding what deserves to go into that store.
And today memory systems often use a general-purpose LLM for that job.
Tencent instead trained a sparse specialist specifically for it.
And Tencent reports it beats much larger frontier models at that specific task
๐ Hy-Memory-30B-A3B โ 55.49
Claude Opus 5 โ 54.62
DeepSeek V4.1 Flash โ 53.64
Qwen3.6-35B-A3B โ 50.37
EverMemOS ranked #1 there at 56.89.
Specs
๐ง 30B total
โก ~3B active/token
๐ Apache 2.0
๐ open weights
โ๏ธ vLLM + SGLang
๐ OpenAI-compatible API
And community quants are already arriving:
Q4_K_M โ ~17GB
Q5_K_M โ ~20GB
Q6_K โ ~23GB
EXL3 4.08bpw โ ~15GB
That puts it roughly in the same memory class as a Qwen3.8-27B Q4.
Except Qwen3.8-27B is a dense general-purpose model.
Hy-Memory activates only ~3B parameters/token and was trained specifically for remembering.
๐ Link in ALT
I think Sipeed may be skipping the MaixStation DGX-Spark-Killer configuration we would want most.
They're planning these memory configs
๐พ 32GB
๐พ 128GB
๐พ 512GB
But no standard 64GB model.
Sipeed's reasoning is
32GB โ dense 27B models
128GB โ 125B-class MoE models
64GB โ โwould run neither betterโ
A ~$2,000 64GB box with 273 GB/s bandwidth would be more interesting to me than a ~$2,000 32GB box.
64GB opens up a very different Local-AI options including, larger quants, larger context/KV cache, and more room for agents.
Especially when Nvidia's new 64GB DGX Spark starts at $4,999.
Sipeed says enterprise customers can custom-order 64GB.
Am I wrong? Would you buy 32GB, 64GB or jump straight to 128GB?
Update: Sipeed gave Tomโs Hardware the MaixStation price I was missing.
The 128GB MaixStation is expected to cost about $3,999.
Current 128GB DGX Spark โ $6,950
MaixStation 128GB โ ~$3,999
Thatโs roughly
๐ฐ $2,951 less
๐ ~42% cheaper
Same:
๐ง 128GB LPDDR5X
๐ 273 GB/s memory bandwidth
Sipeed has shown newer Qwen 27B runs claiming
๐ฅ +25% prefill
๐ฅ +30% decode
But in its email to Tomโs, Sipeed put the like-for-like comparison at
โก ~10% faster prefill
โก ~10% faster decode
using FP4 with speculative decoding OFF.
They say the larger gains are newer runs that aren't official yet.
There are a few more interesting details now too
๐ ~40W typical load
๐ USB-C PD โ Sipeed says even a 65W power bank can run it
๐ 11 ร 11 ร 3 cm
๐ 10G SFP+
๐พ standard M.2 2280 SSD
๐ up to 256 Gbps C2C, with Sipeed claiming up to 8 nodes without an external switch
Kickstarter is planned for the end of 2026, with Sipeed telling Tomโs units should ship within roughly 3 months after the campaign closes.
๐ Links to TomsHardware article in ALT
Okay, now people are benchmarking Nvidia's 24GB Bolt on consumer hardware.
Someone on Reddit put the official Nemotron 3.5 Bolt 74B-A9B IQ2 on an RTX 5090 ๐ช with llama.cpp.
Without speculative decoding
โก 166 tps
Add NVIDIA's LiLiCorr DFlash drafter
๐ 343 tps median
That's about a 2.07ร jump.
And prompt processing
๐ฅ ~5,200 tps
VRAM is interesting too
๐ง Bolt only โ 21.0 GiB
๐๏ธ + vision projector โ 22.7 GiB
๐ + projector + LiLiCorr โ 25.0 GiB
So the regular multimodal model really can squeeze onto a 24GB GPU.
Link in ALT.
Nvidia has delivered a Qwen3.6-35B killer and it's a 74B โ 24GB model, actually, today.
And now we have real numbers. ๐
Nemotron 3.5 Bolt 74B-A9B
๐ง 74B total / 9B active
๐พ 22.1GB quantized weights
๐ข ~2.06-bit experts + QAT
๐ up to 256K context
๐๏ธ text + images + video
๐ค reasoning + tool use
๐ฆ official GGUF + llama.cpp
And NVIDIA reports on RTX Spark:
โก ~84 tps
That's a 74B multimodal agent model running locally.
Check out how much quality survived the 2-bit compression. BF16 โ 24GB IQ2:
๐ฏ PinchBench
90.35 โ 87.82
๐ BrowseComp
70.0 โ 66.75
๐ง GPQA Diamond
88.83 โ 86.43
๐ป Terminal-Bench 2.1
53.93 โ 50.98
Nvidia calculates
๐พ weights โ 22.01GB
๐พ cache + activations โ 0.87GB
๐พ LiLiCorr drafter โ 0.96GB
Total โ 23.83GB
The vision encoder/projector are not included in that number.
So Nvidia's official llama.cpp recipes actually target
๐ข 32GB+ GPU โ 65K context / 28GB config
๐ฅ 48GB+ GPU โ 256K context / 32GB config
๐ Links in ALT
























