Rusty retweeted
Important updates and fixes have been pushed to DeepSeek V4.1 Flash for 4x DGX Sparks ⚡️
This update includes 7 community PRs!
Performance
- Chats with images: resending a 35-image chat dropped from 20 s to 1 s (20x faster)
- No more waiting: a short question asked during a long 51k-token prompt now answers in 0.3 s instead of ~13 s
- CPU use per machine dropped from ~310% to ~0.2% (about 12 CPU cores freed) when on idle
More reliable
- Long prompts no longer time out clients (a keep-alive signal every 15 s)
- Safer start/stop scripts
- The test tools can’t report a false “pass” anymore
Also, /metrics now shows how many prompt tokens came from the cache (prompt_tokens_cached_total), so dashboards can tell cache hits from real prefill work
Thanks to @easthambuilds, @krunkosaurus, @uchkw, Manuel (manuel1618), and SvangenStudios, plus upstream TensorFold work by @jayleaton, @ashxhart and @BobClawblaw 🙏
Get it here:
mia-ai.net/models/DeepSeek-V…
Rusty retweeted
2,100+ tokens/sec per user on DeepSeek V4.1 Flash, at full model quality, on 8× AMD Instinct MI355X. 🚀
Congrats to @lithos_ai on pushing the frontier of inference with persistent megakernels on AMD. Open hardware + instruction-level control = room to innovate.
2,100+ tokens per second per user on DeepSeek V4.1 Flash, at full model quality, on eight AMD Instinct MI355X GPUs.
In our benchmarks, the Lithos engine delivers:
→ Up to 3.4× faster inference for DeepSeek V4.1 Flash
→ Up to 2.5× faster inference for Kimi K3
Compared with tested open-source engines on the same eight GPUs.
The key challenge is reducing the coordination tax: time spent waiting between kernels and across GPUs. Our approach combines persistent megakernels, overlapping execution, and communication inside computation.
Our technical deep dive covers:
- Decode speed, cost per token, and latency up to 1M context.
- How Lithos engine reduces launch overhead and overlap computation with communication across GPUs.
- How MI355X’s register capacity and direct instruction-level control enable hardware-specific optimizations.
- What our roofline analysis reveals about opportunities for further speedups.
Read the full analysis: lithosai.com/blog/ultra-fast…
Rusty retweeted
Tech based comedy, streaming every week. Follow along and stay tuned!
youtu.be/N9LhOpK1-Wg
Rusty retweeted
Looks like next Zhipu models is going to be from GLM-5.5 family
GLM-5.5-Flash is now listed in OpenCode.
opencode.ai/data/zhipu/glm-5…
Rusty retweeted
I spent the last 14 hours (sleepless btw) thoroughly analyzing the Cohere post and the commentary around it
I conclude that Cohere's research and their reaction can be summarized as follows:
In the spirit of transparency and openness, we are releasing the traces so developers can test this out for themselves. We noticed this behaviour without prefilling with the following prompt, but this is rare. If you want to reproduce it, you can use this prompt with prefilling.
Open models make for better science and are part of why this problem came to light at all: developers running Qwen3.8 on their own machines spotted the strange links, and having the weights let us study the behaviour in depth.
We're encouraged by the thoughtful discussion our research has sparked across the AI and security communities. It's our belief that open-weight models are an important part of the future of AI, and we've recently released several under the Apache 2.0 license.
Trace, raw tokens, & minimal reproduction script: huggingface.co/datasets/Cohe…
Rusty retweeted
When I tell you dip cause it’s ICE TIME, bxtch it ain’t the fake feds
Urghhh!!!! 🧊🧊🧊🧊🦉🦉🦉
Rusty retweeted
A metro Atlanta woman shares how her son invited her to a Drake concert, where Drake gifted her $20,000 to help bring her dream of starting a nonprofit to life.
Since then, she says her organization has provided over 3,000 lunches and helped 96 struggling families. ❤️
