🇨🇦 Obsessed with all things Ai Agents, Loops, LLMs, Fine-Tuning https://teapawt.net/t.co/9thyt19bL5 - Entrepreneur. Builder.
Canada
Joined May 2026
- Tweets2K
- Following462
- Followers436
- Likes4.6K
GPTware retweeted
Today we are happy to share one of our first open research releases: Latent Sparse Attention (LSA), a step towards subquadratic attention that matches accuracy of full attention at ultra-long context.
Sparse attention attends to only a few tokens per query, but it has to find the right ones. At 10M tokens, that search takes 98% of the attention layer's time. LSA makes sparse attention up to 22.7× faster and full attention 72.4x faster.
With @kubicek_ales @caciolai @PengLiangzu @SukritSumant @Ben_Hoov @FraserGreenlee @RMBattleday @radim_urban @thoefler @jcrwhittington
Coming next - DSV4.1-Flash-1017 on 1x RTX Pro 6000 MaxQ + 1x DGX Spark - No weights offloaded. Mixed EXL3 expert quants at overall 2.945 bpw
So far single at 100 tok/s code & 50 tok/s prose
I'll be able to improve the 1 RTX + 2 GB10 setup as well
GPTware retweeted
Georgia Tech built ARK for the ugly part of disaggregated LLM serving: shipping a huge KV cache from the prefill GPUs to the decode GPUs without ECMP hashing two elephant flows onto the same spine.
My take: Those transfers last hundreds of milliseconds to seconds, so a hash collision can sit on one link while other spines stay idle. ARK has senders pick source ports whose hashes put concurrent flows on different spines, with no switch changes and no reordering on the receive side. In their RDMA packet simulations, mean flow completion time dropped by as much as 27.3 percent and the P95 by as much as 34.0 percent at moderate load, which cut mean time to first token by up to 12.9 percent. At higher load the fabric saturates and the gain shrinks. The caveat is that this is simulation, not a production fabric. Still, if you are splitting prefill and decode across boxes, the path the KV takes probably matters as much as the kernels.
arxiv.org/abs/2610.41160
GPTware retweeted
50% OFF service on medai.li! Pay @ashxhart's Mac 512gb 2Tb fund: ko-fi.com/ashxhart
Your top supporters from the last 90 days: 🥇 1. iamMrDuncan 🥈 2. Dennis Rye 🥉 3. Adam 4. MedAI.li Natalia Sokolova 5. Ko-fi User ko-fi.com/ashxhart/leaderboa…
GPTware retweeted
1/3
A CPU inside an LLM is one thing, but running another LLM inside an LLM would be taking things a bit too far, wouldn't it?
Anyways, here's an LLM running inside an LLM : QwenPU, an LLM (@karpathy's stories260k) running on an NPU compiled inside Qwen 3.5-4B's weights.
GPTware retweeted
Reddit's CEO said people are getting answers from Google's AI Mode instead of getting sent to Reddit. Google has already said AI Mode is cannibalizing its own search business. Millions of publishers and companies built their whole discovery path on that one handoff.
Once the intermediary answers the question instead of routing it, the traffic goes with it, and you find out how little of that demand you actually owned.
Traditional companies that treat Google as a permanent pipe are going to keep learning that after the quarterly report, not before.
There are many that will be happy to see Reddit go.
The collapse of Reddit will be studied.
After reporting earnings, its stock collapsed 30% in a single day.
It's down 44% in 2 weeks.
But the most important detail is what was revealed in that earnings call.
The CEO said AI is killing Google search.
People no longer get sent to Reddit from Google. They merely get their answers from the AI Mode itself.
Even Google confirmed this themselves with their disastrous results - AI mode is cannibalising their search business.
Pretty soon, no one will visit Reddit voluntarily. Its traffic will continue to crater.
The worst part?
Reddit isn't alone. There are millions of companies and publishers that rely on search traffic.
That's what happens when you rely on a single funnel for discovery.
🧵DeepSeek-V4.1-Flash-1017 Uncensored. Just 2× RTX PRO 6000 Max-Qs - only Engram offloaded to RAM.
Mixed Expert bpw EXL3 + DSpark. Hot experts get higher bpw, cold experts lower.
Single-user decode at 24k context:
⚡ Structured: 185 tok/s
⚡ Math: 162–164 tok/s
⚡ Code: 130 tok/s
⚡ Prose: 113–114 tok/s
⚡ Multilingual: 91–93 tok/s
Concurrent requests → aggregate tok/s
2k / 24k prompts
2 requests → 147 / 147
4 requests → 176 / 163
8 requests → 255 / 230
github.com/GPTwareAiTeam/DSV…
Base model by @dealignai - huggingface.co/dealignai/Dee…
Thank you @turboderp_ for EXL3!
Added several parts from the great work of @diffbot
Hot and cold experts determined from @coldniko routing-frequency profile
GPTware retweeted
Some lessons learned networking DGX Spark's to DGX Station using Microtik CRS804:
al-engr.com/networking-dgx-s…
GPTware retweeted
The question that is stopping me from buying an M5 Ultra: why does a Mac with 1.2 TB/s of bandwidth lose to DGX Spark at long context and with multiple users?
With one user, decode is memory-bound. Every token means reading all the active weights once, and the GPU mostly waits for data. More bandwidth means more tok/s, and that's where the Mac shines. 👍
Batching changes everything. With 8 users you read the weights once and use them for 8 tokens. Same bytes, 8× the math. Throughput scales almost for free… until compute becomes the bottleneck. 👀
That's where the Mac hits a wall: good bandwidth, lower low-precision compute. The Spark's Blackwell tensor cores with native FP4 give far more FLOPs per GB/s of bandwidth, so it keeps scaling while the Mac stops.
Real numbers on DeepSeek V4.1 Flash with TensorFold, total tok/s from 1 to 8 users (chart below):
2× Spark: 77 → 105 → 150 → 194 (@rwojo)
1× M5 Ultra: 63 → 78 → 98 → 99 (@TechMDAI, first runs, capped at 4 slots)
Long context is even worse. Prefill is pure compute, attention grows with context length, and the KV cache is per-user: no way to share it. On DeepSeek V4 Flash the Mac's decode drops from ~60 to ~27 tok/s, while 2 Sparks hold ~51 (tests by @satsuki2694).
Bottom line: the Mac is unbeatable for fitting huge models (MoE) in a single box, but the Spark wins when you need compute: long prompts, many users, agents. Different tools, different jobs.
I run both, and I was planning a deep dive on the M5 Ultra to really understand its limits, but at this price I don't know if it makes sense spending all these money just to make tests and share data. I'll try to figure out something, otherwise I'll wait for others to test these concurrency limits in depth. 💪
GPTware retweeted
How much performance do you actually lose when capping a DGX Spark and where are the sweet spots?
As a follow-up to a previous post I decided to test it using my 2 x Sparks and likely the best model and quant currently available @MiaAI_lab's Deepseek V4.1 1017 EXL3.
Depending on your preference the sweet spots are 2200 MHz for close to original performance or 1900 MHz if you seriously want to reduce Temperatures and power consumption.
Anything below 1900 MHz and the Sparks start fighting with themselves.
Honestly, I think we should all just trust Sam all the other Ai CEOs in everything that they do and just support them.
youtu.be/siwG1P1alv8?si=w_oX…
sam: local inference is a terrible allocation of scarce resources
“a lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we're going to be able to run them on our computers, and so we're not going to need all the stuff in the data centre
but from a global allocation perspective, actually it's quite terrible to have everybody, like, have a bunch of the world's limited supply of DRAM sitting in their own computer and being used five percent of the time
you want maximum utilisation in the cloud if you're trying to deal with this very supply-constrained environment
somehow many people's intuition is the opposite of that”
Whoever TF trusts this guy or any of the major Ai companies with their data is entirely cooked.
sam: local inference is a terrible allocation of scarce resources
“a lot of people want local inference, and somehow they assume that the problem is solved now because we have open-source models, we're going to be able to run them on our computers, and so we're not going to need all the stuff in the data centre
but from a global allocation perspective, actually it's quite terrible to have everybody, like, have a bunch of the world's limited supply of DRAM sitting in their own computer and being used five percent of the time
you want maximum utilisation in the cloud if you're trying to deal with this very supply-constrained environment
somehow many people's intuition is the opposite of that”
GPTware retweeted
🐳鲸鱼兄弟们！DeepSeek 要放大招来啦！
V5、V5-open、V5-next都端上桌了。
deepseek Harness 的最新代码中发现已经有相关的代码了。
看来离发布也不远了。
github.com/deepseek-ai/deeps…
