Amol (@aumbark) profile picture

LLM Perf @ E2E Networks Building https://teapawt.net/t.co/AxmWH0CVkj In the past: founder hyperML (compute platform)

Joined March 2009
Photo by Amol (@aumbark)Photo by Amol (@aumbark)Photo by Amol (@aumbark)Photo by Amol (@aumbark)Photo by Amol (@aumbark)Photo by Amol (@aumbark)Photo by Amol (@aumbark)Photo by Amol (@aumbark)Photo by Amol (@aumbark)Photo by Amol (@aumbark)
The BlazeInfer models page is up. Open-weight models, priced per token, running on GPUs we own in India. DeepSeek V4.1 Flash: $0.045 in / $0.50 out GLM 5.3 Flash, 1.25M context: $0.075 / $0.25 Kimi K3: $3 / $15 $15 credit to try it. blazeinfer.ai/models @e2enetworks @aumbark
4
11
21
69,423
Sure. Pls reach out to us via the platform itself and we can setup credits for you. There is a link to “Talk to team” once you sign up. The docs link is at blazeinfer.ai/docs for integrations @aumbark fyi
1
1
81
I agree with one thing. Making linear attention work at scale takes talent, and not every lab has it. The rest doesn’t hold up. In order: (1) “Why are the Western labs choosing to drop context instead of compressing it?” The first rigorous, controlled, large-budget test of compression was what we did at NVIDIA’s in June 2024(arxiv.org/abs/2406.07887). We trained 8B Mamba, Mamba-2 and Transformer models on the same data for up to 3.5T tokens. Additionally we trained an 8B hybrid of 43% Mamba-2, 7% attention and 50% MLP and we showed it beats the 8B Transformer on all 12 standard tasks (+2.65 points on average). Interestingly this hybrid variant was up to 8× faster at generation, and matched or beat the Transformer across 23 additional long-context tasks. This is seven months before MiniMax-01, fifteen before Qwen3-Next, sixteen before Kimi Linear. And since then NVIDIA has been shipping a hybrid architecture, with modifications, repeatedly. Good examples are Nemotron-H, Nemotron Nano 2, and Nemotron 3 (Nano, Super, Ultra) which is a Mixture-of-Experts hybrid Mamba-Transformer family with context up to 1M tokens, post-trained with multi-environment RL. Nano keeps exactly 6 attention layers out of 52 and the other 46 are Mamba-2 and MoE blocks. Super is 120B total / 12B active, pre-trained on 25 trillion tokens. IBM’s Granite 4.0 (October 2025) interleaves Mamba-2 and attention layers at 9:1. To this list, you can also add Jamba (AI21, March 2024, a 52B production hybrid), RecurrentGemma (Google, April 2024), Codestral Mamba (Mistral, July 2024) and Phi-4-mini-flash (Microsoft, July 2025). (2) “SWA is a lossless local transformation, therefore superior.?” SWA is lossless inside the window and zero outside of it and it is exactly why it fails so badly. (arxiv.org/abs/2509.24552) for example shows that larger sliding windows hurts the long-context performance. This is because a precise local softmax lets the model avoid training the long-term memory it would otherwise rely on. They show that a hybrid with a 128-token window beat the same hybrid with a 2048 window by 16 NIAH points (88.9% higher recall), and at 131k the short-window models still retrieved about 30% of needles while the 2048-window model was near 0%. And the same degradation from larger training windows shows up in local-global stacks that alternate SWA with full attention ( e.g. the GPT-OSS/Gemma recipe). (3) “Compression is strictly harder to serve.” This was true in 2024. But now vLLM, FlashInfer, SGLang and TensorRT-LLM serve hybrid linear models extensively and without any issues. The most exciting part is the recent FlashInfer additions that let you use chunked gated-delta-rule prefill and gated-delta-rule decode kernels, with an SM100 (Blackwell) path. It even supports much newer models like Gated DeltaNet-2 and also lets you use efficient kernels for training. Also, let's not forget ReplaySSM (arxiv.org/abs/2610.13778) which enables a very optimized speculative decoding path for linear attention. (4) “If you write all the kernels anyway, why not roll your own attention in the meantime?” On NVIDIA hardware, linear attention is just as free as attention. The Mamba CUDA kernels exist. flash-linear-attention, the Triton library that the GDN and KDA layers in Qwen3-Next, Qwen3.5 and Kimi Linear use, was written for NVIDIA GPUs, and vLLM and SGLang use it directly instead of reimplementing it. Megatron has supported Mamba-2 training since 2024. A Western lab on H100s that wants a hybrid doesn’t need to write a kernel. It can just import it. So kernel cost can’t be what keeps them on using the SWA. Also consider that decode is memory-bandwidth-bound, so the cost of every generated token is dominated by reading the KV cache, which grows with context length. A constant-size state removes that growth. That’s why Nemotron is mostly Mamba-2, and why Nemotron 3 Super gets 2.2× the throughput of GPT-OSS-120B at 8K in / 16K out. (5) “For your first release, it’s a much safer option to ship SWA. Everyone knows it’s working just fine.”? Safe compared to what? “Safe” should mean proven in production at scale. By that standard the hybrids are the safe choice and SWA is the bet. Look at what has shipped on Gated DeltaNet alone. Qwen3-Next, then Qwen3.5 from 27B up to 397B-A17B, then Qwen3.6. Kimi Linear, then Kimi K3 which is a 2.8 trillion parameters, open weights, 1M-token context, a 3:1 hybrid of Kimi Delta Attention and gated MLA. GLM-5.3-Flash on the same KDA layout. KDA is Gated DeltaNet with channel-wise instead of scalar gating. Right now, three frontier labs run their model in production with 1M context, RL post-training, etc. all on the same layer. Meanwhile SWA is the architecture with the documented issue. It fails past the window and drags any hybrid it sits in down to pure-SWA recall. And when MiniMax reached for an SWA hybrid as the fallback in M2, it failed and they went to full attention instead. In my view, shipping SWA for a first release in late 2026 isn’t conservative. It’s picking the layer with a known failure mode because it needs no new code.
Given Meta also shipped SWA, I decided to go deeper into the "is linear attention a Chinese psyop" rabbit hole. Plus, as a person who spent some time working on architectures. 1. I would frame the question the following way: “Why are the Western labs choosing to drop context instead of compressing it?" If you think about it this way, there is a clear advantage of SWA: it preserves the full signal from the local context, while linear attention compresses everything, losing some information about local context as well. Given that the local memory is roughly equal, as well as the time complexity and assuming that you believe in "only global signal layers and only local signal layers", SWA seems to be superior. It's a lossless "local signal" transformation. Thus, the discourse about the West not adopting a "superior" architecture is not entirely correct IMO. 2. Compression is strictly harder to serve. We saw this a lot internally; inference engineers hate it. One of the reasons is all the stuff is optimised around KV cache, and recurrent state is breaking lots of nice properties: prefix state now needs snapshots at block boundaries, spec dec needs per-token rollback. RL becomes problematic. So why do the Chinese seem to adopt compression anyway? 3. Hardware but in a bit of a weird way. On Hopper/Blackwell, you already have FA; on Ascend, etc., one needs to write all the kernels in Triton or TileLang. If you write all the kernels anyway, why not roll your own attention in the meantime? You're hiring a lot of kernel people that need to write all this stuff. 4. Selection bias. If you take a closer look at the Wester labs that ship SWA (usually OSS ones, we don't know what Ant/OAI/GDM ship to the closed models), they are publishing their first or second model iterations. For your first release, it's a much safer option to ship SWA. Everyone knows it's working just fine. (This was our own motivation at Mellum). They may switch to something smarter down the line and I believe this already somewhat happened, take Nemotron for example. The only case I'm aware of people actually going backwards is Poolside??? I'd expect OSS labs to start digging into compression along with subsequent releases. 5. It seems that talent plays a big role. Linear attention field is historically pretty much Chinese and Chinese labs seem to borrow the ideas from each other. Wester labs are mostly GDM/OAI alums, which are likely used to SWA (see GPT-OSS and Gemma). To summarise, I expect linear attention (and whatever DeepSeek is cooking) to start slowly being adopted in the West with the next model releases and when the inference infra catches up.
23
28
5
351
28,613
A #Christmas gift from the Hashnode community to the #devcommunity on Twitter. 🙌 Here's our first FREE community-contributed book 📕 "Become A Better Dev In 2021" 40 helpful guides written by 30⭐️ awesome writers from @hashnode 🧑‍💻 👩‍💻 Download ⬇️ hashnode.com/better-dev-2021
5
60
21
166
Repetition unlocks value. The value of your first workout increases the more you exercise. The value of your first article increases the more you write. The value of your first conversation increases the longer you stay in the relationship. Day One continues to compound.
50
788
104
4,462
1/ Tuesday was my last day as CEO of @CircleUp. I’ve been CEO since starting the co. in 2011 with my co-founder @roryeakin. This is a thread about what happened, why and my emotions about it. For more detail: ryancaldbeck.medium.com/tran… Much of this I have never talked about.
218
338
360
3,311
I looked at 200 tools for developing & deploying machine learning applications: - how the market evolved over time - the difference between ML applications that traditional software engineering applications - open-source vs open-core business model huyenchip.com/2020/06/22/mlo…
13
394
37
1,482
115 KiB of our binary is #golang compiler-generated equality funcs for structs used as value types. Make it all go away with this one dirty trick: Make your struct types not comparable by adding an unnamed 0-sized array of func pointers. Now it can't ==: _ [0]func() 🤮 🤷‍♂️
19
91
6
546
More clarity about where results come from in the paper: today we are rolling out our new results interface that directly links results to their original tables in arXiv papers! Example: paperswithcode.com/paper/sel…
2
10
1
45
My father passed away in Basti, UP. I reside in Vellore, TN. Please guide me so I can get a travel pass from TN to UP, to perform the final ritual. @PMOIndia @HMOIndia @narendramodi @AmitShah @MoHFW_INDIA @CMOTamilNadu @CMOfficeUP @NHM_TN @DrBeelaIAS @Vijayabaskarofl
46
86
21
37
My TEDx talk has now been posted. Among other things, I share some thoughts on the question "when will I ever use this" as it comes up in math education. youtu.be/s_L-fp8gDzY
55
452
87
2,684
cloudskew.com now offers professional services centered around #Azure 1⃣ Need architecture proposals created? 2⃣ Need custom deployments & solutions? 3⃣ Need an architecture audit for best practices? Reach us: services at cloudskew dot com PS: Retweets appreciated.
1
22
1
21
cloudskew.com is now in public preview. Create slick Azure architecture diagrams for free. Please give it a try! PS: Retweets appreciated. Please help spread the word.
23
398
26
533
How To Be Successful (At Your Career, Twitter Edition)
217
3,988
503
14,678
The breadth of AWS is just ridiculous—take any software company, and it's likely AWS has a direct competitor. As a standalone company, your product better be 5x better. Or else the AWS default product wins.
34
182
34
688
Remember that video about how block collisions can compute the digits of pi? A friend, Adam Brown, just showed that the math underlying this is actually identical to the math behind a very famous quantum search algorithm (Grover's): arxiv.org/abs/1912.02207 Genuinely crazy!
70
2,909
309
10,437
If you spend a lot of time creating #azure architecture diagrams, I've created a free app for you. It's in early preview and I'd like to roll it out slowly. Please DM me and I'll reach out with details by Dec 31st.
3
6
1
19
I wrote an 8k word doc on machine learning systems design. This covers: 1. Project setup 2. Data pipeline 3. Training & debugging 4. Serving with case studies, resources, and 27 exercises. This is the 1st draft so feedback is much needed. Thank you! github.com/chiphuyen/machine…
75
1,576
74
6,811
Obama lecturing young people to stop being social justice media warriors and to go out and actually do some thing in the world #Priceless
149
3,683
677
12,867