New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s! This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it. Specifically: -(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time. -Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used. -Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2. -Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step. -Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params. -Hand-rolled flash attention for 64 dim heads. There are several additions that add accuracy too: -(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15. -(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application. -A couple additional dynamic skip connections in the network. The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large, only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead. github.com/KellerJordan/modd… As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes: hyperstition.cc/training-nan…

We're publishing our most detailed threat intelligence report to date. It covers how people tried to misuse Claude—for cyberattacks, influence operations, surveillance, biology, and building weapons—and how we found and stopped them. We disrupted every operation in the report, and used the lessons from them to strengthen our safeguards. Where appropriate, we also shared what we found with authorities and other AI companies. These cases are not typical: we’re highlighting some of the most sophisticated misuse we’ve seen. But they’re especially important to discuss, because they show us where AI misuse is headed, where our safeguards work, and where they need to improve. We’re publishing this report so others can spot the same activity on their own platforms, and so we can give the public a clearer view of how emerging threats develop. Read the report: anthropic.com/threat-intelli…

Qwen3.8-Flash-Next Drops TOMORROW And I'm Deploying It LIVE 🚨 The architecture is wild: 125B brain, only 6B active per token, PLUS a 51B lookup table of pre-baked phrases. It's not more brain, it's flashcards 🧠⚡️ The model looks up what it already knows for FREE and spends its compute purely on thinking. Same quality as their flagship at 1/9th the training cost. Their last model this size ran 10B active. This runs 6. That's the speed unlock 📈 Tomorrow 11 AM I'm going live to crack this on the DGX Sparks. FP8 on TP2 first, then the community quant hunt begins. Can we fit it on ONE Spark? Can it run on four 3090s? We find out together 🔥 Stay tuned. You already know the numbers get posted here first 🎯

Qwen 3.8 Flash Next is releasing Tomorrow. 125B paramters +51B N-gram and 6B active. Its based on the next generation Qwen 4 architecture. Qwen 4 is coming

Hey everyone — our high-performance MSA kernel library is now open-source. The M3 weights are expected to drop this Friday. Thanks for waiting! Github: github.com/MiniMax-AI/MSA Paper：github.com/MiniMax-AI/MSA/bl…

not sure how you can say the first major innovation since adam was muon and not shampoo. although I'll admit that muon is much simpler to implement in practice! unless you are saying muon is a special case of shampoo with b2=0?

🚀 Scaling embeddings, not just experts—introducing a new path for efficient LLMs. Key Finding: In high-sparsity scenarios, N-gram embeddings yield a better Pareto frontier than just adding more MoE experts. Therefore, we introduce LongCat-Flash-Lite—the first opensource model built on this insight. ⚙️ 68.5B Total Params(37.13B non-embedding) | 2.9B~4.5B Active 📊 High Performance: SWE-Bench 54.4 | τ²-Bench 72.8 | TerminalBench 33.75 📃 256K Context Window (YARN-powered) ✨ Optimized for Agentic/Coding, strong in general reasoning ⚡ ~700 tokens/s peak inference speed The result: Achieves competitive performance within its scale at a significantly lower cost and latency. Hugging Face: huggingface.co/meituan-longc… Tech Report: huggingface.co/meituan-longc…