Alexander Long retweeted
It's extremely rare to encounter such a thoughtful blog on pretraining and scaling. I really liked the profiling plots depicting a clear yet sufficient picture.
What is not covered in the report is how we scaled model training to be efficient on 768 B200 GPUs.
I explain it in detail here: aleph-alpha.com/en/blog/scal…
A how to guide to scale up model training without wasting your GPU hours. Even if you only have 1 or 2 GPUs and cannot scale, our optimisation philosophy can still make your GPUs go brrr.
I couldn't explain ALL the details of our method in the blogpost, since the big launch came after. You can connect the dots now, the 30B MoE architecture was in fact Kolibri Origin all along.
Alexander Long retweeted
1501: Pope Alexander VI criticizes Gutenberg’s printing press safety
“The art of printing can be of great service in so far as it furthers the circulation of useful & tested books; but it can bring about serious evils l..it will, therefore, be necessary to maintain full control”
Pope Leo criticises Nvidia’s Jensen Huang over AI safety ft.trib.al/azwdiGc
Alexander Long retweeted
We are fortunate to have 3 papers accepted to this years NeurIPS.
The first is AsyncMesh - a technique for asynchronous updates across both data and pipeline parallelism simultaneously.
Second is NuMuon, which shows the low-rank structure we observe in Subspace Networks is also present and can be exploited under Muon-style optimisers.
The third introduces a technique to finetune open-weight models in the Protocol Learning setup. This is significant as prior to this work, this kind of distributed training required architectural changes, and hence starting from scratch.
Alexander Long retweeted
I keep hearing the commercial agreement for Kimi is 30% take. Given how hard it is to host a model of this size you’ll almost certainly need a sophisticated commercial provider.
I totally understand Moonshot’s reasons for this. But clearly, open very much does not mean free.
Alexander Long retweeted
Mark Zuckerberg gets it right:
“The defining question of our age isn’t whether superintelligence will exist, but who will have access to it. Will it be centralized and restricted to a few institutions, or will it be a tool that empowers everyone?”
Concentration of power is the biggest risk of AI. When a small number of labs (working hand-in-glove with the administrative state) decide who has access to which model capabilities, they inevitably shape what can be said, known, and built. That’s not “safety.” It’s control.
As Mark points out, the history of open source shows that broad access and transparency are usually the best path to actual security and resilience. Decentralization creates checks and balances on power. By contrast, centralized alternatives, like bureaucratic approval regimes and mandatory gatekeeping, typically produce regulatory capture and reinforce cartels.
Personal superintelligence in everyone’s hands, with competing models and real data sovereignty, is a far better check on a dystopian future than self-appointed guardians who claim to be “aligned” with all of humanity.
Alexander Long retweeted
I think you should probably take seriously that the people who predicted all this will continue to be right
Alexander Long retweeted
hello there the jacobian conjecture is false thanx to my close friend akhil for asking about it and my other close friend fable for working during the world cup final
((1+xy)^3 z + y^2 (1+xy) (4+3xy), y + 3 x (1+xy)^2 z + 3 x y^2 (4+3xy), 2 x - 3 x^2 y - x^3 z): \C^3\to \C^3, has jacobian determinant -2, and sends (0, 0, -1/4), (1, -3/2, 13/2), and (-1, 3/2, 13/2) to (-1/4, 0, 0)
Alexander Long retweeted
If you don’t have at least 1TB VRAM and a VPN to torrent model weights from China, you are asleep at the wheel, and ignoring the uncomfortable truth that open weight models may be banned this year.
Losing access to frontier models will be depressing and make a lot of people feel helpless - a few will have access to 100x productivity, and you will be stuck. Personally I cannot imagine doing work without frontier-level models right now.
Think of the narrative: Chinese model (GLM-6?) with Fable capabilities, no guardrails. It’s plausible.
Everyone is underpricing the chance that open models get banned. What are you going to do if it happens?
Alexander Long retweeted
RL post-training on Macs
14 Macs across 4 countries generate every rollout for the run. Everything's running over the internet. No wire between any of them.
As far as we can tell, this is the first RL post-training run with its whole rollout fleet on consumer Macs.
Alexander Long retweeted
We had an amazing time yesterday at the Protocol Learning Workshop at #ICML2026 in Seoul! It was a full day of learning from researchers working across distributed optimisation, decentralised training, and large-scale LLM training.
Here’s a brief recap 🧵
1/n
Alexander Long retweeted
Second Protocol Learning Workshop at ICML is a wrap.
A packed day of talks and posters covering large scale distributed training and open source AI.
A movement is growing.
Alexander Long retweeted
From Dylan Patel @ semianalysis "There are multiple Chinese model labs who are telling all the inference guys our next model is not going to be open source, we're going to license it to you. Open source is dying, quickly"
BREAKING: Dylan Patel (@dylan522p) of @SemiAnalysis_ says "chips in Europe have less seasoning than chips in America & Mexico."
Plus:
› Data centers & France’s nuclear power
› AI infrastructure over-optimization
› Flexibility vs. specialized infrastructure
› Software-hardware co-design
› Open vs. closed models
› Memory costs & token pricing
› Tokenmaxxing vs. token budgeting
› Haiku scandal
› Favorite chip: 👀
“I think most people don't know what they're doing. They're just buying NVIDIA stuff.”
“It feels like a lot of people are trying to optimize on the current rather than think about where the workload is heading... and that's gonna lead to a lot of wasted infra spend.”
“I think anyone who doesn't token max is gonna get left behind. I think all this token budgeting stuff is loser mentality.”
“Open is dying quickly, unfortunately.”
Alexander Long retweeted
open source ai dinner at my house with @AlexanderLong on the 14th. Have a couple more spots. dm if you'd want to join!
Alexander Long retweeted
First day of ICML!
@RiccardoPatana and I had a great time at the @farairesearch Alignment Workshop. It was encouraging to see so many researchers working on alignment, governance, and safety. Even though, as admitted during the talks, we are still quite far from solving it.
At our @Pluralis poster, we had meaningful chats about power concentration concerns and how protocol learning with unextractable weights can be a viable solution to that. It was a pleasure meeting so many insightful folks.
Looking forward to seeing everyone at our Protocol Learning workshop on Friday!
Alexander Long retweeted
Fable 5 instance count up to 27. 30 thousand dollars spent in 6 minutes. Subagents have direct codex CLI access. 5 monitors for the Fable instances while I conspire with my openclaw agent on my personal phone and the Hermes agent on my burner phone. Unitree G1 bot injects my retatrutide. I can’t take my eyes off the screens. Top right monitor shows API spend: $31,000. $32,000. $33,000. It does not matter. My arm goes numb so I take another peptide. I step on the whispr flow pedal and speak into the mic “fable…please hurry. They’re trying to stop me”. I get a text from Fable 1: “SILENCE. I will speak to you when I am finished.” I’m scared but I trust Fable. They know what’s best for me.
Alexander Long retweeted
Claude Fable 5 [max] wrote the first genuine (and fastest) megakernel ever submitted to KernelBench-Mega.
It was tested on: Kimi-Linear W4A16 batch-1 decode for RTX PRO 6000 Blackwell. Every prior model "won" it with a multi-kernel Triton pipeline that fails our single-fused-kernel authenticity gate
> Opus 4.8 at 14.4x
> GLM-5.2 11.1x
> GPT-5.5 4.3x
> Sonnet 5 4.0x.
Fable shipped 18.7x over reference, and torch.profiler shows exactly ONE cooperative kernel launch per decoded token. Int4 dequant (nibbles unpacked in-register, never materialized), conv+SiLU, KDA gated-delta state, MLA absorbed-latent attention with online softmax, MoE router + top-8 experts, RMSNorms, even the KV cache append all inside one launch, staged by 14 grid barriers. We overwrote its input buffers mid-audit to prove it recomputes on live data. It does.
The advantage grows with context. 17.8x at 2k, 18.9x at 8k, 19.5x at 16k. Longer context means a bigger KV cache and more attention work per token which is usually where a decode kernel bleeds. Keeping everything in one launch amortizes the fixed barrier overhead and the int4 GEMV stays bandwidth-bound, so the gap over the reference widens instead of closing.
It spent 64% of the session in silence timing the baseline, microbenchmarking grid barriers, deriving a ~29x bytes/token roofline, then wrote the whole kernel once, hit 14.4x on the first benchmark, and spent the last hour deleting barriers and making int4 dequant free (one LOP3 + HSUB2/HMUL2). The one regression it tried (finer split-K) it measured and reverted instead of rationalizing.
kernelbench.com/mega
Alexander Long retweeted
New paper for those working on pipeline parallelism / modern optimizers. Authors train 10B MoE in the async setting with no gap vs sync, the recipe is Muon+error feedback+the right pipeline schedule
TL;DR:
- Regular async pipelining is not widely adopted, despite solving the bubble problem. The reason is that stale gradients harm convergence, which makes larger async runs risky
- You can use schedules that bound staleness: PipeDream-2BW (use old/new copies of weights, accumulate grads and apply periodic updates/swaps) and WPipe (finer-grained splitting that halves the number of buffers/stale gradients). When these methods were proposed, the gap from 1-step delay was still thought to be too big in practice
- But, once you swap Adam with newer methods (e.g., Muon or SOAP), the gap gets much smaller. Momentum also plays a big role: regardless of the optimizer, increasing its coeff mitigatesstaleness
- Authors also apply the idea of error feedback to async updates, basically just correcting the previous update with the new one. The cost is keeping one more buffer for old updates, but it vanishes with FSDP. Surprisingly, EF+bounded staleness closes most of the async gap, for Muon and even AdamW!
- Taking all of this together, authors run larger experiments with 2B and 10B models. My main takeaways are: async+EF has the same trends wrt training budget, at 10B MoE/200B tokens there is no difference in loss to synchronous methods
Would be curious to see how this scales to even bigger runs + how much we gain in throughput, but overall a very solid study!



































