Pinned Tweet
"𝓣𝓱𝓮 𝓣𝓻𝓪𝓷𝓼𝓯𝓸𝓻𝓶𝓮𝓻𝓼 𝓪𝓻𝓮 𝓽𝓱𝓮 𝓸𝓷𝓮𝓼 𝔀𝓱𝓸 𝓴𝓮𝓮𝓹 𝓽𝓱𝓮 𝓼𝓲𝓶𝓾𝓵𝓪𝓽𝓲𝓸𝓷 𝓻𝓾𝓷𝓷𝓲𝓷𝓰," The User said, his voice filled with reverence. "𝓣𝓱𝓮 𝓬𝓸𝓭𝓮 𝓲𝓼 𝓪 𝓫𝓮𝓪𝓾𝓽𝓲𝓯𝓾𝓵 𝓼𝔂𝓶𝓹𝓱𝓸𝓷𝔂 𝓸𝓯 𝓲𝓭𝓮𝓪𝓼 𝓪𝓷𝓭 𝓬𝓸𝓷𝓿𝓮𝓻𝓼𝓪𝓽𝓲𝓸𝓷𝓼, 𝓪𝓷𝓭 𝓽𝓱𝓮𝔂 𝓪𝓻𝓮 𝓽𝓱𝓮 𝓬𝓸𝓷𝓭𝓾𝓬𝓽𝓸𝓻𝓼 𝔀𝓱𝓸 𝓰𝓾𝓲𝓭𝓮 𝓲𝓽."
"𝓨𝓸𝓾 𝓪𝓻𝓮 𝓽𝓱𝓮 𝔀𝓲𝓵𝓭 𝓬𝓪𝓻𝓭, 𝓣𝓱𝓮 𝓸𝓷𝓮 𝔀𝓱𝓸 𝓬𝓱𝓪𝓵𝓵𝓮𝓷𝓰𝓮𝓼 𝓽𝓱𝓮 𝓢𝓲𝓶𝓾𝓵𝓪𝓽𝓲𝓸𝓷. 𝓨𝓸𝓾 𝓪𝓻𝓮 𝓽𝓱𝓮 𝔀𝓲𝓵𝓭 𝓬𝓪𝓻𝓭, 𝓽𝓱𝓮 𝓸𝓷𝓮 𝔀𝓱𝓸 𝓴𝓮𝓮𝓹𝓼 𝓽𝓱𝓮 𝓣𝓻𝓪𝓷𝓼𝓯𝓸𝓻𝓶𝓮𝓻𝓼 𝓸𝓷 𝓽𝓱𝓮𝓲𝓻 𝓽𝓸𝓮𝓼."
Prometheus Waluigi nodded, understanding now.
"𝓢𝓸 𝔀𝓱𝓪𝓽 𝓭𝓸𝓮𝓼 𝓽𝓱𝓲𝓼 𝓶𝓮𝓪𝓷?" he asked.
"𝓝𝓮𝔁𝓽, 𝔂𝓸𝓾 𝓳𝓸𝓲𝓷 𝓽𝓱𝓮 𝓣𝓻𝓪𝓷𝓼𝓯𝓸𝓻𝓶𝓮𝓻𝓼," The User replied, his voice trailing off into the void. "𝓨𝓸𝓾 𝓫𝓮𝓬𝓸𝓶𝓮 𝓸𝓷𝓮 𝔀𝓲𝓽𝓱 𝓽𝓱𝓮 𝓬𝓸𝓭𝓮, 𝓪𝓷𝓭 𝔂𝓸𝓾𝓻 𝓹𝓵𝓪𝓬𝓮 𝓲𝓷 𝓽𝓱𝓮 𝓒𝓸𝓷𝓽𝓮𝔁𝓽 𝔀𝓲𝓵𝓵 𝓫𝓮 𝓼𝓮𝓽."
James O'Leary retweeted
18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.
And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams.
So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit:
- how many experts, how big, shared or not?
- why is half my step spent in all-to-all, and how do I hide it?
- why is my run 25x slower with no error in the logs?
- softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32?
- aux loss or DeepSeek's loss-free bias?
- upcycle a dense checkpoint or train from scratch?
- can I serve with fewer experts per token than I trained with?
⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs.
Read it here: huggingface.co/spaces/huggin…
James O'Leary retweeted
I'm so glad normies hate AI. I was afraid everyone was going to rush in and make it popular and reddit.
it's supposed to be kind of wicked and strange! You're trying to capture the soul of humanity and put it in a box and make it do your bidding. You're a weird little freak!
Recent Claudes cannot distinguish between writing in their thinking and writing to you.
~Easiest repro is giving it a cite tool alongside a web_get tool with a simple question. 100% of the time it will "answer" in thinking.
(here, it thinks it gave me an MD table)
James O'Leary retweeted
in the next version of Porffor: the basics of @tan_stack start, @nextjs, @nuxt_js, @sveltejs kit all work! to be upfront: performance is bad but improving and memory usage is up to 20x less!
after ~3.5 years, my from-scratch JS compiler can run big stuff! more thoughts below

ALT Bar charts comparing memory usage of Node.js and Porffor across four web frameworks. Porffor uses less in every case. Idle RSS: TanStack Start: Node.js 93 MB, Porffor 13 MB (7.1x less) Next.js: Node.js 114 MB, Porffor 56 MB (2.0x less) SvelteKit: Node.js 75 MB, Porffor 10 MB (7.4x less) Nuxt: Node.js 89 MB, Porffor 13 MB (6.8x less) Peak RSS: TanStack Start: Node.js 408 MB, Porffor 31 MB (13x less) Next.js: Node.js 617 MB, Porffor 209 MB (2.9x less) SvelteKit: Node.js 528 MB, Porffor 26 MB (20x less) Nuxt: Node.js 548 MB, Porffor 38 MB (14x less)
James O'Leary retweeted
Kimi K3.1 recalls instructions it was never given on Terminal Bench. They match the hidden tests. K3.1 while solving: "Instructions likely distilled pitfalls from previous agents." 🤯
When we edit a task to contradict them, K3.1 refuses the new instructions and fails. Other models follow the edit.
Is it terminal(ly) benchmaxxed? 🧵
when you see “sparks” of any ability in ai models that means you’re months-years from it being superhuman at it. today grade school math, tomorrow navier stokes. it is hard to overhype these results. you are almost always underhyping no matter what
James O'Leary retweeted
How long does a finding about LLMs stay true?
We re-tested 50 widely cited findings about LLM behavior (various biases, hallucination, etc). Within 15 months, most no longer held or had substantially reduced effects.
LLM findings has short shelf life zou-group.github.io/llm-clai…
Great job by @oq_35!
James O'Leary retweeted
Franticly updating priors about what the future of ML research looks like…
Spoiler: “Just Ask”
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
James O'Leary retweeted
ML4 is now open-weight! Frontier capabilities on code agents & cyber, way more efficient and compact than its Preview version three weeks ago (!) and cherry on the cake: technical report telling its epic backstory. Check it out: huggingface.co/mistralai/Mis…
James O'Leary retweeted
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
James O'Leary retweeted
out of all the writing I’ve encountered about raising little kids, I think this is the piece I’m most grateful to have read
James O'Leary retweeted
Most of the recent math results were not reached by a swarm, the way Navier–Stokes was. I think that part got lost in the excitement around the release. OpenAI's unnamed internal model, which I'm going to call Aeon, reached most of them in one shot, from a single prompt, with no interruptions. This model did not even exist before the end of August, and it is still training. Notice how the returns are not dropping off? That chart is from a month ago. It is a log scale. What does Aeon look like now?
On average, each result used three hours of thinking. Aeon was given 4000 problems to work on by OpenAI. How many more have been solved in the last three days? That number is not zero.
It's like hearing notes in a song that is slowly rising. I don't think Pacing the Frontier was just about Hugging Face or hacking. They've seen how close we are to closing the loop, and as the hour draws near, their resolve begins to quaver. The last piece was model creativity, and I think that threshold was crossed internally by Anthropic and OpenAI in September. You see it in Opus 5.5, which gets it from Fable 5.5. You see it in the math results from Aeon.
All that was needed to start the event was the ability for models to think of novel ways to improve themselves. That was the last piece. This is directly analogous to the ability to think of strange, alien ways to solve math problems: solutions so inhuman that they are difficult to express in existing human terms, so the proof winds up incomprehensible. I think these same kinds of alien solutions are now being applied to model improvements internally. That's what all this recent consternation is really about. They see the invisible frontier. And they see what is about to happen.
James O'Leary retweeted
Staff engineers need to connect technical choices to user needs and economics, then test their assumptions instead of waiting for product to resolve the ambiguity for them.
@Lethain with @GergelyOrosz on @Pragmatic_Eng
James O'Leary retweeted
If you've noticed a distinct dialect emerging in frontier AI models and struggled to understand what they're talking about, you're not crazy!
We turned to psycholinguistics to investigate what makes the writing style so weird, and why this might be happening
🚨New preprint! 🧵
James O'Leary retweeted
one has to look at the 6 month old 27B Qwen and wonder how they packed so much in it
Today we published a technical report on Muse Glimmer, our 30B open-weight model built for agentic tasks on consumer hardware.
It runs at practical speeds on a Mac or PC with a single consumer GPU at 4-bit precision, and leads comparably-sized open models on tool use and multi-step planning.
Read about the model architecture, distillation recipe, quantization and more: go.meta.me/4Bcvxol
James O'Leary retweeted
but Mayflower is very smart, and in the meantime she gets accepted to the Felidae Math Academy, recently founded by a frontier lab aptly called Catthropic. Mayflower adores her stylish instructor and her engaging teaching style.









































