Pinned Tweet
The Shape of Inference
sethmorton.com/blog/the_shap…
Most AI progress today builds on something fundamentally limiting: autoregressive inference.
When you discover something important, you don’t simply add another step to a chain of thought.
Old observations take on new meaning and new paths appear. Your mental model restructures and settles somewhere new.
I call this omnidirectional inference, and I believe it is one of the most important missing capabilities in modern AI.
I wrote about why this matters, what architectures could enable it, and why they may require a different compute substrate.
Thank you to @AdamMarblestone for inspiring this essay.
I was missing the OG Arc days
and again, search 1.0.2 is out.
new, each off until you turn it on
→ a bookmarks bar
→ peek at a link with shift-click
→ the video follows you when you switch apps
→ pages at 120 hz
→ see where a link goes before you click
also new
→ passkeys with your mac's own sheet: touch id, iphone, security keys
→ mute a tab, share a page, copy as markdown link
→ mouse back/forward buttons, middle-click opens a new tab
→ spaces in the top bar
→ brave search, qwant, homebrew
40+ pull requests from you went into it, thank you
and so many fixes, all in the replies ↓
223
Effective continual learning is achieved. RSI is achieved.
Any sort of online RL is continual learning. Any model generating critiques used to post-train a successor is RSI.
These terms have become somewhat meaningless because you can stretch their scope to cover a wide range of phenomena.
I really enjoyed @carnot_cyclist 's maximal definition of continual learning a while back – we need something similar for RSI.
I think having good maximal operational definitions are important because it gives us something to stride towards & compare against.
In this case, Info-SDFT seems to be an interesting solution to the stability-plasticity subproblem, however it is not in any way maximal continual learning. Its preservation mechanism is a KL anchor to the frozen original model.
We can now do effective continual learning for #LLMs
I am happy to present Info-SDFT: Information-Proximal Self distillation. The method nicely balances plasticity versus stability and works very well.
We did over 500 experiments on this! Thanks to @KickItLikeShika!
We open source code and checkpoints for you to use:
Code: github.com/KickItLikeShika/i…
Checkpoints: huggingface.co/collections/K…
Paper: arxiv.org/pdf/2609.24646
5
374
I've been working more on causal discovery foundation models and came across this today.
I really like the framing of what should be compiled into reusable structure vs what should remain as test-time inference.
I'm excited to push that boundary even further - so perhaps the reasoning process can leave behind a small persisted change immediately rather than having to distill after the fact
How can we improve robot generalization beyond collecting more demonstrations?
A robot can use inference to work out how to solve tasks it was never explicitly trained to perform. With learned models of the world, we can plan our future actions and goals before acting.
I wrote a research perspective, Generalization by Construction, on how learning can be combined with inference for flexible generalization.
yilundu.github.io/perspectiv…
2
7
582
The current paradigm has become a star in the economic sky, bending capital, energy, and talent toward its gravity.
The idea that intelligence is not yet “solved”, and we should actively explore alternative paradigms has been increasingly clear to me since my time in biology.
I'm incredibly excited about this collective, would love to contribute!
Introducing the Sakana AI Frontier Intelligence Group 🪷
sakana.ai/frontier-intellige…
Current AI systems are incredibly capable, but is intelligence “solved”? And if not, what’s missing?
At Sakana AI’s Frontier Intelligence Group (FIG), we believe that there are still breakthroughs to be made in AI. The Transformer and language modeling may be incredibly powerful, but it doesn’t mean that better alternatives don’t exist.
Natural intelligence still beats artificial intelligence across many dimensions. Agents lack the deep insights and creativity of humans. Individual models require far more data than the brain to learn robustly, and require far more energy to run. If we set these as targets, what kinds of AI systems could we develop?
Research at FIG has sought to address the gaps between natural and artificial intelligence. Here are some of our works, and the fundamental research questions that motivated them:
• Continuous Thought Machines: How can we improve information processing by leveraging temporal dynamics?
• Augmented Lagrangian Predictive Coding: How can local learning solve multilayer credit assignment?
• Sparser, Faster, Lighter Transformer Models: How can we massively increase data efficiency and generalization?
• The AI Picbreeder Experiment: How can we make artificial open-ended systems?
• Smart Cellular Bricks: How can physical systems achieve collective intelligence and self-repair without a central brain?
We hope that this encourages other researchers to also explore different paradigms, and take a leap of faith with us. After all, in the words of a dear friend of ours, “greatness cannot be planned”.
10
579
To achieve full RSI, more people should be talking about Discovery Foundation Models.
Came across this paper by @LingYang_PU & @zhenfei_yin_yzf, super interesting work
arxiv.org/pdf/2609.15973
3
289
Omnidirectional inference: the past (x) and future (y), updating together
The Shape of Inference
sethmorton.com/blog/the_shap…
Most AI progress today builds on something fundamentally limiting: autoregressive inference.
When you discover something important, you don’t simply add another step to a chain of thought.
Old observations take on new meaning and new paths appear. Your mental model restructures and settles somewhere new.
I call this omnidirectional inference, and I believe it is one of the most important missing capabilities in modern AI.
I wrote about why this matters, what architectures could enable it, and why they may require a different compute substrate.
Thank you to @AdamMarblestone for inspiring this essay.
1
9
502
The Geometry of Surprise: sethmorton.com/blog/the_geom…
Reward models are trained on a snapshot of the world. For continual learning that's a dead end - the model has to learn from its own surprise.
The question is what shape that surprise takes. 1/2
I wrote about what people get wrong thinking about incorporating surprise into models, and why continual learning starts when you can read that geometry off the system itself using @extropic's Z1s.
Thanks to: @deepakpathak and @HarriLEdwards for the research that inspired this
1
16
1,072
Why what you attend to can't be static:
sethmorton.com/blog/what_you…
Transformers can't change what they attend to after training. Backprop is too global and too destructive for continual learning. The brain doesn't work this way. 1/2
I wrote about why I think thermodynamic computation is the path forward, what @Extropic_AI is doing with their Z1 chip, and what I'm building to get there.
thanks to:
@trevormccrt1 for the UWaterloo lecture that inspired this and @carnot_cyclist for the great article
50
4,133
self-supervised loss is not talked about enough - incredibly cool paper. the problem is raw compute cost & memory cost -- this alludes to @carnot_cyclist 's blog, one of tenants of continual learning is efficiency. once again, thermodynamic computation is needed
6
578
everyone's talking about mythos - gpt 5.4 pro extended thinking is incredibly powerful and imo underrated
1
238
A beautiful article - i think a lot of the proposed solutions make sense but all have their unique limitations. To truly unlock continual learning @extropic ‘s TSU’s are the way. We need EBMs.
2
10
803

























