quo retweeted
New paper from my group.
We gave coding agents a formal executable semantics of the language plus check.intentcomputing.org, a kit of proof procedures, then asked them to prove the code they generate correct.
➡️ HumanEval: 164/164 proved, and all passed audit.
➡️ arxiv.org/abs/2609.39022
quo retweeted
Replying to @d_g_clark
Wonderful work! 🤩 I printed the paper for a proper weekend read, but couldn't resist making a 3Blue1Brown-style explainer to start digesting it. Plots use the data from your repo. Sharing in case others find it useful, corrections welcome! (Watch with sound on 🔊)
quo retweeted
Working with AI models, I've arrived at what I consider the most significant pure-theory breakthrough of my research (in theoretical neuroscience) so far:
arxiv.org/abs/2610.12426
Chaotic systems show what is colloquially called the butterfly effect, namely, a tiny nudge to the initial state grows exponentially in time. How fast the nudge grows depends on its direction. Each direction has its own growth rate, called a Lyapunov exponent, and the full set of rates is the Lyapunov spectrum.
The Lyapunov spectrum is a rich (and diffeomorphism-invariant) characterization of chaos, telling us how many directions are chaotic, how many dimensions the attractor occupies, and how fast the dynamics generate information.
In 1988, Sompolinsky, Crisanti, and Sommers (Physical Review Letters) introduced a nonlinear recurrent-network model of N randomly coupled neurons that becomes chaotic when the coupling strength exceeds a critical value. This network has since become a cornerstone of theoretical neuroscience, both as a model of the spontaneous activity of cortex and as the starting point for training recurrent networks on tasks. I have worked on this model for much of my research.
In the same paper, the authors obtained the largest Lyapunov exponent from the ground-state energy of a Schrödinger equation and named the full Lyapunov spectrum as an open problem. In 2023, beautiful work by Engelken et al. (Physical Review Research) computed the full Lyapunov spectrum numerically. However, an analytical calculation of the full Lyapunov spectrum has remained out of reach.
Four decades after the problem was posed, this paper derives the full Lyapunov spectrum at large N from a simple self-consistent single-site problem.
This calculation has three steps.
First, we lower every exponent by a constant s, which turns the question "how many exponents lie below s?" into the question "how many tangent directions decay?" We show that, if we drive the shifted tangent dynamics with a source and take the minimum-norm solution, the one-step response is the projector onto the decaying directions. Thus, the trace of this projector counts the decaying directions. This step is exact at finite N.
Second, to work with the minimum-norm solution, we cast it as the vanishing-regularization limit of the solution of a least-squares problem. This regularized solution in turn solves a dynamical system with one field running forward in time and another running backward.
Third, we apply the cavity method, that is, we add one neuron to the network, along with its forward and backward fields. The effect of the rest of the network on this neuron then reduces to the classical dynamical mean-field description of the activity of this neuron, plus two kernels that couple the forward and backward fields of this neuron to each other across time. Thus, the full Lyapunov spectrum follows from a single-site problem, which boils down to the self-consistent calculation of these kernels.
The resulting theory agrees closely with simulations for the Lyapunov spectrum, the attractor dimension, and the entropy rate, and recovers known results, including the Schrödinger equation of Sompolinsky, Crisanti, and Sommers for the largest Lyapunov exponent.
As a byproduct, the theory establishes analytically that the chaos is extensive, i.e., the attractor dimension and the entropy rate grow in proportion to N.
I had assumed that this problem was analytically intractable. After working independently for 100 minutes with Max thinking, GPT-6 Astra produced the initial derivation, yielding a self-consistent problem whose solution matched simulations of large networks.
I found this derivation intriguing, albeit complicated and somewhat mysterious (see AI Methodology in the paper). After investigating the structure of the proposed solution more deeply, together with AI models, I arrived at a physically transparent derivation.
I am excited to apply this approach to other network models, trained networks, and, more broadly, to high-dimensional systems to which Lyapunov analysis is relevant, including AI models themselves. Of course, as with any paper, I took care to write up the paper clearly.
A few broader thoughts...
Neural circuits are high-dimensional nonlinear dynamical systems, and such systems are notoriously hard to understand. I would frame the central problem of theoretical neuroscience as understanding how such systems function, process information, learn, and generate behavior.
Many people in the theoretical sciences are anxious about AI. However, in my field at least, we are nowhere close to fully understanding the high-dimensional computations unfolding in our brains. With AI, we could reach a level of understanding that so far we have only gestured toward. So, I am excited.
quo retweeted
Backprop has been the only credit assignment algorithm capable of training large neural nets.
Introducing Dust, the first zeroth-order method to pretrain transformers to approach and sometimes even *exceed* backprop with large amounts of computation (bitter lesson!)
- Dust is on the order of 1,000 to 10,000x more compute efficient than EGGROLL, the state-of-the-art ES method, for training transformers.
- We chose pretraining transformers because it's arguably the hardest task possible for zeroth-order optimization.
- Search in high dimensions is really misunderstood: zeroth-order usually gets better with overparameterization, not worse, i.e. at a fixed population size, larger models (up to 120x) reach a lower loss than smaller ones.
- Backprop-like gradients emerge from a large population of activation perturbations, and the alignment holds up at every scale we tested, up to 1B tokens, which is critical for scaling.
The core idea is a *virtual population*, which helps scale to large populations in transformers by perturbing activations in parallel.
w/ @bishmdl76, @cs_serdar, @akshayvegesna
quo retweeted
RNNは前の状態を使って次を計算するため、時間方向には順番に処理するしかない――これが長年の弱点でした。ところがこの研究は、非線形RNNを時間方向にも並列化し、3万ステップを超える長い時系列で、従来の逐次学習に対して最大870倍の高速化を報告しています。
鍵になるのがDEERという手法です。RNNを「時刻1→2→3と順番に計算するもの」として扱うのではなく、全時刻の状態を同時に満たす方程式を解く問題へ置き換えます。さらに、カオス系で計算が不安定になる問題をGeneralized Teacher Forcing(GTF)で抑え、非線形なRNNでも長い系列を並列学習できるようにしました。
ここで重要なのは、単にRNNが速くなったことではありません。TransformerやMambaなどが広がった背景には、RNNの逐次処理がGPUと相性が悪いという事情がありました。その制約がアルゴリズム側から外れるなら、「再帰モデルは表現力があるが遅い」というこれまでのトレードオフ自体が変わる可能性があります。
高速化によって、この研究はもう一つ重要な問いを検証できるようになりました。同じ量のデータを使うなら、短い時系列を大量に学習するのと、長い時系列を連続して学習するのは同じなのか、という問いです。
結果は同じではありませんでした。Lorenz-96や神経活動モデルでは、総データ量を揃えても、長い連続系列で学習したRNNほど長期的な力学構造を正確に再現しました。一方で短期予測性能は、比較的短い系列でも早い段階で頭打ちになります。
arxiv.org/abs/2605.12683
We found a way to make SFT match the performance of RL in post-training!
Excited to share finetuning through sampling, which allows us to combine sampling with SFT to allow an LLM to learn new knowledge without forgetting past information.
aakaran.github.io/finetuning…
quo retweeted
Can an LM, starting from random init (!!), learn to generate all of its pretraining data?
Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities.
A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya @noahdgoodman, and @YoavLevine.
quo retweeted
1/9
We rarely post here. This felt worth sharing.
What if everything we can explain about an AI model is only the familiar shore of a much larger ocean?
Our paper, “Xeno-Interpretability”, asks what lies beyond our conceptual reach.
arxiv.org/abs/2609.20408
quo retweeted
📚 Open Access (free)
Rolls, E. T. (2023). Brain Computations and Connectivity. New York: Oxford University Press. doi.org/10.1093/oso/97801988…
quo retweeted
Our group discovered that reasoning models produce fractals when asked to solve hard problems. We can use nonlinear dynamics to probe the thinking processes of recurrent depth models on Sudoku, mathematics, and even ARC-AGI (1/N)
arxiv.org/abs/2609.04963
Real brains follow Dale's principle: a neuron can either excite its neighbors or suppress them, but never both. Standard deep learning ignores this and uses backpropagation.
In our new paper, Diffusing Blame, we fix this disconnect. By introducing a routing method that broadcasts error signals directly to the hidden layers, we can train networks made of dedicated positive and negative neurons to strictly obey Dale's principle, all without relying on backprop!
This method works surprisingly well on image recognition tasks despite the strict biological constraints. We also achieved competitive, backprop-free reinforcement learning on complex locomotion tasks and the open-ended Craftax environment.
It is neat to see that representation learning remains possible even when we force deep learning to play by the rules of real neurons.
Introducing "Diffusing Blame": can a neural network learn competitively while strictly obeying Dale's principle, the rule that real neurons follow? We show it can, across both image classification and reinforcement learning. 🧠
Accepted at #ALIFE2026
arxiv.org/abs/2606.31700
Real neurons generally follow Dale’s principle: each neuron is predominantly excitatory or inhibitory. Standard artificial networks usually ignore this constraint, allowing every unit to mix positive and negative outgoing weights.
Backprop makes the gap even wider. Its backward pass needs exact transposed copies of the forward weights, the so-called "weight transport problem,” which biology doesn’t seem to have a mechanism for.
So we asked: can a network that strictly enforces Dale's principle still learn well, without weight transport?
Our approach builds on Error Diffusion (ED), a local rule that routes a single global error signal directly to every hidden unit, where each layer is split into separate excitatory and inhibitory streams with four non-negative weight matrices, so a synapse's sign comes from fixed population identity rather than a learnable weight. Our main contribution is to extend ED from binary to multi-class problems via modulo error routing.
We then asked whether this routing mechanism could provide useful credit signals in the noisy setting of reinforcement learning. During PPO training on Ant, Humanoid, and HalfCheetah, we compared each local ED update with the corresponding true backpropagation gradient. Among the routing schemes we tested, modulo routing consistently produced the strongest alignment.
Taken together, these results show that Dale-constrained networks can still learn without transporting weights backward, suggesting a potential path toward learning rules that are both effective and more biologically plausible.
quo retweeted
So basically,
This is the Thinking Machines’ attention mechanism: GRAPE-A-Query-Gated
github.com/model-architectur…
No more RoPE, no more attention sink, no more gated attention, WE ARE SO BACK!
arxiv.org/abs/2512.07805
thinkingmachines.ai/news/int…
RoPE is dead, Long live GRAPE! 🚀🚀🚀
Today, we give robots a /skills library that self-evolves and compounds indefinitely! Introducing ASPIRE: a robot solving its 100th task is no longer as clueless as solving its first. Coding agents observe multimodal sensory traces from simulation and real robots, launch an evolutionary search over control programs, and distill the best know-how into an ever-expanding library.
ASPIRE is a new type of continual learning: "training" is skill refinement instead of gradient descent.
"Trained model" is a repo of sensorimotor skills instead of floating weights.
“Distributed training” is a panel of agents each practicing a different skill instead of sharded minibatches.
Here's the beauty: ASPIRE gives the tired terms "sim2real transfer" and "cross-embodiment transfer" a whole new meaning. Bridging the sim-to-real gap is notoriously brutal. An end-to-end policy has to swallow both the visual shift (sim looks toyish next to a real camera) and the subtle contact physics it never quite gets right. ASPIRE sidesteps the mess, because it doesn't ship pixels or weights across the gap, but ships the know-how. The robot still has to practice in the real world, not zero-shot, but it gets there way faster because it isn't rediscovering the strategy from scratch. Same for going single-arm to bimanual hardware, which usually requires new data and retraining from zero. ASPIRE achieves up to ~10x cut in "transfer learning” tokens (yes, tokens are the new unit of *training* compute ;)
Check out our gallery of 150+ tasks and 90+ skills the robots taught themselves, all on the website! Kind of wild that we can ship the "learned weights" as an HTML page rather than a GGUF. We'll open-source the full stack so your own robot library starts compounding from ours!
Deep dive in thread:
quo retweeted
[1/9] What happens when you treat vision as a first-class citizen during multimodal pretraining? To find out, we studied the design space of training Transfusion-style models that input and output all modalities, from scratch. Here is what we learned about visual representations, data, world modeling, architecture, and scaling behavior!
Paper: arxiv.org/abs/2603.03276
Website: beyond-llms.github.io/
@TongPetersb, @DavidJFan, @__JohnNguyen__, @ellisbrown, @GaoyueZhou, @JasonQSY, @boyangzheng, @webalorn, @han_junlin, @rob_fergus, @NailaMurray, @gh_marjan, @ml_perception, Nicolas Ballas, @_amirbar, Michael Rabbat, Jakob Verbeek, @LukeZettlemoyer, @koustuvsinha, @ylecun, @sainingxie





















