There is very interesting overlap actually between the Dust paper and the YOCO paper. Dust is a biased estimator that prevents the model from being able to learn to generate kvcache beneficial to predicting upcoming tokens. But YOCO architecturally pre-removes this possibility.
Why does this take two years to build what am I missing, I feel like I could do this in a couple weeks.
We're announcing the public beta of datavendor.ai
We spent the last two years building the best solution for people buying and selling data, whether you're a researcher in a lab or a company trying to sell data.
Data is the fundamental bottleneck for AI progress, and DataVendor will be the infra that solves it.
There are many ZO papers with results like this. Many different optimizers which choose directions to perturb in, using one manner of trickery or another.
It's pretty obvious that these need to be adapted to Dust, and re-tested. I do not expect the results to move the needle. But it's probably worth setting a bunch of research agents on.
The holy grail would be solving it for large networks, in a pretraining context. I think we still do not understand what happens to these loss landscapes to make SPSA-like algorithms fail, and better understanding of what's going on would be huge.
Conventional zeroth-order optimizers such as MeZO estimate gradients through random parameter perturbations, wasting evaluations on directions unrelated to meaningful updates.
AGZO from ICML26 exploits a simple but powerful geometric property: the weight gradient of a linear layer lies in the subspace spanned by its input activations. Rather than sampling perturbations across the entire parameter space, it extracts low-rank activation subspaces during forward computation and restricts gradient estimation to those directions.
Under spectral concentration assumptions, the authors prove better expected alignment with the true gradient than isotropic perturbations. On Qwen3-0.6B, AGZO achieves 89.2% accuracy on CB versus 80.3% for MeZO, while enabling Qwen3-4B fine-tuning in a hardware setting where full first-order training runs out of memory.
The deeper insight is that avoiding backpropagation does not require treating neural networks as structureless black boxes. Forward activations already reveal useful information about optimization geometry, connecting zeroth-order learning with randomized linear algebra and low-dimensional representation structure. However, the method remains substantially more iteration-intensive than first-order training, and its advantages have not been established at frontier pretraining scale.
The next question is whether adaptive activation subspaces can reduce both gradient-estimation noise and the number of forward evaluations required for effective learning. Perhaps the next generation of optimizers will use forward computation not merely to evaluate loss, but to determine where parameter updates should occur.
arxiv.org/abs/2601.17261
This concurs with my own findings on training looped transformers, by the way. Although their work is much more polished than mine.
Looped transformers often save compute by truncating backpropagation to the last few loops.
Is it worth it? We trained ~600 models to find out 👀
Short answer: only if the loop count is fixed. Otherwise, run fewer loops and train them all!
📄 arxiv.org/abs/2610.26030
🧵👇
The Dust paper is really big. A breakthrough.
After spending a few days to really digest it though, I'm somewhat disheartened that so much of it is implementation details which are not explained. I'm working on a blog post to reverse engineer the math, methods, and choices.
Backprop has been the only credit assignment algorithm capable of training large neural nets.
Introducing Dust, the first zeroth-order method to pretrain transformers to approach and sometimes even *exceed* backprop with large amounts of computation (bitter lesson!)
- Dust is on the order of 1,000 to 10,000x more compute efficient than EGGROLL, the state-of-the-art ES method, for training transformers.
- We chose pretraining transformers because it's arguably the hardest task possible for zeroth-order optimization.
- Search in high dimensions is really misunderstood: zeroth-order usually gets better with overparameterization, not worse, i.e. at a fixed population size, larger models (up to 120x) reach a lower loss than smaller ones.
- Backprop-like gradients emerge from a large population of activation perturbations, and the alignment holds up at every scale we tested, up to 1B tokens, which is critical for scaling.
The core idea is a *virtual population*, which helps scale to large populations in transformers by perturbing activations in parallel.
w/ @bishmdl76, @cs_serdar, @akshayvegesna
I'm going to be presenting my research at the Latent Space paper club tomorrow, thanks @yikesawjeez.
Link: luma.com/k8u3voz2
As for my origin story, circa 2019 I did my own research on perceptual image hashing methods as an honors college project and ended up discovering deep learning, because it happened to beat every known heuristic. I wrote some tensorflow and pretrained tiny image embedding models on my school laptop. This annoyed my mom, because I left my laptop on the porch overnight so it wouldn't overheat, with a cable running out the back door. I was too poor to afford a GPU at the time, but ended up training some very good models using what little compute I had in the house.
Then I graduated, got a job writing airplane firmware to be close to a girl, then did compilers for a while. I was always doing deep learning projects in the background, but I eventually circled back to my true passion.
I would have still been interested in deep learning without the girl involved, but I'd like to think she helped by dragging me around, inducting me into her friend group, getting me over my stage fright, helping me figure out gender stuff, and forcibly socializing me. I owe her a lot. I definitely would not be the person I am today without her.
I think there are perks to not working at a frontier lab.
You get to think "Oh, well, at those labs they've really got their shit together." They probably do X, and Y, and Z, which are the things which would obviously be done by any Serious Individual working on this problem."
Then you meet people from frontier labs and they tell you "No we aren't doing anything like that at all, that sounds like a lot of work."
Very frustrating, but there is alpha in there somewhere.
I'm finally doing a whole ton of ZO runs, trying to figure out what hparams matter. I've been having success with having agents write mlsweep files, submit them, interpret results, and use the results to inform new experiments. Can save a lot of effort on open ended problems.
>OpenAI awarded us $6,500.
If it was me I would be offended.
I've been thinking about this paper for a couple days.
Neural networks have to do a whole bunch of error correction. They have to. By construction. This is because you're training by feeding in data which is mostly noise from an information theoretic perspective. Computation cannot create signal, only destroy it. The trick is to destroy only the noise, and preserve the signal relevant to the task. Any network which does not maximize this objective would have higher loss. So training must produce models with this quality.
This is not to say, though, that other views on optimization are not correct. I still subscribe to Ilya's idea that pretraining is compression. Compression, too, can be seen as throwing out only the noise and retaining the relevant signal to the task, which can be solved by approximate memorization. But these small changes in perspective are important and useful, I think.
So, from this perspective I got to thinking. Gradient updates must preserve this behavior. What does this imply about the loss geometry? Is there some sort of invariant that can be asserted to build a better optimizer?
I think that there is. I'm still thinking about it. But I feel like this is useful for something.
The smoothed objective from Evolution Strategies and the Taylor-expanded objective from SPSA are literally the same objective, just interpreted through different lenses. Both lenses just happen to produce exactly the same equation.
