
⬢ chaos dancing star @paradigmainc
Icecrown Citadel
Joined February 2022
- Tweets8.7K
- Following443
- Followers5.9K
- Likes42.7K
tensorqt in SF retweeted
Next couple days will be exciting for flywheel, we are cooking
tensorqt in SF retweeted
We're releasing a number of updates to Flywheel in the coming days.
First up, new ways to work on Flywheel graphs together:
• Share a whole graph in one save
• Ask for access from any link
• Link straight to a graph or a node
we are seeing today the last days of Europe’s age of exploitation. we must enter a new age of exploration
tbc, this was recorded before any millennium problem was solved using AI ;)
Can AI make the next great scientific discovery?
Francesco Pappone (@tensorqt), co-founder of @paradigmainc , joins Europe on Edge to discuss AI scientists, the SPRIND Frontier AI Challenge, and Europe’s AI ambitions.
🎥 Trailer below
🎧 Full episode in the first reply
tensorqt in SF retweeted
Can AI make the next great scientific discovery?
Francesco Pappone (@tensorqt), co-founder of @paradigmainc , joins Europe on Edge to discuss AI scientists, the SPRIND Frontier AI Challenge, and Europe’s AI ambitions.
🎥 Trailer below
🎧 Full episode in the first reply
tensorqt in SF retweeted
update: training without a backward pass is now faster than backprop.
what I did: a data-free start (block 1’s heads begin as lag-1 RoPE heads), a tail average of the weights, skipping 4 blocks’ attention credit late in the run, then overlapped collectives and a compiled forward.
we did this (mainly for lr) while training violetto (both during pre-training and post-training)
Excited to introduce reactive optimizers: We let an agent reactively adapt the optimizer code during the training run! The agent acts like a team of scientists constantly choosing the best algorithm, learning rate, momentum, etc, based on its own analysis of step-to-step training dynamics! The results are really exciting. Besides out-performing standard "best-practice" recipes across everything we threw at it, the agent gains a few "superpowers" that no human could have designed into the optimizer... (1/n)
tensorqt in SF retweeted
This next paper is a bit quirky, but here goes:
Introspection is hard to study in both humans and LLMs. But, we do have internal senses that are verifiable, like proprioception (the sense of where your arm is right now).
What would proprioception look like in language models?
As a digital analogue to the sense of position, a model could verbalize which expert activated in a certain layer for any previous token.
It turns out that LLMs can learn a range of such proprioception skills where they understand and report on their internals. We can calibrate and score these reports, and evaluate whether training for proprioception changes model behaviors.
tensorqt in SF retweeted
we got backprop's loss at ~4× its time, and 3× faster than the record.
this was made possible by using @paradigmainc's Flywheel infra from the get go, preventing rabbit holes and encouraging branching and backtracking.
fun thing about Emanuele is that he will casually mention he did this in the middle of some unrelated conversation
tensorqt in SF retweeted
forward-only pretraining now matches backprop on the Noise benchmark.
Relay: val 3.704 in 2.5 min on 8×H100.
backprop with the same model, data and optimizer: 3.712. the current record, Boost→Cut: 3.945 in 7.6 min.
no autograd anywhere. 🧵
Inspired by Nick Cammarata’s interpretability essay “Neural Networks Are Already Clean”, I turned a couple of its ideas into speedups for pretraining without backpropagation. Still in the works. 2.3x speedup, cool xitter.cf/industriaalist/status/…
this has always been the case, the universe doesn’t owe you simplicity. everything looks simple because that’s all we could discover

















