
Trying to solve sample efficiency prev: Engineering physics @iitbombay '23
Joined June 2023
- Tweets837
- Following694
- Followers4.3K
- Likes2.9K
Looping is the biggest driver so far:
- 8.4M model: 47% → 66% on ARC-1 when looped
- Beats non-looped models of the same or greater logical depth, with up to 4x the params
- Tracks other sample-efficiency benchmarks: looping set records on the Slowrun too
(caveat - both looped and non-looped can prob be tuned more)
1
3
443
ARC Data Squeeze:
How high can you score on ARC-1/2 when trained from scratch on very limited data?
Goal: beat GPT-6 Astra's score without LLM pretraining
Current record: 70%, Looped mdlARC
Rules:
- Fixed dataset: 1307 puzzles (~2M tokens)
- Any architecture, any compute
- no LLMs, DSLs or hard-coded ARC info (everything must be learned)
Inspired by @kellerjordan0's NanoGPT Speedrun, Q Labs' Slowrun and @OpenAI's Parameter Golf
Repo: github.com/mvakde/arc-data-s…
Announcing ⚡⚡
1) New records on ARC-AGI public eval !!
2) A new sample-efficiency benchmark
70% on ARC-1 with just 19M params (15% on ARC-2)
49% with just 1M params!!
(Latter beats TRM's 45% with ~400x fewer params)
Trained from scratch on only 1,307 ARC puzzles.
Just ~$13 to train
No text, no LLMs
🧵
Looping is a huge win:
At equal depth, it beat my non-looped stacks with up to 4× the params, by 2.3–7.6% (even at equal depth)
Even sub 1M models perform very well!
Room for growth -- I ran out of time, so final model is severely undertrained. Performance can go much higher
2
9
1,244
What's new and different:
Looping the middle layers
All demo pairs in context
Hub-and-spoke sparse attention (to fit in 4xL4s)
Custom synthetic data (inspired by NVARC by @JFPuget and Ivan Sorkin)
1
15
1,619
I got 37% on ARC-AGI-2 Kaggle without LLMs!
100M looped transformer trained from scratch
- Likely highest score without an LLM
- Peaked at 6th place
- Beats last year's winning LLM
- Looping, added in the last week: 33% → 35% → 37%
thanks to @arcprize @fchollet @mikeknoop and @GregKamradt for the competition!
Link: mvakde.github.io/blog/looped…
(writeup for the paper track)
Author has a history of lying about results
"SWA beats Linear attention" when this is only a very specific case
"TRM is a 7M parameter model that beats much bigger models" when its actually ~440M, with 7M active
Simple beats complicated:
We show that switching to a sliding-window attention mask with attention sinks (at no cost) beats linear attention post-training.
Huge thanks to my collaborators @RheaSukthanker, @CameronPashmina, and @Emy_Aze.
Paper: arxiv.org/abs/2608.28444
Actually lecun says exactly that in the slide before this
I assume the subtext is "for human level ai", build these types of world models.
2
61
irl in a separate interaction I mean
also found a video of this talk. He says "world models should be ..." youtube.com/watch?v=vJKC31Yp…
1
1
52
Then why does it outperform others with such a small transformer / such little training data?
Probably good representations -- ablations point to 3D RoPE and the per-task embeddings
Although I'm sure it'd perform the same if I scale it up very high
2
26
4,165
A big change -> No training on inputs
Interestingly, this makes the test loss much worse and yet it scores better!
Clearly a compression framework / val loss alone isn't a perfect metric for sample efficiency
(I am bullish input training will make a comeback though)



















