Mithil Vakde (@evilmathkid) header
Mithil Vakde (@evilmathkid) profile picture

Trying to solve sample efficiency prev: Engineering physics @iitbombay '23

Joined June 2023
Looping is the biggest driver so far: - 8.4M model: 47% → 66% on ARC-1 when looped - Beats non-looped models of the same or greater logical depth, with up to 4x the params - Tracks other sample-efficiency benchmarks: looping set records on the Slowrun too (caveat - both looped and non-looped can prob be tuned more)
1
3
443
ARC Data Squeeze: How high can you score on ARC-1/2 when trained from scratch on very limited data? Goal: beat GPT-6 Astra's score without LLM pretraining Current record: 70%, Looped mdlARC Rules: - Fixed dataset: 1307 puzzles (~2M tokens) - Any architecture, any compute - no LLMs, DSLs or hard-coded ARC info (everything must be learned) Inspired by @kellerjordan0's NanoGPT Speedrun, Q Labs' Slowrun and @OpenAI's Parameter Golf Repo: github.com/mvakde/arc-data-s…
1
1
4
558
Announcing ⚡⚡ 1) New records on ARC-AGI public eval !! 2) A new sample-efficiency benchmark 70% on ARC-1 with just 19M params (15% on ARC-2) 49% with just 1M params!! (Latter beats TRM's 45% with ~400x fewer params) Trained from scratch on only 1,307 ARC puzzles. Just ~$13 to train No text, no LLMs 🧵
23
24
4
184
16,129
Looping is a huge win: At equal depth, it beat my non-looped stacks with up to 4× the params, by 2.3–7.6% (even at equal depth) Even sub 1M models perform very well! Room for growth -- I ran out of time, so final model is severely undertrained. Performance can go much higher
2
9
1,244
What's new and different: Looping the middle layers All demo pairs in context Hub-and-spoke sparse attention (to fit in 4xL4s) Custom synthetic data (inspired by NVARC by @JFPuget and Ivan Sorkin)
1
15
1,619
I got 37% on ARC-AGI-2 Kaggle without LLMs! 100M looped transformer trained from scratch - Likely highest score without an LLM - Peaked at 6th place - Beats last year's winning LLM - Looping, added in the last week: 33% → 35% → 37% thanks to @arcprize @fchollet @mikeknoop and @GregKamradt for the competition! Link: mvakde.github.io/blog/looped… (writeup for the paper track)
52
37
7
541
25,212
Author has a history of lying about results "SWA beats Linear attention" when this is only a very specific case "TRM is a 7M parameter model that beats much bigger models" when its actually ~440M, with 7M active
6
7
2
272
41,619
Who dat
moved to bangalore recently to build. traffic is wild. but people are nice. i definitely feel the ambition + energy accelerating. we need to accelerate scientific discovery. biology is the ultimate hill to climb. but before that, we have work to do. excited for the next decade.
1
12
2,795
Actually lecun says exactly that in the slide before this I assume the subtext is "for human level ai", build these types of world models.
2
61
Replying to @yacinelearning
Yeah some of the tasks it can solve are not easy!
2
259
Then why does it outperform others with such a small transformer / such little training data? Probably good representations -- ablations point to 3D RoPE and the per-task embeddings Although I'm sure it'd perform the same if I scale it up very high
2
26
4,165
A big change -> No training on inputs Interestingly, this makes the test loss much worse and yet it scores better! Clearly a compression framework / val loss alone isn't a perfect metric for sample efficiency (I am bullish input training will make a comeback though)
3
2
2
41
14,279
44% on ARC-AGI-1 in 67 cents! Trained from scratch in 2hrs on a 5090 Matches TRM, beats HRM and is way faster & cheaper No recursion, just a transformer Also, 7% on ARC-2 🧵
30
75
14
772
66,736