
Trying to solve sample efficiency prev: Engineering physics @iitbombay '23
Joined June 2023
- Tweets837
- Following694
- Followers4.3K
- Likes2.9K
Pinned Tweet
Announcing ⚡⚡
1) New records on ARC-AGI public eval !!
2) A new sample-efficiency benchmark
70% on ARC-1 with just 19M params (15% on ARC-2)
49% with just 1M params!!
(Latter beats TRM's 45% with ~400x fewer params)
Trained from scratch on only 1,307 ARC puzzles.
Just ~$13 to train
No text, no LLMs
🧵
Middle layer looping!
This is what helped my ARC model score incredibly high
Try out ideas like this on ARC Data Squeeze to improve sample efficiency!
Some of you thought that Inkling 2 Small was too big at 594B parameters. We heard you: Inkling 2 Mini is a looped transformer with just 26B parameters, small enough to run on your laptop, powerful enough for agentic work with a score of 20.1 on AA Index.
thinkingmachines.ai/news/ink…
Announcing ⚡⚡
1) New records on ARC-AGI public eval !!
2) A new sample-efficiency benchmark
70% on ARC-1 with just 19M params (15% on ARC-2)
49% with just 1M params!!
(Latter beats TRM's 45% with ~400x fewer params)
Trained from scratch on only 1,307 ARC puzzles.
Just ~$13 to train
No text, no LLMs
🧵
1
130
People have already built on top of mdlARC!
@kschweig_ 's transductARC got 76% on ARC-1 with extra data (RE-ARC), scale aug and answer filters
With the same data and tricks, looped mdlARC gets 81% (20% on ARC-2)
These break the rules, so they go on a separate Open leaderboard
teapawt.net/kschweig_/status/20994…
76% on ARC-AGI 1 training from scratch + inference in a bit more than 4h on a single H100.
Throughput-optimized recipe attains 44% (TRM-level performance) in 17min.
Just a simple autoregressive transformer with a few twists. 🧵
2
4
401
Earlier mdlARC threads: v1 (Dec) teapawt.net/evilmathkid/status/200…, v2 (Mar) teapawt.net/evilmathkid/status/202…
Benchmark: github.com/mvakde/arc-data-s…
Baseline code: github.com/mvakde/mdlARC
Blog: mvakde.github.io/blog/looped…
44% on ARC-AGI-1 in 67 cents!
Trained from scratch in 2hrs on a 5090
Matches TRM, beats HRM and is way faster & cheaper
No recursion, just a transformer
Also, 7% on ARC-2 🧵
1
355
Mithil Vakde retweeted
Scaling in almost every domain is bottlenecked by data. That makes data efficiency the most important algorithmic problem in AI.
I'm thrilled to share *universal* data-efficient algorithms with massive gains, up to 56x data efficiency for molecules, 24x for DNA, and 24x for crystals against SOTA autoregressive recipes.
After fine-tuning, we beat models pretrained on hundreds of times more data.
I believe this will fundamentally change our approach to data-constrained scaling across all domains.
w/ @asatheesh6561, @bishmdl76, @madhavsinghal_, @akshayvegesna
we should not like a future where there are 1 to 2 AIs and the rest are illegal. we should want a cambrian explosion of the types and possibilities of minds, experiencing new joys and thinking new thoughts. ensuring that this is compatible with human flourishing is only step 1
Mithil Vakde retweeted
LLMs require very large amounts of data and arguably the path to ASI runs through a different model architecture that’s more data sparse. Amazing what @evilmathkid has done with compute constraints on ARC-AGI-2.
cc @fchollet
I got 37% on ARC-AGI-2 Kaggle without LLMs!
100M looped transformer trained from scratch
- Likely highest score without an LLM
- Peaked at 6th place
- Beats last year's winning LLM
- Looping, added in the last week: 33% → 35% → 37%
thanks to @arcprize @fchollet @mikeknoop and @GregKamradt for the competition!
Link: mvakde.github.io/blog/looped…
(writeup for the paper track)
Mithil Vakde retweeted
And Mithil strikes again, this time casually beating opus
I got 37% on ARC-AGI-2 Kaggle without LLMs!
100M looped transformer trained from scratch
- Likely highest score without an LLM
- Peaked at 6th place
- Beats last year's winning LLM
- Looping, added in the last week: 33% → 35% → 37%
thanks to @arcprize @fchollet @mikeknoop and @GregKamradt for the competition!
Link: mvakde.github.io/blog/looped…
(writeup for the paper track)
Mithil Vakde retweeted
THE GOAT (of resource efficient arc impls) RETURNS
I got 37% on ARC-AGI-2 Kaggle without LLMs!
100M looped transformer trained from scratch
- Likely highest score without an LLM
- Peaked at 6th place
- Beats last year's winning LLM
- Looping, added in the last week: 33% → 35% → 37%
thanks to @arcprize @fchollet @mikeknoop and @GregKamradt for the competition!
Link: mvakde.github.io/blog/looped…
(writeup for the paper track)
I got 37% on ARC-AGI-2 Kaggle without LLMs!
100M looped transformer trained from scratch
- Likely highest score without an LLM
- Peaked at 6th place
- Beats last year's winning LLM
- Looping, added in the last week: 33% → 35% → 37%
thanks to @arcprize @fchollet @mikeknoop and @GregKamradt for the competition!
Link: mvakde.github.io/blog/looped…
(writeup for the paper track)
Why do it without an LLM?
Because I care about sample efficiency. LLMs see so much data, ARC puzzles included, that you can't measure it. We now know that with enough data, ARC-1/2 become trivial
So the interesting question is how far a model gets on a few ARC puzzles alone
1
4
932
This is a scaled up and upgraded version of my previous work on ARC-AGI
teapawt.net/evilmathkid/status/202…
44% on ARC-AGI-1 in 67 cents!
Trained from scratch in 2hrs on a 5090
Matches TRM, beats HRM and is way faster & cheaper
No recursion, just a transformer
Also, 7% on ARC-2 🧵
1
6
910










