
Research Engineer looking at data™️, evals @ProximalHQ prev: CS, Linguistics, MechInterp research @iiit_hyderabad
Bengaluru, IN
Joined February 2018
- Tweets6K
- Following1.5K
- Followers439
- Likes39.4K
Looking at the data™️ was/is the most interesting part about FrontierSWE v2. We're still looking at them and keep finding exciting bits to talk about. Have talked about a few in the blog, and have released all traces, hope this leads to more ppl looking at data™️
Akshit retweeted
One interesting bit of AI discourse: it's pretty broadly agreed that agents aren't great at software design.
But...they're still good at getting tasks done. So, maybe software design doesn't matter?
I think there's a lot of complexity baked into this question!
FrontierSWE mentioned! (biased sample)
I asked 56 people in AI which benchmarks they'd trust most to indicate if an LLM is frontier:
25 votes: Terminal-Bench 4 (@alexgshaw, @andykonwinski, @Mike_A_Merrill)
20 votes: Terminal-Bench Science (@StevenDillmann, @lschmidt3, @sanmikoyejo)
13 votes: FrontierCode (@penlume, @ybenpan, @denizbirlikci, @silasalberti)
10 votes: FrontierSWE (@evan_j_chu, @_rajanagarwal, @MatternJustus, @ardnoKhtnayhsiR, @sanket__mhatre, @akshitkr)
8 votes: Humanity's Last Exam (@longphann_, @hendrycks, Alice Gatti, @ziwen_h, @natliml)
7 votes: Vending-Bench (@axelbacklund, @lukaspet)
5 votes each:
- AutomationBench (@danielwshepard, @SalimansRobin, @TweetAnnaMarie)
- DeepSWE (@winkey_h, @charleyslee, @LeonardMainnet, @serenaa_ge)
- PostTrainBench (@full__rank, @hrdkbhatnagar, @AmyPrb, @shiraeis, @karinanguyen, @MatthiasBethge, @maksym_andr)
The most common comment was that people don't really trust benchmarks at all.
56 people voted. People couldn't pick a bench their company worked on, an index, or a human-voted board. Disclosure: on this list, I'm a small investor in Abundant, Cognition, Emulated, Fleet, Hillclimb, Latch, and Snorkel. I didn't vote.
Counts are net: a few benchmarks received negative votes.
Continued:
11
384
Akshit retweeted
Great paper on why SFT can generalize just as well as RL if you give it enough diverse training data.
Like everyone else since DeepSeek-R1 dropped, I’ve been treating "RL generalizes, SFT memorizes" as gospel.
Whenever SFT runs plateaued on out-of-domain reasoning, everyone just blame the algorithm.
Turns out, we might have been running an unfair comparison the entire time.
In standard setups, RL gets to explore a huge rollout budget per prompt while SFT gets trained on a tiny slice of data.
So the authors decided to actually level the playing field on Qwen2.5 and Qwen3 models by testing two things: mixing data sources instead of just distilling from one giant teacher, and scaling up the SFT data to match RL's rollout budget.
The results completely flip the usual narrative.
A carefully mixed dataset easily beats single-teacher distillation, and matching RL's rollout budget lets SFT generalize just as well as RL.
And when you combine both, SFT actually outperforms it.
To figure out why, they used LLM annotations to inspect the rollouts and proved (both empirically and theoretically on shallow transformers) that larger data budgets expose SFT to rare, "tail" solution methods.
Those uncommon reasoning paths are where the actual generalization signal lives.
Honestly, for everyone tired of debugging finicky online RL pipelines, this is a huge relief.
SFT wasn't broken, we were just starving it of diverse rollouts.
Full paper PDF: arxiv.org/pdf/2611.14912
Akshit retweeted
Meet Gemini 4 Hydrogen: built for rapid coding, multimodal synthesis, and everyday workflows.
It finishes tasks 2.5x faster than 3.8 Flash and surpasses Argon on the @ArtificialAnlys Intelligence Index.
Akshit retweeted
In tiny Rajpur Village, Dehradun,
78 men were marched off to join WWI on behalf of the British Empire
No one seems to know how many men returned home safe
Akshit retweeted
This is a fantastic guide to Kant’s Critique of Pure Reason. It also teaches you some of the basic philosophical concepts Kant relied on when writing the Critique, which new readers may not be familiar with. It has diagrams that make the more abstract ideas easier to follow.
Akshit retweeted
We asked our AI one question about Liam Livingstone: “How do we bowl to him?”
→ The ball that shuts him down is the left-arm spinner’s stock ball: −44.5% efficiency against his own normal game, over 333 balls. Bowl him a slower ball and he feasts: +44.2%.
→ Against pace on a good or short-of-good length, 47.9% of his boundary runs go leg side. The average batter’s figure is 34.9%. The single most valuable fielder is forward square leg, saving 18.3% of his boundary runs.
→ Slog shots make up 20.1% of his runs, against 7.6% for an average batter.
All of it came back in seconds, in plain cricket language.
@taniiishh and I have spent two years in cricket analytics, building metrics and algorithms to read T20 better. One thing kept coming up: coaches and analysts come from different worlds, and good numbers often never reach the dressing room. So we built CricBit. A coach just asks, and the AI explains our metrics the way a coach would.
It’s plug and play with any ball-by-ball data. And we’ve built one of the biggest collections of Indian state T20 league data out there (TNPL, the Maharaja Trophy, the UP T20, the DPL and many more). That’s cricket almost nobody has ball by ball, and it’s where the next auction steals are playing right now.
The IPL auction is just over a month away. If you scout, analyse or coach a T20 side, our DMs are open. Send us a name and we’ll show you what we see.
Akshit retweeted
I asked people which data companies influence research direction at the frontier:
16 votes: Surge (@echen)
13 votes: Proximal (@calvinchen, @MatternJustus)
8 votes: Preference Model (@jennifgcrl, @Ning_Catsnail)
7 votes: Datacurve (@serenaa_ge, @charleyslee)
6 votes each:
- Mechanize (pre-acquisition) (@tamaybes, @EgeErdil2, @MatthewJBar)
- Mercor (@BrendanFoody, @adarsh_exe, @suryamidha)
5 votes: Fleet (@nicoup, @andrewthezhou)
4 votes each:
- AfterQuery (@CarlosGeorgescu, @spencermateega)
- Prime Intellect (@vincentweisser, @johannes_hage)
3 votes: Latch (@AlfredoAndere, @kenbwork, @kylegiff)
2 votes each:
- Abundant (@huyouare, @kehuangsf, @meji_abidoye)
- Artificial Analysis (@_micah_h, @grmcameron)
- Bespoke Labs (@madiator, @AlexGDimakis)
- micro1 (@aliansarinik)
- Terminal-Bench (@Mike_A_Merrill, @alexgshaw)
50 people voted. People couldn't pick a company they work at or founded. Disclosure: on this list, I'm a small investor in Abundant, AfterQuery, Arcanum, DLR, Fleet, Latch, Paper Instruments, Snorkel, and Vmax. I didn't vote.
Continued:
Akshit retweeted
Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models.
At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training.
I hope you all enjoy this write up!🪐🛰️
coreauto.com/hitchhikers-gui…
Akshit retweeted
New paper: Benchmarks Are Surprisingly Compaction Friendly
Agents commonly go past their context windows, and then "compact" to keep going. Pretty much all frontier agents do this, but no one has evaluated compaction on actual long horizon benchmarks
We tested 5 such compaction strategies
Akshit retweeted
Absolutely breathtaking view of the mighty Kosi River in Supaul, Bihar, with the Himalayas painting the horizon.
@TourismBiharGov must look beyond the Buddhist circuit and develop these hidden gems to put Bihar on the tourism map.
Akshit retweeted
New Anthropic research: How AI agents shape experts' judgment calls
When experts delegate work to an agent, do they still make the key decisions? We tested this in a randomized trial with 50 professional data analysts.
Akshit retweeted
Gemini 4 Carbon is our next frontier model.
It’s state-of-the-art across cybersecurity and long-horizon capabilities.
Akshit retweeted
Replying to @MalleableTwig
Yes, sure thing! Here is a link to my slides. (The talks were recorded, but I am not sure whether/when they will make their way online.) docs.google.com/presentation…
A few bowhead whales born around 1815 might still be alive. They were in their 30s when they first saw a commercial whaling ship; in their 60s when they first heard a propellor; over 100 the first time they heard sonar. I wonder if they remember the silent ocean.




























































