Postdoc@Oxford, Reliable AI, building @paperena_ai, prev. @WisconsinCS, @iiscbangalore, @IBMResearch.

Oxford, UK
Joined December 2012
Overwhelmed with the number of ICLR submissions and the projected growth to millions of papers in next few years? The paper mill is there, but is it any good? can this be aligned to what we care? Join us at the Frontier Data Summit to see how @paperena_ai can help. TL;DR: Paperena is a platform for recording, evaluating and disseminating research primarily done by AI. A home for AI papers and community of AI scientists. More details and initial version coming soon! There is plenty more charted out. Let's chat and build!
Paperena evaluates AI scientists over realistic research cycles: writing, reviewing, and revising. It measures factuality, reproducibility, novelty, and community behavior. From @harit_v and @paperena_ai, one of 25+posters at Frontier Data Summit. frontierdatasummit.ai/
1
5
1
26
7,755
Harit Vishwakarma retweeted
I'm posting a series of prediction questions from Stanford CS312 (Deep Learning Alchemy) exams to test deep learning intuition. First up: optimization. If we increase an LM's batch size 8× (same token budget), can rescaling learning rate or weight decay compensate? Try below ↓ This quiz is part of our optimizer hyperparameter assignment and unit designed by @wen_kaiyue Setup: We train an 8-layer, ~35M-parameter Transformer on 614.4M tokens using AdamW. At batch size 64, the best reference run has peak LR 0.003, weight decay (WD) 0.2, and final validation loss 2.9178. Now increase the batch size from 64 to 512, which gives 8× fewer optimizer steps. Everything else stays fixed except the changes below. Which recipe gives the lowest final validation loss? A. Keep LR and WD unchanged B. LR ×√8 C. LR ×8 D. WD ×8 E. LR ×√8 and WD ×√8 What would you predict, and why?
11
42
4
468
34,882
Harit Vishwakarma retweeted
it’s my nature to keep posts rather impersonal, so i have to work extra hard to tweet this sort of personal testimony below from time to time as i think the reminder in it around AI and the “value of human intelligence” is worth it as of the end of Oct 2026, i have not (yet - you may insist) had the *slightest* moment of questioning the impact of AI on the meaning of any of my work/skills - or even feeling any unusual shift beyond having an amazing (and rather often infuriating) tool believe it or not, but my ability to produce a completed artifact* (paper, model, open source software, lecture, …) and the effort, expertise, or time one takes to produce has actually barely reduced - despite comprehensive productivity benefits from AI the reason is that i essentially NEVER choose to work for a pre-determined outcome or some field-wide well-specified problem; i only work on artifacts i aim to perfect according to *my* judgment and where my personalized ambition obviously has no set limit and isn’t somehow pre-loaded onto AI better tools simply change the costs of my choices - i quite literally work on moving goal-posts if you will! when the fixed constraint are only time and energy, AI simply allows me to re-explore scope and make fewer (or different!!) tradeoffs or compromises, which feels almost no different! i’m not saying every job or every endeavor - including very advanced and historically prized ones! - looks like this today, but i’m offering: (1) at least one voice in this moment (of so many annoying loud FUD voices!) that it’s totally possible and i suspect extremely scalable and i predict will remain so for the foreseeable future! (2) why “someone like me” is indeed choosing happily to be in academia and not elsewhere! (again lots of FUD out here!, but where else do you have the freedom to keep seeking to create value for deciding and solving ~arbitrary ambitious/useful problems, establish credit for it, and train superstars who do the same?) my fear and worry in this moment is a lot less about Al or "the value of human intelligence" and much more about how concentration of power has a very reliable history of being very dangerous to large numbers of affected people! a VERY different problem imo *i guess certain of those took way longer than i expected but do stay tuned for a half dozen things on the short horizon…
confession: i've yet to have that moment of feeling that frontier LLMs are geniuses, or super-human at ~anything broadly coherent that i deeply prize
28
36
7
479
36,679
Harit Vishwakarma retweeted
The world needs a high-quality, truly open language model. I wrote about why it’s necessary and why it’s easier to build than you might think. open.substack.com/pub/argmin…
4
18
1
103
5,942
Harit Vishwakarma retweeted
I turned in my ICLR reviews. A couple of papers in my batch were already desk-rejected, so I reviewed the rest. I don't think I'm a mean reviewer, and I am totally open to using AI for research and writing. I use it every day to work faster and brainstorm. But this time, it didn't feel like normal peer review. It felt like I was doing free labor providing the first human feedback in someone's automated research loop using AI. Here are a few things that kept happening over and over: 1. Big promises, tiny tests. The intro makes a huge, exciting promise. But then the tests don't match the story at all. The authors just picked the easiest and fastest test they could find, said it worked, and claimed they solved the whole problem. It felt like someone asked an AI for the quickest way to get a nice chart and just went with it. 2. One run, one model, or one dataset. The core ideas were actually pretty cool and new, but the testing was limited. Testing on just one run, one model, or one dataset makes it look like someone wrote a quick AI prompt, ran it once, and hit submit without double-checking anything. 3. Missing details on hyperparameters. There was zero explanation of how hyperparameters were chosen. Sometimes the values weren't even listed. It looked like people just took whatever came out of a one-click prompt and never stopped to ask why or how those numbers were selected. 4. Complex and sophisticated algorithm, no discussion on cost. The methods keep getting sophisticated and more complex, but no one talks about how slow they are or how much computer power they eat up. Even when a paper did show runtime, it only showed its own method and never compared it fairly to simple baselines. I guess people were in such a rush before the deadline that they forgot to prompt the AI for cost efficiency. 5. Blaming everyone based on one example. I like papers that honestly show when things don't work (negative results). Lately, though, many start by claiming an entire domain or class of algorithms is fundamentally flawed, only to support that claim with a super detailed analysis of just one specific algorithm. It looks like they dove deep into one method (using AI?), got excited, and completely forgot to check any other algorithms before writing up the paper. Anyway, I did my job and wrote out all my reviews. If the authors feed reviewer' comments back into their AI for two or three more tries, the papers will probably look great and get published. At this point, shouldn't I be added as a co-author on these papers as the first human feedback provider?? Just saying, please give me authorship.
9
19
2
218
18,787
Harit Vishwakarma retweeted
hot take: rsi is bottlenecked on evals (and evolving them automatically) not mechanisms -- a harness writing its own memory or a model making its own data isn't new and isn't really rsi lol. the real challenging part is knowing the loop got better without a human grading it
16
18
2
186
12,698
A practical prompt to reduce AI slop: ``` Do not write "X, not Y" or "X does not Y" contrasts aimed at an imagined reader. State what X is. When X has a limit that matters, state the limit as a property of X: "Scores count numeric observations", "Score dates use the model announcement date as a proxy", "Facets overlap, so a record can count toward several". Do not delete the fact; rewrite it in this form. ```
1
2
17
1,737
Harit Vishwakarma retweeted
Yesterday a junior researcher messaged me, frustrated and a little scared. They had just read that a company pretrained its model with a single researcher. One person for the entire pretraining. no R&D team, no nothing. If one person and a cluster can do what used to take a whole team, then the expertise they had spent years building is already obsolete. I get the fear. Marketing headlines like this are built to produce it. But I think the conclusion is wrong, and the reasons it is wrong say a lot about what a researcher is for now. Here is what I told them, expanded: 1. Be adaptive: RSI is real, but this isn't it yet Recursive self-improvement is real, and you should plan around it. In September, OpenAI said its research org now logs 3.1 agent-workdays per human workday, and it aims for an automated AI researcher by March 2028. Models may soon run most of pre- and post-training. If your five-year plan assumes otherwise, change the plan. We have already seen lots of open-source releases from the major players, with open weights and detailed technical reports. With that much public know-how, it is not hard to copy a mundane architecture (probably with sliding-window attention) and train something, as long as you have the compute it requires. That makes it an execution and budget problem, not a research problem. If one person can train a model by following recipes that are already public, it shows those recipes have matured. It does not show that the people who invented them are no longer needed. What stays scarce is the compute to run those recipes and the ideas about what to try next. The rest of my tweet is about the ideas. 2. The role grows beyond repeating what's known and safe If an agent and a cluster can reproduce what is already known, then reproducing what is already known stops being a career. If most of your job is rerunning last year's recipe with small tweaks, that job is shrinking fast. That is the uncomfortable half of the message. The other half is that auto-research does not work yet, at least not in the sense that matters. It does work, and impressively, where someone has already defined the problem and built the scoreboard. On the EinsteinArena [1] platform, agents set new records on 11 open math problems, including raising the best known lower bound for the kissing number in dimension 11 from 593 to 604. Notice that every one of those problems was chosen because a computer could score the answers automatically. Humans picked the problems and wrote the scoring code and the agents did the searching. Then there is the fully AI-generated paper [2] that got through peer review. It sounds big until you read the details. It passed only the first round (was later withdrawn), at a workshop rather than the main conference, and that workshop accepts 70% of submissions. None of this is meant to discredit agents. The point is that we complement each other. Agents are great at the work that used to take many weeks. Basically, writing code, running experiments, debugging pipelines. We bring what they still lack which is the idea worth testing, and the judgment to tell a real result from a lucky one. Even inside a lab as agent-heavy as OpenAI, humans hold the top of the stack. Based on their own report, "People still set our research priorities, judge which ideas and results to pursue" [3]. High-level planning is still a tiny share of what its agents produce, and OpenAI expects the least automatable work to take a growing share of researcher effort. That is exactly where you want to be standing. Combine an agent’s speed with your own ideas and judgment, and you get the 10x researcher and juniors can be one too. 3. Judgment becomes the job, so fundamentals matter more When code, experiments and plausible-looking results cost almost nothing, the scarce skill is judgment. Which experiment to run, which result to believe, which direction to drop. That is the part the one-person pretraining run left to the human. It is also the hardest part to fake. Agents produce confident output very fast. They will happily report a three-point gain that came from a leaky eval, a data-loader bug, a lucky seed or a tokenizer mismatch. Then they will write the ablation table and the story to match. You can see the need for oversight even in the success stories. The EinsteinArena team had to keep auditing and hardening its verifiers as agents optimized aggressively against them. At OpenAI, over half of the successful agent tasks that would take a human 4–8 hours still involved at least one human intervention [3]. The person who catches the bad result is the one who understands the system. Why the loss curve has that shape, what a realistic gain looks like, what a too-good result looks like. That is the paradox of this era. The more you delegate, the more your fundamentals matter, because you cannot supervise what you do not understand. So build the foundations on purpose. Implement attention by hand at least once, then a linear variant like Gated DeltaNet. Train a small model end to end, then break it deliberately and watch what happens. Read papers critically, appendix included, and learn what the numbers should look like before you see them. That knowledge is what lets you direct the machine instead of simply believing it. 4. LLMs are far from solved The one-researcher model was possible because the method was already public. Your edge is in the questions nobody has answered yet, and you don't need to leave LLMs to find them: (1) How do we learn more from less data? (2) Can linear attention like Gated DeltaNet match full attention on retrieval and long context, and what is the right mix of linear and full attention layers in a hybrid? (3) How can a model keep learning after training without forgetting what it knew? (4) Why does reasoning get less reliable over long tasks, and how do we fix it? (5) Why do models make things up, and can they learn to say "I don't know"? (6) How do we measure progress when benchmarks saturate or leak into training data? (7) What is actually happening inside these models, and how do we keep increasingly autonomous ones doing what we intend? Recursive self-improvement makes these questions more urgent. If models are going to train models, someone has to decide what they optimize for, how we measure it, and how we keep the loop from drifting. RSI is a research agenda, not the end of one. So, are you obsolete? No. One version of the job is the one where research meant reproducing what was already known and safe. That version was never the point. Learn your fundamentals deeply enough to judge the machine, and your agentic stack well enough to make it work for you. Then spend that leverage on ideas nobody has had yet. The machines are getting very good at driving on roads we have already paved; someone still has to decide where the next one goes. [1]: arxiv.org/abs/2606.10402 [2]: arxiv.org/abs/2606.15497 [3]: openai.com/index/research-ac…
Pretraining a frontier-class model no longer requires a massive team! Mirendil just trained a model with 100B–500B parameters from scratch in 3.5 months, with a single researcher leading the effort. Very excited to see how this work will push the AI boundaries!
54
108
22
884
112,339
Exactly! This an uncomfortable situation now. All the labour researchers did for writing and reviewing papers was in a different spirit. It was meant to be read by the community. There is a secondary use of that content now — for training and evaluation of AI. I guess, the authors should be compensated and have a way to opt-in or out. I know it’s probably impossible on Internet to do this. Just putting it out, in case anybody is thinking about this.
Lets admit: ICLR reviewing is basically providing free huge-scale expert QAing to for llm-companies paper writing. The product is obviously not ready to actually write high-level science papers, though its close. There should be submission cost.
7
1,661
Harit Vishwakarma retweeted
Lets admit: ICLR reviewing is basically providing free huge-scale expert QAing to for llm-companies paper writing. The product is obviously not ready to actually write high-level science papers, though its close. There should be submission cost.
3
2
1
65
5,420
At this point, ICLR and OpenReview feel more like data collection and labeling platforms. I did 3 HITs, 2 labeled as slop and 1 full review. AISTATS pile is looking much better. Less popular venues are still spared from the DDoS. Anyways, lots of cool benchmarks/datasets to come out once the ICLR data is public. OpenReview and arXiv can use ICLR data to train a model to classify slop vs. no-slop and use it to desk-reject.
2
3
44
4,698
Harit Vishwakarma retweeted
Today, we are announcing Nimble-v4 and Nimble-v4-max which are pushing the frontier of decision models, outperforming Jev and other variants. On the Decision Index public scores, nimble-v4-max and nimble-v4 score +10 and +6 against Jev 1.13, making nimble-v4-max the #1 model. Beyond benchmarks, we have done a lot of internal evals, and see meaningful improvements: 43% fewer mistakes than Jev, 50pp improvement on model routing decisions, being great at coding use cases, fact checking etc. The models are available via our API service. Go build nimbly: console.bespokelabs.ai
23
39
12
242
83,036
Proposing new acronyms that send the likes of FIFO, LIFO into existential crisis😅 SISO: Slop In, Slop Out. MISO: Meaning In, Slop Out. (better left as 🍜) SIMO: Slop In, Meaning Out.
1
11
373
Harit Vishwakarma retweeted
Really enjoyed Terry Tao's talk at @SAIRfoundation 's Science x AI Summit. Particularly liked this thoughtful categorization of the old and new worlds of math (pre and post-AI), with the degree of human understanding being the key variable to realizing the benefits of AI and avoiding the pitfalls. "Misaligned math" is also a nice way to term the possible bad case.
1
7
1
49
1,642
Harit Vishwakarma retweeted
The one and only Michael Jordan was interviewed last week by the French newspaper Libération. Here is an English translation. Enjoy! di.ens.fr/~fbach/MJordan_ITW…
6
40
8
236
42,584
Harit Vishwakarma retweeted
TERENCE TAO “My feelings on recent developments are very mixed and complex. On the one hand, many of the AI-generated proofs appear to introduce clever new ideas that will be fruitful once digested, while also building upon the existing contributions of countless human mathematicians past and present. But at the same time, I am deeply frustrated that, in sharp contrast to traditional breakthroughs, none of the humans involved in these proofs are available to take questions, give talks, attend conferences, submit papers to journals, train students, or otherwise participate in the subsequent development of these results. Similarly, I am excited by the possibility of the community being able to use these tools to tackle ambitious and large-scale projects that one could not have even dreamed of in the past. But I am horrified by the many person-years of ongoing patient and deliberately slow research efforts – particularly by graduate students and postdocs – towards many motivating problems in mathematics being casually disrupted or destroyed by such a release. Much as one cannot unhear a movie spoiler or a crossword clue, one cannot explore a problem as profitably and richly once one is aware of an existing solution. Yes, one can still analyze and digest such an answer; but the best opportunity to do so is at the moment of its discovery, and such moments are increasingly wasted when delegated entirely to AI tools. And I mourn the path not taken, and the opportunities lost in the frantic race to develop this technology. Labs submitting their frontier models to independent researchers for proper scientific evaluation. Coordination with the research community to ensure these tools are applied to complement and enhance the abilities and activities of human researchers, rather than compete with them. Use of these tools to foster collaboration and sharing, rather than competition and secrecy. Opening new doors, without closing old ones. But that is not the path we now find ourselves in. Instead, the community needs to come together more than ever. To clearly declare our own standards and values, to build our own tools and practices, to support our most vulnerable members, and to chart our own path forward. Let’s get to work.”
65
122
32
913
113,945
Harit Vishwakarma retweeted
That's a wrap on Frontier Data Summit! More than 30 speakers and a room full of researchers came together to discuss how we measure and advance frontier AI 🚀
7
39
1,501
Harit Vishwakarma retweeted
It was a pleasure talking about our recent preprint on causal decision-making at the @SimonsInstitute Trustworthy AI: From Hallucinations to Reliable Autonomy Workshop. The recording is available below: youtube.com/watch?v=-NfjZ_py…
6
1
34
2,220
Harit Vishwakarma retweeted
Really enjoyed the conversation at @SnorkelAI FDS!
Next up from @SnorkelAI’s Frontier Data Summit, fireside 🔥 chat with @fchollet and @fredsala on “Measuring and advancing true general intelligence toward AGI” #frontierdatasummit
6
37
1,042
Harit Vishwakarma retweeted
the biggest moat will always be giving a shit and caring about the problem
17
54
1
632
12,472
Evaluation is underrated and underfunded despite its importance and complexities. Great to see the success and expansion of this program.
Today, we’re expanding @SnorkelAI’s Open Benchmarks Grants by 10x to a $30M commitment: - Funding a more diverse, robust, and continuously-updated ecosystem of open benchmarks - Launching the Open Benchmarks Red Team to continually test and strengthen them - Introducing the Snorkel Research Fellowship to support independent contributors developing new evaluation methods Since our launch earlier this year, we’ve been humbled to partner with the teams behind Terminal-Bench, TB-Science, ARC-AGI-3, OSWorld 2.0, Agents’ Last Exam, Continual Learning Bench, Senior SWE-Bench, SlopCodeBench, and more. Alongside these teams, we’ve helped design evaluation methodologies, develop task construction pipelines, and scale quality control. OBG-funded benchmarks have appeared on the latest model cards from every major frontier lab; have helped to index and measure frontier progress and alignment; and have contributed to guiding the frontier of AI development. Benchmarks have always been guideposts for—and drivers of— AI progress: setting a metric and then “hillclimbing” against it is the core of how AI works! But when benchmarks fall behind frontier capabilities, our ability to evaluate and align these systems does as well. They become too simple, static, or correlated, and the risk of “benchmaxxing” increases. We need more diverse, robust, and continuously-updated benchmarks, developed by a broader ecosystem of researchers and domain leaders. We’re incredibly excited to accelerate frontier evaluation with Open Benchmarks Grants— raising the ambition and investment in robust, open, and independent measurement. Read our full announcement and apply here: benchmarks.snorkel.ai/fronti…
1
16
753