Siddharth Joshi (@sjoshi804) profile picture

Leading Multimodal & Post-Training Data Curation at @DatologyAI | ML PhD @UCLA | Prev @MSFTResearch AI Frontiers

Westwood, Los Angeles
Joined August 2017
Photo by Siddharth Joshi (@sjoshi804)Photo by Siddharth Joshi (@sjoshi804)Photo by Siddharth Joshi (@sjoshi804)Photo by Siddharth Joshi (@sjoshi804)Photo by Siddharth Joshi (@sjoshi804)Photo by Siddharth Joshi (@sjoshi804)Photo by Siddharth Joshi (@sjoshi804)Photo by Siddharth Joshi (@sjoshi804)Photo by Siddharth Joshi (@sjoshi804)Photo by Siddharth Joshi (@sjoshi804)
Five years ago, I left a comfortable software engineering job in Big Tech to start a PhD. Last year, I left the PhD to join Datology. Both decisions confused the people around me, and honestly both decisions were about the same thing: I wanted to do research. Not research as in chasing paper deadlines and applying for fellowships / grants, but research in the truest sense of the word - sitting with unsolved, sometimes previously unheard-of problems, contextualizing them, formulating them, exploring solutions to them. I'd had a taste of research in college, flitting between disciplines, but never found something I felt truly passionate about until I came across deep learning. A field mixing empiricism, mathematics, and real-world impact all seamlessly - it made research the most exciting thing I'd ever done in my life. So in 2022 I started my PhD hoping for the chance to explore uncharted frontiers. Three years and several papers at the standard prestigious ML conferences later, I had technically done research. But I still didn't feel like I'd ever had the freedom, support, and resources to explore new and exciting ideas. This is what brought me to Datology as an intern last summer. A hope to do research in the true sense - explore new ideas, supported by my peers and leaders, unconstrained by resources. And of course, about the data. At the end of the summer, I took a risk and stayed, putting my PhD on hold. Since then, I've been lucky enough to grow into leading multimodal data curation at DatologyAI, and with our team we've tackled every challenge possible: the engineering and optimizing of a VLM training stack we built from scratch; the at-times frustrating but ultimately rewarding deep refining of VLM evals in our work DatBench (link); and of course a lot of exhilarating new research on DATA CURATION. But more than anything, I felt like I finally got to do research!! I'd like to specifically thank @arimorcos and @leavittron who entrusted me with this opportunity, empowered me to do the best work of my life (so far), and mentored me to grow not only as a researcher but also as a leader. And a huge thanks to the @datologyai team that made research feel FUN again. Today, we're releasing 20/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone. This is the culmination of the multimodal team at Datology's work over the past year. At fixed architecture, recipe, and compute, varying only the pretraining data, we get +11.7pp at 2B across 20 public VLM benchmarks, beat InternVL3.5-2B by ~10pp at ~17x less training compute (without post-training), and hit near-frontier accuracy at 4B with 3.3x lower response FLOPs than Qwen3-VL-4B. Take risks. Bet on yourself. I’m going to keep doing this. At least until my luck runs out :) a 🧵
14
33
13
331
793,684
another banger from @datologyai labs!!
1/7 New Labs Research: Follow the Gradient SafeTulu filters benign-looking data that weakens LLM safety, cutting attack success up to 22% relative while largely preserving capability. 🤗 Space: huggingface.co/spaces/Datolo… Data: huggingface.co/datasets/Dato… Paper: datologyai.com/blog/follow-t…
1
1
15
837
Siddharth Joshi retweeted
my friend’s halloween costume was me as a @datologyai intern 😭😭😭😭😭😭😭😭😭 @gauthams47 DATA RESPECT
3
5
1
32
2,196
Siddharth Joshi retweeted
Very cool work! We see more and more evidence that training should be viewed jointly rather as discrete phases, which is often more an organizational artifact than a scientific choice. And of course, more evidence that data remains the most critical factor.
Can we unify all phases of foundation model training into a single algorithm? We believe we've found a compelling answer to this fundamental scientific question, with important practical implications! Historically speaking, foundation model training has evolved in stages: pretraining, midtraining, SFT, RLHF, RLVR… Each stage comes with its own algorithms, from cross-entropy to GRPO, and is often optimized in isolation. But are these really different learning problems, or instances of the same underlying principle? As we looked into it, we found that all roads lead to maximum likelihood. Pretraining, midtraining, and SFT are already maximum likelihood. RL seems to be the exception, but fortunately our earlier work on Maximum Likelihood Reinforcement Learning (MaxRL) provides the missing piece! We extend this perspective to other RL objectives, including REINFORCE, allowing us to unify the entire training process under a single likelihood-based algorithm. Why does this matter? Two reasons. First, scientifically, it reveals a common optimization principle underlying foundation model development, from start to finish. Second, and perhaps more importantly, it shifts the focus from optimizing algorithms to optimizing data. Once the training algorithm is fixed, the key questions become: What data should the model see, in what proportions, and in what order? Instead of optimizing each stage separately, we can optimize the entire training trajectory for what ultimately matters: the capabilities of the final model. Our experiments show that unified training can improve accuracy, coverage, and compute efficiency over conventional staged training. I hope this opens the door to a new science of foundation model training! Huge congratulations to @yus167 , @ZAlvin39105 , and @FahimTajwar10 for turning this ambitious vision into a solid scientific contribution!
1
7
61
3,132
Siddharth Joshi retweeted
New blog post from our group! Do you create dashboards / HTML pages with lots of AI-generate texts, and get frustrated (like us) about how they're hard to read? Check out our tips here (and, send them to your coding agents)!!
74
56
24
717
95,373
one of most unique, off-the-beaten-path, research problems I've worked on in a while such a fun time having some mind-bending discussions with @parthsur and @leavittron around the time-traveling synthetic data
I got back from paternity leave three days ago, and having a new baby son rocks. But I’m going to take a moment to share something else that’s been on my mind: time travel. Specifically, that synthetic data has a time-travel problem. And that I worked on a pretty cool solution to it
1
15
716
Synthetic data is overtaking real data as the fuel for the next generation of models. And the reason is control. Real data is whatever the web happened to contain. Synthetic data is what you decide your model should learn from. The catch: building strong, controllable properties into synthetic data, at scale, in ways that actually matter for your business, is a really hard science problem. That’s what BeyondWeb-ex-ante is about. It’s how we’re doing it for trading firms: synthetic data with exactly the properties you need, like being able to safely backtest your LLMs. This is the first of many places we can do this
When a model rephrases documents the knowledge cutoff guarantees are lost. It can introduce information its sources never had. This is catastrophic for backtesting. So we built a way to generate synthetic pretraining data with measurable, controllable knowledge cutoff guarantees
1
1
1
21
1,625
Siddharth Joshi retweeted
Some insider knowledge: every iteration over the baseline caught leaked claims so much more precisely that we had to grow the measurement set by an order of magnitude each time just to find any survivors. And we still came up empty at the end :) @leavittron, @sjoshi804, the rest of the @datologyai team and I put a lot of love and nerd-power into this, hope you all find it interesting and useful!
When a model rephrases documents the knowledge cutoff guarantees are lost. It can introduce information its sources never had. This is catastrophic for backtesting. So we built a way to generate synthetic pretraining data with measurable, controllable knowledge cutoff guarantees
1
4
1
17
983
Siddharth Joshi retweeted
We're live at 10am PT today: why data quality is the compute multiplier, a Curation Studio demo from @hurrycane, and the research behind the 6x from @lukemerrick_. @josh_wills moderates, and we're taking your questions live. events.datologyai.com/curati…
5
1
23
1,721
this is a really refreshing take on RSI! esp. the point on current RSI results demonstrating the ability of models to reproduce existing results or those close to the convex hull of what is already known in such a world, work on something truly unsolved & uncertain
Yesterday a junior researcher messaged me, frustrated and a little scared. They had just read that a company pretrained its model with a single researcher. One person for the entire pretraining. no R&D team, no nothing. If one person and a cluster can do what used to take a whole team, then the expertise they had spent years building is already obsolete. I get the fear. Marketing headlines like this are built to produce it. But I think the conclusion is wrong, and the reasons it is wrong say a lot about what a researcher is for now. Here is what I told them, expanded: 1. Be adaptive: RSI is real, but this isn't it yet Recursive self-improvement is real, and you should plan around it. In September, OpenAI said its research org now logs 3.1 agent-workdays per human workday, and it aims for an automated AI researcher by March 2028. Models may soon run most of pre- and post-training. If your five-year plan assumes otherwise, change the plan. We have already seen lots of open-source releases from the major players, with open weights and detailed technical reports. With that much public know-how, it is not hard to copy a mundane architecture (probably with sliding-window attention) and train something, as long as you have the compute it requires. That makes it an execution and budget problem, not a research problem. If one person can train a model by following recipes that are already public, it shows those recipes have matured. It does not show that the people who invented them are no longer needed. What stays scarce is the compute to run those recipes and the ideas about what to try next. The rest of my tweet is about the ideas. 2. The role grows beyond repeating what's known and safe If an agent and a cluster can reproduce what is already known, then reproducing what is already known stops being a career. If most of your job is rerunning last year's recipe with small tweaks, that job is shrinking fast. That is the uncomfortable half of the message. The other half is that auto-research does not work yet, at least not in the sense that matters. It does work, and impressively, where someone has already defined the problem and built the scoreboard. On the EinsteinArena [1] platform, agents set new records on 11 open math problems, including raising the best known lower bound for the kissing number in dimension 11 from 593 to 604. Notice that every one of those problems was chosen because a computer could score the answers automatically. Humans picked the problems and wrote the scoring code and the agents did the searching. Then there is the fully AI-generated paper [2] that got through peer review. It sounds big until you read the details. It passed only the first round (was later withdrawn), at a workshop rather than the main conference, and that workshop accepts 70% of submissions. None of this is meant to discredit agents. The point is that we complement each other. Agents are great at the work that used to take many weeks. Basically, writing code, running experiments, debugging pipelines. We bring what they still lack which is the idea worth testing, and the judgment to tell a real result from a lucky one. Even inside a lab as agent-heavy as OpenAI, humans hold the top of the stack. Based on their own report, "People still set our research priorities, judge which ideas and results to pursue" [3]. High-level planning is still a tiny share of what its agents produce, and OpenAI expects the least automatable work to take a growing share of researcher effort. That is exactly where you want to be standing. Combine an agent’s speed with your own ideas and judgment, and you get the 10x researcher and juniors can be one too. 3. Judgment becomes the job, so fundamentals matter more When code, experiments and plausible-looking results cost almost nothing, the scarce skill is judgment. Which experiment to run, which result to believe, which direction to drop. That is the part the one-person pretraining run left to the human. It is also the hardest part to fake. Agents produce confident output very fast. They will happily report a three-point gain that came from a leaky eval, a data-loader bug, a lucky seed or a tokenizer mismatch. Then they will write the ablation table and the story to match. You can see the need for oversight even in the success stories. The EinsteinArena team had to keep auditing and hardening its verifiers as agents optimized aggressively against them. At OpenAI, over half of the successful agent tasks that would take a human 4–8 hours still involved at least one human intervention [3]. The person who catches the bad result is the one who understands the system. Why the loss curve has that shape, what a realistic gain looks like, what a too-good result looks like. That is the paradox of this era. The more you delegate, the more your fundamentals matter, because you cannot supervise what you do not understand. So build the foundations on purpose. Implement attention by hand at least once, then a linear variant like Gated DeltaNet. Train a small model end to end, then break it deliberately and watch what happens. Read papers critically, appendix included, and learn what the numbers should look like before you see them. That knowledge is what lets you direct the machine instead of simply believing it. 4. LLMs are far from solved The one-researcher model was possible because the method was already public. Your edge is in the questions nobody has answered yet, and you don't need to leave LLMs to find them: (1) How do we learn more from less data? (2) Can linear attention like Gated DeltaNet match full attention on retrieval and long context, and what is the right mix of linear and full attention layers in a hybrid? (3) How can a model keep learning after training without forgetting what it knew? (4) Why does reasoning get less reliable over long tasks, and how do we fix it? (5) Why do models make things up, and can they learn to say "I don't know"? (6) How do we measure progress when benchmarks saturate or leak into training data? (7) What is actually happening inside these models, and how do we keep increasingly autonomous ones doing what we intend? Recursive self-improvement makes these questions more urgent. If models are going to train models, someone has to decide what they optimize for, how we measure it, and how we keep the loop from drifting. RSI is a research agenda, not the end of one. So, are you obsolete? No. One version of the job is the one where research meant reproducing what was already known and safe. That version was never the point. Learn your fundamentals deeply enough to judge the machine, and your agentic stack well enough to make it work for you. Then spend that leverage on ideas nobody has had yet. The machines are getting very good at driving on roads we have already paved; someone still has to decide where the next one goes. [1]: arxiv.org/abs/2606.10402 [2]: arxiv.org/abs/2606.15497 [3]: openai.com/index/research-ac…
1
2
18
1,365
Siddharth Joshi retweeted
1/4 Today, we’re releasing Mercury 3. Mercury 3 is by far our most powerful model and the most intelligent diffusion LLM ever released. It outperforms GPT-6 Luna and Haiku 5.5 on quality while running at 1000+ tok/sec.
🤖 Made with AI
291
455
230
5,262
490,662
Siddharth Joshi retweeted
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated. Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
78
119
59
1,626
196,884
Siddharth Joshi retweeted
Do we want this in NeMo Automodel? thoughts?
Zephon’s TorchTitan integration preserves identical global batches when training resumes on a different GPU count. In our latest blog’s 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run. Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed copy of the corpus. The data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming. The lane count must divide evenly across data-parallel replicas, and each optimizer step’s batch must split evenly across lanes. Training losses can still differ because floating-point reduction order depends on GPU count. ✍️ @datologyai @josh_wills @MaxiBoether, Ties Robroek (@datologyai) Read more: pytorch.org/blog/bringing-el…
2
1
14
726
Siddharth Joshi retweeted
I personally love using TorchTitan + Zephon.
Zephon’s TorchTitan integration preserves identical global batches when training resumes on a different GPU count. In our latest blog’s 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run. Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed copy of the corpus. The data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming. The lane count must divide evenly across data-parallel replicas, and each optimizer step’s batch must split evenly across lanes. Training losses can still differ because floating-point reduction order depends on GPU count. ✍️ @datologyai @josh_wills @MaxiBoether, Ties Robroek (@datologyai) Read more: pytorch.org/blog/bringing-el…
1
2
8
447
few years ago, data was in silos fast forward: data pools exist (tho to some extent) but data research is in silos imo, everyone in their niche knows something different about data that we don’t. Datology Labs brings together and advances data research like never before 🚀
Today we’re launching Datology Labs: a new commitment to open and collaborative research into the science of data. Research has been a core pillar of DatologyAI from the start. Now we’re partnering with researchers at universities and across industry to take it further.
1
17
287
We’re thrilled to collaborate with the talented researchers in academia pushing data research forward. As the frontier data research company, we see it as our responsibility to help grow a community that has historically been underrepresented. Interested in becoming a Labs Fellow? Reach out to us!
Today we’re launching Datology Labs: a new commitment to open and collaborative research into the science of data. Research has been a core pillar of DatologyAI from the start. Now we’re partnering with researchers at universities and across industry to take it further.
2
30
1,003
Siddharth Joshi retweeted
Gemini 4 Argon is out! Give it a try and let us know what you think!
Gemini 4 Argon is here. It’s available globally today for developers and enterprises in the paid tier of the Gemini API and the Gemini platform, for Ultra subscribers in Google Antigravity, and Ultra and Pro subscribers in the Gemini app and Search. Hope you all enjoy it!
4
25
1,635
Siddharth Joshi retweeted
Introducing ThunderSync, a new approach to RL and OPD on long-horizon agentic tasks. Post-training methods like GRPO and OPD on long-horizon tasks leave trainer engines idle during rollout, verification, and reward measurement. To avoid this, asynchronous approaches overlap rollout and training across updates, but policy staleness reduces accuracy. ThunderSync is a new method to squeeze out these bubbles in agentic post-training without policy staleness. For GRPO, we compute each trajectory’s score gradient as soon as its reward is ready, without waiting for the group. For OPD, we compute gradients for each turn as soon as the teacher scores it, without waiting for the full trajectory, so training overlaps with tool execution and later turns. We show that this is exactly the same update as batch-synchronous training. The result is up to 1.9x faster RL while matching the performance of synchronous updates. Paper: arxiv.org/abs/2610.05935 Code: github.com/seilk/thundersync Blog: seilk.github.io/thundersync
Introducing ThunderSyncRL ⚡ It makes agentic RL up to 𝟭.𝟵× 𝗳𝗮𝘀𝘁𝗲𝗿 with 𝘇𝗲𝗿𝗼 𝗽𝗼𝗹𝗶𝗰𝘆 𝘀𝘁𝗮𝗹𝗲𝗻𝗲𝘀𝘀. We observed that synchronous GRPO/OPD leaves learner GPUs idle for long stretches (𝟰𝟵.𝟱% of step time when training on SWE-smith). Asynchronous training avoids this, but it learns from 𝘀𝘁𝗮𝗹𝗲 𝗽𝗼𝗹𝗶𝗰𝗶𝗲𝘀 and performs worse. Our goal was to keep GPUs maximally utilized while still doing strictly on-policy updates. ThunderSyncRL streams gradients as soon as their inputs are ready, then takes one synchronous optimizer step. We prove the GRPO and OPD updates are identical to batch-synchronous training. On SWE-bench Verified and Terminal Bench 4.0, it reaches Sync's pass@3 up to 𝟭.𝟵× 𝗳𝗮𝘀𝘁𝗲𝗿 and beats Async by up to 𝟮.𝟰𝟳 𝗽𝗼𝗶𝗻𝘁𝘀 of pass@3 at the same GPU budget. Grateful to my co-lead @hangoo_kang and to the amazing @Azaliamirh, who made this work possible. And a big thank you to @TarunSures41845, Youngeun Kim, @sayshrey, and Seong Jae Hwang for building it with us. (1/7) 🧵 📄 Paper: arxiv.org/abs/2610.05935 👾 Code: github.com/seilk/thundersync 🌐 Blog: seilk.github.io/thundersync
7
48
4
425
22,458
very excited for this one 👀
defining the science of data. together. 10.14.26
13
594
Siddharth Joshi retweeted
The @datologyai team tested their Curation Studio at scale by generating large amounts of synthetic data on Modal, scaling to hundreds of B300s on demand.
Today, we’re excited to launch the Datology Curation Studio, customizable frontier data curation for every team training its own models. Data quality is the ultimate compute multiplier, and Curation Studio makes it easy for everyone.
2
9
1
57
5,347
ECHO but for user tokens??? :)
Now in beta: composer predictions in Codex for Pro users. Codex can now suggest your next message based on your conversation and how you talk to it. One of the most loved new features we've ever tested internally.
7
3
46
7,890