making data loaders go brr (deterministically) | mts @datologyai | prev. Ph.D. @SystemsGroupETH @anaklimovic
Zurich, Switzerland
Joined October 2012
- Tweets891
- Following1.2K
- Followers515
- Likes3.2K
Pinned Tweet
At @datologyai we run thousands of data experiments. Two things keep biting us:
- Same run on 16 vs 8 GPUs → different evals
- Every pipeline change → re-materialize the dataset
Introducing Zephon, the data loader we built to process data on the fly, deterministically. 🧵
Zephon launched two weeks ago. v0.2 is out today.
→ Use data stored in your @lancedb tables
→ Run on @modal & setups w/o a shared FS
→ Mixture weight redistribution
→ More zero-copy ops + reliability fixes
w/ @josh_wills & the @datologyai team
uv pip install -U zephon
New to Zephon?
It's @datologyai's open-source, elastically deterministic data loader: process data on the fly, get the same training batches on 8 GPUs or 16.
Why we built it: datologyai.com/blog/zephon
Docs: docs.zephon.io
Code: github.com/datologyai/zephon
Maximilian Böther retweeted
Awesome to see Zephon, our recently open sourced data loader, integrated with @PyTorch TorchTitan!
Check out their blog, and congrats to @MaxiBoether, @josh_wills, and @SGui.
We're just getting started with Zephon!
Zephon’s TorchTitan integration preserves identical global batches when training resumes on a different GPU count.
In our latest blog’s 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run.
Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed copy of the corpus.
The data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming. The lane count must divide evenly across data-parallel replicas, and each optimizer step’s batch must split evenly across lanes. Training losses can still differ because floating-point reduction order depends on GPU count.
✍️ @datologyai @josh_wills @MaxiBoether, Ties Robroek (@datologyai)
Read more: pytorch.org/blog/bringing-el…
Very happy that the Zephon team and Pytorch/Torchtitan are working together to make data loading more deterministic.
Check out our example integration: github.com/datologyai/torcht…
Check out our blog post together with the folks at Meta 👇
Zephon’s TorchTitan integration preserves identical global batches when training resumes on a different GPU count.
In our latest blog’s 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run.
Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed copy of the corpus.
The data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming. The lane count must divide evenly across data-parallel replicas, and each optimizer step’s batch must split evenly across lanes. Training losses can still differ because floating-point reduction order depends on GPU count.
✍️ @datologyai @josh_wills @MaxiBoether, Ties Robroek (@datologyai)
Read more: pytorch.org/blog/bringing-el…
Maximilian Böther retweeted
MosaicML’s Streaming dataloader was one of the most underrated pieces of ML infrastructure ever built imo. It spoiled me. One of my goals when founding Datology was to take all the best things about Streaming and build its torchbearer. Because well-designed dataloaders enable entire domains of data experimentation like real-time curricula, arbitrary mixing schedules, and seamless transitions between pre- and post-training. And even better-designed dataloaders enable productionizing and scaling those experiments. I am so excited that we're finally releasing Zephon. It leveled up Datology and we think it'll level you up, too.
At @datologyai we run thousands of data experiments. Two things keep biting us:
- Same run on 16 vs 8 GPUs → different evals
- Every pipeline change → re-materialize the dataset
Introducing Zephon, the data loader we built to process data on the fly, deterministically. 🧵
A data loader is only as useful as the data it loads. This is why this week is so exciting: we're sharing our Curation Studio and how we use it to generate the best tokens (or raw strings to be tokenized and packed on-the-fly) possible. Check out our release announcement!
Maximilian Böther retweeted
Oh! This is cool! xitter.cf/i/trending/21078793786…
[9/] Tweets are way too short to cover the pitfalls around state management in data loading and offline materialization. Hence we wrote a whole blog post around it: datologyai.com/blog/zephon
We’re also excited to share our research paper: arxiv.org/abs/2610.03087
[10/] This is a collaboration with many amazing people at @datologyai: @josh_wills, Ties Robroek, @hurrycane, @arimorcos, and many more, but of course also my PhD supervisor @anaklimovic was a tremendous help over the course of the project.
Please reach out if you’re interested!
Maximilian Böther retweeted
Maybe a bit late, but I wanted to comment on this: I think the AI boom made people forget what a PhD was meant for. Now it's going back to its original purpose: not short-term profit, but long-term research that might not have a practical use for another 10 or 20 years -> Backprop existed decades before the deep learning boom.
A PhD isn't for everyone, and that's okay. Industry may suit you better if you want to build something useful now, or academia if you believe in a problem industry doesn't care about yet. The AI boom made a PhD look like preparation for an industry job, but that was never its main purpose.
Students sometimes ask me if it still makes sense, in this accelerating age, to pursue a PhD in AI.
Perhaps counterintuitively, I think it's a great time to do so.
I wrote up some thoughts on this here: web.mit.edu/phillipi/www/wri…
Maximilian Böther retweeted
Of course, love me some confirmation bias!
DCLM was for text, DCVLM is the same for vision: analyzing VLM data mix across scales. Filtering is no bueno, but the "type" mixing matters, with unfortunately (but imo expectedly) the best small-scale mix != the best mid-scale mix.
Finding 1: Data filtering barely helps for VLMs! 🤯
Tested 60+ filter configs (CLIP-score, DCLM-fasttext, Nemotron classifiers, UniFilter, perplexity). Best case: +0.8pp over no-filtering. Most filters do nothing or actively hurt!
@giffmana @tatsu_hashimoto you might like this!
DataComp-VLM aka DCVLM is finally out! It was such a pleasure working with this team of cracked ML data folks. Check out what changes in comparison to DCLM as we have surprising findings for our VLM data pool.
Models are what they eat, and it continues to show!
🚀New Paper
arxiv.org/abs/2606.28551
Everyone obsesses over VLM architectures & training recipes. But what about the data?
Presenting the latest work in the DataComp-series: a testbed for VLM data curation with 1,000+ controlled experiments and some surprising lessons 👀
🧵👇
