Building open models for biology @ OpenAthena. Previously @OpenVax @ImprintLabs @MountSinai @DEShawResearch
Somerville, MA
Joined February 2008
- Tweets318
- Following501
- Followers750
- Likes807
Gearing up for post-training our oss 535b model and looking for more ideas like Minos for bio evals/envs. If you have cool screening data (mol interactions/function, cell perturbations, ...) and want to work with a nonprofit to make the model good at real tasks, pls get in touch
Tim O'Donnell retweeted
Marin's 535b is now >80% through pretraining! Here’s loss vs. our scaling-ladder forecasts. The sharp drop comes with the 4K→16K context switch, so we're kind of cheating the forecast. (This was planned.) We'll re-eval at 4k too. forge.coreweave.com/wandb/ma…
ALT Forecast versus observed loss: Hero’s evaluation loss falls below both scaling-ladder forecasts, reaching 1.528 at step 314,999. A sharp drop accompanies the switch from 4K to 16K context near step 258,000, making subsequent values not directly comparable with the 4K forecasts. The cooldown mixture begins at step 312,192.
Tim O'Donnell retweeted
MHCflurry 2.3 is out
Lots of internal improvements to make both training and prediction faster on multi-GPU
Also trained a slightly better model (first new weights in a long time)
Thanks to @nvidia / @AllThingsApx for the Brev credits
Try it out and tell me what I broke
Want to watch a 535B parameter (23B active) LLM get trained live? Follow along here wandb.ai/marin-community/mar… github.com/marin-community/m…
Tim O'Donnell retweeted
Excited to share MarinDNA, a 1B gLM that rivals Evo 2 40B on variant effect prediction while being 2,330x faster.
With @eczech0, we built around a standard Transformer so we could reuse LLM infra and methods while focusing on data curation and scaling.
openathena.ai/blog/marin-dna… 🧵
Tim O'Donnell retweeted
Coding agents are helping scientists spend more time advancing research, taking on everything from routine maintenance and targeted optimization to complete redesigns and new systems.
While agents can reliably execute on ambitious projects, researchers must still define the scientific questions, verify results, and take a stance on long-term ownership.
BoltzGen preprint was updated to include proteome-wide screening for off target interactions for one design (against IDI2) using CDI HuProt. The results were crazy clean - no off target binding (>21,000 targets)! With n=1 it's just a demo but I hope this becomes more common
Preprint: biorxiv.org/content/10.1101/…
Tim O'Donnell retweeted
🍹Weekend project: I (with help from claude - budget $40), was able to get OpenFold3 weights running inside AlphaFold3 codebase (jax). Works for proteins, ligands, rna/dna (1/3)!
AF3 source code is now open source (aka Apache 2.0)! 😎
Though not the weights... 🙃
github.com/google-deepmind/a…
Tim O'Donnell retweeted
We're training the largest open-source, open-development, AND open-weight base LLMs of any (actual) non-profit. The latest on our journey to the frontier is a 129B-A16B 1T tokens (1e23 FLOPs) MoE.
We've improved our training efficiency (i.e. loss per FLOP) by >5x in just the past couple months.
Follow along the day to day work in Discord, watch us make mistakes and discuss tradeoffs on GitHub, see our loss curves on Weights & Biases, and watch us consistently hit our preregistered loss targets within 1% @ marin.community
Building momentum at Marin! Upgrading from Dense -> 129B parameter MoEs -> architecture improvements -> optimizer improvements gives our pretraining recipe an estimated 6x cumulative learning speedup, accounting for MFU. Includes community contributions. openathena.ai/blog/pretraini…
I wanted to run some old protein structure analysis software (ConFind) at scale... written in C++, custom build instructions, slow. A slog of a task a year ago. But now easy to just reimplement it github.com/timodonnell/pycon… . Yes to agents rescuing good ideas from bit rot!
Tim O'Donnell retweeted
When you come into biology from CS you scoff at the obscurity of everyone's research, look at all these underpaid postdocs churning out papers on ridiculously niche topics. They don't have the right (startup-adjacent) cultural traits to tackle ambitious goals.
Then...
We're hosting a co-working session for anyone working in Bio+ML in Kendall sq next week. Industry / startups / academia / nonprofits welcome. I will likely do a series of these - if you would like invites to future ones DM me your email!
Tim O'Donnell retweeted
To train better open models, we need predictable scaling.
Delphi is Marin’s first step: we pretrained many small models with one recipe, then extrapolated 300× to predict a 25B-param / 600B-token run with just 0.2% error.
Getting there took some work 🧵
Anyone willing to donate GPU time to re-train MHCflurry?
MHCflurry, despite its age, is still somehow the most reliable thing I have for MHC-I presentation prediction.
Anyone want to sponsor the GPU time to train a new major release on updated data?
(@modal? This model gets used quite a bit in vaccine design & cancer immunology)
Tim O'Donnell retweeted
MHCflurry 2.2.0rc2 is on PyPI:
pypi.org/project/mhcflurry/2…
Try it out and let us know if you spot any problems in our transition from TensorFlow to PyTorch
Thanks to @iskander 's robot army for rewriting mhcflurry in torch! Available now as "pip install mhcflurry==2.2.0rc1" github.com/openvax/mhcflurry…
