AI/ML Engineer @ Dana-Farber Cancer Institute | Stanford alum
Joined May 2024
- Tweets37
- Following54
- Followers86
- Likes39
Pinned Tweet
Very excited to share the release of BioClinical ModernBERT!
Highlights:
- biggest and most diverse biomedical and clinical dataset for an encoder
- 8192 context
- fastest throughput with a variety of inputs
- sota results across several tasks
- base and large sizes
(1/8)
Thomas Sounack retweeted
We are setting out to develop some new recommendations (TRIPOD-CODE) to provide guidance on reporting the availability & structure of code for predictive AI healthcare tools
Watch this space & read the protocol here
link.springer.com/article/10…
#transparency #code #reproducibility
Our Medslice paper was just accepted at @JAMIAOpen!
We provide a pipeline to extract clinically relevant sections of medical notes (HPI, Interval Hx, Assessment and Plan) using fine-tuned language models.
These small open-sourced LLMs can run on laptops (and even good smartphones), meaning that any institution can run them securely behind their firewall. This is significant since HIPAA-compliant LLM access is still rare for medical institutions.
Want to continue training an encoder on your own data, but not sure where to start?
Our step-by-step guide for reproducing the BioClinical ModernBERT training was just released!
1/5
If you are working with a lot of biomedical and/or clinical text, consider continuing MLM training of BioClinical ModernBERT on your own data!
The resulting encoder will be much easier to fine-tune on your various downstream tasks (embedding model for RAG, classifier...)
4/5
If you would like to see more details about a certain aspect of the guide, please don't hesitate to reach out! Your contributions are welcome and will be acknowledged.
Link to our HF collection: huggingface.co/collections/t…
Link to our paper: arxiv.org/abs/2506.10896
5/5
Exciting work from @neumll !
🧬🔬⚕️ Building on the popularity of our PubMedBERT Embeddings model, we're excited to release a long context medical embeddings model!
It's built on the great work below from @tsounack
Model: huggingface.co/thomas-sounac…
Paper: arxiv.org/abs/2506.10896
huggingface.co/NeuML/bioclin…
Thomas Sounack retweeted
codepen.io/jmikedupont2/pen/… colab.research.google.com/dr… next demo visualizing BioClinical-ModernBERT-base embeddings on a sphere
Thomas Sounack retweeted
BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP
→ Built on ModernBERT with 8K context, RoPE, and fast unpadded inference
Trained via two-phase continued pretraining:
- Phase 1: 160.5B tokens (PubMed + PMC + 20 diverse clinical datasets)
- Phase 2: 2.8B clinical-only tokens for specialization
- Retains biomedical knowledge, improves clinical performance
Architecture & Training
- Alternating local/global attention
- 50K+ vocab for medical terms
- MLM rate: 30% → 15% w/ WSD scheduler
- Base (150M) and Large (396M) models trained on H100s
Performance (Base)
- ChemProt: 89.9 F1 (↑ vs BioBERT 89.5)
- Phenotype: 58.1 F1 (↑ vs Clinical BERT 25.8)
- DEID NER: 82.7 F1 (↑ vs BioMed-RoBERTa 81.1)
- SOTA on 4/5 tasks
- 71–75k tokens/sec at all sequence lengths
- Outperforms baseline encoders (BioBERT, Clinical BERT, Clinical-BigBird, etc.)
- Outperforms BigBird, Longformer, Clinical-ModernBERT in throughput
𝘚𝘖𝘔𝘌𝘛𝘐𝘔𝘌𝘚 𝘠𝘖𝘜 𝘑𝘜𝘚𝘛 𝘕𝘌𝘌𝘋 𝘈 𝘉𝘌𝘙𝘛
PAPER: arxiv.org/abs/2506.10896
Thomas Sounack retweeted
we are so back
"Mitochondria is the powerhouse of the [MASK]."
Thomas Sounack retweeted
BioClinical ModernBERT is out!
Built on the largest, most diverse biomedical/clinical dataset to date
‼️Delivers SOTA across the board
Thrilled to be part of this effort led by @tsounack
Thomas Sounack retweeted
Your daily reminder that fine tuning is just continued pretraining.
Super cool results from @antoine_chaffin who is putting this knowledge into practice to improve medical AI:
Thomas Sounack retweeted
You can just continue pre-train things ✨
Happy to announce the release of BioClinical ModernBERT, a ModernBERT model whose pre-training has been continued on medical data
The result: SOTA performance on various medical tasks with long context support and ModernBERT efficiency
