Ronnie Cutler

The "Single-cell foundation models don't work for perturbation prediction" narrative was a measurement problem. Our new paper is out in @NatureBiotech : "Deep Learning Perturbation Models Can Outperform Baselines on Calibrated Metrics" Last year, several papers concluded that deep learning perturbation models couldn't beat simple baselines, a finding that cast doubt on the entire Virtual Cell direction. We asked a different question: can the metrics used in those comparisons reliably tell a good prediction from a bad one? Often, they can't. Common metrics like MSE and control-referenced Pearson Δ frequently rank uninformative baselines above positive controls. With calibrated alternatives (eg, WMSE, weighted R²Δ, NIR), deep learning models do outperform mean, control, and linear baselines. The data matters too. Norman19 is approximately 96% additive, and models see less than 1% of possible gene combinations during training. The ceiling for beating an additive baseline is nearly zero, and the training signal for non-additive interactions is almost absent. Some dataset rankings aren't model rankings, but just noise! To address this, we developed the Dynamic Range Fraction (DRF): a measure of metric calibration. Before trusting a model ranking, verify that your benchmark can detect a useful prediction at all. The models aren't shiny toys. The field just needed better rulers. Paper: nature.com/articles/s41587-0… Code: github.com/shiftbioscience/P… Huge congratulations to the first authors @Henrymiller2012 and Gabriel Mejia, and the team of Shift Bio!