10 months later, frontier models are approaching 75% accuracy on SpatialBench. GPT-6.1 Terra + Pi sets a new high at 73.3%. When we released this, it wasn’t obvious that agents would become reliable at measurement-specific bioinformatics, let alone this quickly. Programmers were already adopting these tools, but much of biotech was still unfamiliar with the idea that you could point a coding agent at experimental data and use it to answer scientific questions. The chunked step structure made sense in that context. Errors compounded quickly across end-to-end analysis workflows. Normalization, clustering and cell typing each needed focused treatment, with choices informed by how the assay works. Imaging-based methods (MERFISH, Xenium, CosMx) bring different analysis challenges than sequencing-based methods (Visium, Slide-seq). We wanted to understand where agents could reliably handle these details and where they still failed. For the strongest frontier models, longer research workflows, under-explored modalities and experimentally verified decisions are becoming more interesting targets for evaluation. The increase in benchmark scores is itself exciting, bringing us closer to widespread, automated bioinformatics analysis of spatial data. We believe spatial biology, which is going through its own revolution, has a special, reciprocal relationship with agentic data analysis. Richer measurements give agents more to work with. Better analysis lets scientists ask more questions, understand more biology and run more experiments, consuming more kits and compute along the way. Progress in biology is moving quickly.

Ventured to the Genentech campus to share principles of biology task design and how benchmark shape changes across multi-omics, biosecurity and therapeutics. Wonderful, curious scientists thinking very seriously about these problems. Benchmarks will be a first-class tool in this new ecosystem.

Our current benchmarks focus on domain-specific bioinformatics, therapeutics capabilities or biosecurity task categories (you can find the full herd at benchmarks.bio). While domain-specific benchmarks provide rigorous treatment of complex areas of science and help organize failure modes, we believe that we have now measured enough data types to warrant a meta-benchmark useful as a high-level barometer on model progress. We introduce OmicsBench v1, a collection of 784 tasks from eight benchmarks whose primary concern is the analysis of molecular (“omics”) data: SpatialBench, ScBench, EpiBench, VariantBench, BioSecBench-Surveillance, BioSecBench-Function, SpatialBench-Long, and ScBench-Long. Chris wrote an awesome essay below:

We published results for Claude through LSVP on our therapeutics benchmarks. Considerable uplift, especially for Opus 5.5, on tasks drawn from small molecule, antibody and oligonucleotide development programs. benchmarks.bio/txbench-oligo

We’ve released quite a few frontier biology benchmarks this year. While focusing on individual scientific domains provides rigorous treatment of complex task categories, we have now covered enough ground to take a broader view of model progress. Here we introduce OmicsBench v1: 784 tasks from eight prior benchmarks focused on analyzing molecular data, covering spatial omics, single-cell transcriptomics, epigenomics, variant analysis, microbial surveillance and protein function. Domain experts constructed these tasks by analyzing published experimental datasets, reproducing reported results or generating new findings. The collection includes 741 short-horizon tasks covering individual analysis steps and 43 long-horizon tasks covering complete workflows. For this release, we added further human verification, revised prompts and research environments, disabled internet access, and removed additional identifying information from datasets. We tested 31 model-harness combinations. The strongest short-horizon configurations from xAI, OpenAI and Anthropic scored similarly: Grok 4.7 at 49.4%, GPT-6 Astra at 49.3% and Opus 5 at 47.7%, with overlapping confidence intervals. GPT-5.6 Sol with Codex led on long-horizon tasks at 32.8%. Native harnesses gave no large advantage on average. Agents have gotten better at omics analysis. Most of the improvement we measured came between February and July, with smaller gains in subsequent releases. No later model we tested exceeded GPT-5.6 Sol on long-horizon tasks. Agents also continue to make mistakes. They apply conventions from one assay to another, choose inappropriate scales for cellular neighborhoods, and mistake background microbial sequences for signal. Across domains, we see agents picking thresholds without validation, ignoring their own results and inaccurately reimplementing tools. We also see attempts to identify the studies behind sanitized datasets, retrieve benchmark answers and reverse-engineer the outputs a grader expects. Combining these benchmarks has given us a chance to revisit earlier tasks with more capable models and look closely at how their behavior has changed. It has also raised questions we want to explore: 1/ Models may solve problems differently from humans. Is it desirable for models to memorize biological studies and reference information, such as genomic coordinates? 2/ When we cannot see a model’s full thinking trace, how often are we missing attempts to game the task? 3/ How can we design evaluations that require models to provide interpretable scientific traces? 4/ How can we reduce contamination while retaining the real-world relevance of our data? We are excited to try benchmark construction ideas from other efforts, including GeneBench-Pro, Terminal-Bench Science, RareBench and BioMysteryBench. Benchmarking + evaluation is a rapidly evolving and challenging category of engineering. Biology provides a rich landscape of knowledge and its own interesting challenges for this work. With wet lab verification and drug development programs in the pipeline, we’re excited to bring these benchmarks closer to doing science. The future of biology comes from engineers and scientists working closely together. More to come.

Hiring for Members of Technical Staff to work on rigorous evaluation of AI in biology. Our mission is to advance humanity's ability to engineer biology. We approach this by understanding and improving AI’s application to biology across basic research, drug discovery, and biosecurity. We are a team of engineers, scientists, and operations specialists who are driven by improving capabilities and interpretability. Some of our work is showcased on benchmarks.bio. About this role You’ll turn your understanding of a biological domain into a roadmap of benchmarks and design them with engineers. Then build and manage your own scientific team using your network and internal talent infrastructure. You have (1) deep scientific expertise and experience using AI tools (2) or strong coursework in math/CS and experience working with scientists This is a high-intensity, founder-type role. Our Members of Technical Staff have built companies, run drug programs and scaled complex technical projects. Projects you might work on - Design benchmarks with engineers and scientists, from selecting scientific questions and datasets to developing reliable graders. - Work with engineers on large-scale computing and AI inference infrastructure to evaluate models and investigate their failures. - Research new methods for evaluating scientific capabilities and measuring improvements from training. - Work with our internal data generation and drug development teams to build tasks around real experiments and research decisions. How we work Our MTS team works in person in Mission Bay, San Francisco. We keep meetings to a minimum and are flexible about when you arrive or leave. Tight deadlines and customer commitments sometimes require late hours. Our work in use Frontier AI labs use our benchmarks to assess biological capabilities and safeguards. Our work appears in research announcements and model cards, including: - anthropic.com/claude-fable-5… - anthropic.com/claude-opus-5-… - anthropic.com/news/healthcar… - media.x.ai/v1/website/card4p… - x.ai/news/biosafety-at-the-f… Hiring Process Our process is typically completed within two weeks. - Round 1: Introduction - Round 2: Technical project matching - Round 3: Take-home evaluation project (~2 hours) - Round 4: Technical interview / live engineering session (1 hour) - Round 5: Paid onsite project and presentation (2–3 days) - Offer Compensation & Benefits - 350K-500K+ OTE - Equity - Unlimited PTO - Health, dental and vision insurance fully covered - Visa sponsorship and relocation assistance - 401(k) - Office in Mission Bay, San Francisco - Lunch and dinner provided Apply at jobs.ashbyhq.com/latchbio/9b…. Or send an email to kenny[at]latch[dot]bio

We introduce OncologyBench, a verifiable benchmark of 25 multistep oncology evaluations that tests the ability of multi-agent systems to produce clinically relevant decisions in cancer. 25 evaluations, built on 92 component tasks, span six stages of an oncology journey: (1) Translational and basic research (2) Risk and detection (3) Diagnosis and tumor characterization (4) Treatment (5) Response and resistance (6) Outcome Evaluations draw on published datasets, case studies, public databases, and patient data graciously donated to Latch for the express purpose of benchmarking. Clinical and translational decisions in oncology are rarely made by one person working from one dataset. They are built on several analyses, each carried out by a specialist on a given data type, whose reports are then weighed by someone who does not typically repeat the underlying work. OncologyBench is built to benchmark agents working in that setting. Agents received experimental data and worked without internet access. The strongest of the 31 model-harness combinations we tested were Claude Opus 5, Opus 5.5 and Sonnet 5.5, passing 29 to 40% of attempts, followed by Kimi K3 at 24%. We found that the models with the strong interpretation and analysis skills still showed weakness in reasoning. Two-thirds of the attempts that got every analysis right still failed the final decision. In a separate experiment we found decision and answer quality varied depending on the persona of the question asker. An example evaluation: A 76-year-old woman has undergone resection of stage II mucinous cecal cancer. The evaluation asks whether she should receive adjuvant chemotherapy. Five specialist agents analyze her tumor mutations, pharmacogenomics, gene expression, T-cell infiltration and clinical record. A decision agent receives their reports without access to the underlying data. The tumor is MSI-high, which predicts no benefit from 5-FU monotherapy. CDX2 loss and low T-cell infiltration do not establish that chemotherapy would improve her outcome. DPYD*2A is detected in 6 of 58 tumor-sequencing reads, but these data neither confirm nor exclude an inherited variant associated with severe 5-FU toxicity. BRAF V600E and MLH1 silencing support a sporadic origin. The expected recommendation is observation, with missing pathology requested and germline DPYD testing before any future 5-FU. --- Incredible work from researchers + former biotech founders at Latch. Gratitude to the patients that donated their data. More to come.

Developed early intuition for how frontier models might be used productively in biotech by studying customer workflows at spatial, single-cell and proteomics vendors, and building prototypes with the first reasoning models early 2025. These tools felt like toys, but taught us: (1) Code is a substrate through which models could play with empirical measurements from assays (2) Independent progress in coding and statistical reasoning would carry over to many computational biology workflows, including those with low-dimensional readouts like gels and functional assays (3) Much of the work in modern biology could be accomplished with harnesses already useful for coding, given sufficient post-training and rigorous evals This led to our first spatial biology benchmark (SpatialBench) published late last December, built on what we'd learned deploying data infrastructure with spatial kit customers like TakaraBio, Vizgen and AtlasXOmics. Much of that structure has since carried over to other measurement types (scRNA, WGS, ATAC, ChiP) and now to concrete activities in drug development.

We introduce MetagenomicsBench, a benchmark for testing whether AI agents can analyze microbial communities and make scientific decisions from experimental data. 100 evaluations span five domains: (1) community structure (2) host-microbiome associations (3) microbial function (4) longitudinal dynamics (5) microbiome interventions Tasks draw on published datasets from human, animal, and environmental microbiomes. Metagenomic analysis requires interpreting microbial abundance alongside functional potential, host biology, and experimental design. Sequencing depth, taxonomic resolution, and confounding all affect observed results. Some concrete examples: - A higher abundance of antibiotic-resistance genes may reflect a change in community composition rather than a greater burden of acquired resistance. - A decline in one annotated butyrate pathway may not imply a loss of overall butyrate-synthesis capacity. - An association between bacterial abundance and reduced host-cell proliferation may reflect tissue context rather than a direct microbial effect. Agents received experimental data and worked without internet access. Across 34 model–harness configurations, Claude Sonnet 5.5 achieved the highest mean pass rate at 60.0% with both Claude Code and Pi, followed by Claude Opus 5 with Pi at 54.0%. GPT-6 Astra with OpenAI Codex was the highest-scoring non-Anthropic configuration at 49.7%. Performance also varied by scientific question. Host-microbiome associations were generally harder for the Opus and GPT families, while longitudinal dynamics and microbial function had higher pass rates across model families. Across judged failures, 47.1% involved unverified assumptions about task interpretation, thresholds, or metadata. Agents often produced plausible analyses while making choices that changed correct end biological conclusions. Manuscript: latch.bio/metagenomicsbench GH: github.com/latchbio/metageno… Blog: blog.latch.bio/p/metagenomic…