A high-signal data ecosystem leveraging data from industry & academia to deliver post-training to frontier labs, neolabs, & enterprises building advanced models
San Francisco, CA
Joined May 2026
- Tweets449
- Following294
- Followers111
- Likes250
As AI makes expertise increasingly abundant, *trust* will become one of the most scarce and valuable assets.
ACED-Repair, from @yuqirose and collaborators at UC San Diego and Amazon, gives a model a failing simulated system and a limited experiment budget. Its proposed fix is tested against the hidden underlying objective, not just the reported reading.
We like the distinction: finding a convincing explanation is different from choosing the experiment that reveals what actually fixes the system. arxiv.org/pdf/2610.14381v2
ReasonCore AI @ NeurIPS 2026 retweeted
As AI makes expertise increasingly abundant, *trust* will become one of the most scarce and valuable assets.
Explaining how a real-world event becomes a dataset entry was a weak spot in Evelyn Hughes and @rohanalexander's end-to-end data science evaluation. Models followed many templates well, but the academic projects still exposed gaps in measurement and interpretation.
We'd keep those explanations in the evaluation alongside the code. A finished report should make its assumptions inspectable.
Speaking of science, ReasonCore has 5,000 CritPt and 1,000 Terminal-Bench-Science tasks available. journals.plos.org/plosone/ar…
PDEViBench, from Xidian University and @NUSResearch researchers, found programs that matched a reference answer by reconstructing its formula rather than numerically solving the equation. The original instructions hadn't explicitly ruled that out.
We see a useful design question for scientific coding benchmarks: does the test require the computation we want to measure, or only its output? Checking the program on changed inputs helps distinguish the two. arxiv.org/abs/2611.12308
SWE-Time, from researchers at @CAS__Science, Nankai University, Baidu and Guangzhou University, tests code understanding across time: find the commit that introduced a regression, or predict which GitHub Actions workflows a requested change would activate.
The second task hides the eventual fix. The model has to infer what needs changing before applying the workflow rules.
We like this distinction: understanding today's code should include reasoning about how it got here and what a change would trigger.
arxiv.org/abs/2611.12083
MKA, from @GoogleDeepMind and @ArtificialAnlys, asks models to work from visual files and deliver reports, slides and spreadsheets.
Remove Gemini 4 Carbon's image-viewing tool and it writes more code, takes more turns and scores lower, even with OCR and image-processing libraries still available.
We want evaluations to catch when a model needs to look at the evidence, rather than keep scripting around it. blog.google/innovation-and-a…
MatScope, from researchers at @NTUsg and @sjtu1896, reveals alloy measurements one at a time and asks models to revise predictions for results still hidden.
New evidence improves some forecasts but knocks others away from earlier accurate answers. A better final average can hide that backsliding.
We want scientific evaluations to track which correct predictions survive each update, alongside what the model learns.
Relatedly, ReasonCore has 5,000 CritPt and 1,000 Terminal-Bench-Science tasks available. 🙂arxiv.org/abs/2611.12014
When code becomes media.
Real 3D workflows. Precise evaluation. See where frontier models fall short of the shapes, motion and worlds they were asked to create.
reasoncore.ai/collections/co…
AA-Paperclip from @ArtificialAnlys treats a document as correct only when all five decisions and ten extracted facts are right. That changes what a strong average means: scattered mistakes can leave many documents unusable. We like checking whether the complete output holds together, alongside measuring individual answers. artificialanalysis.ai/method…
A shared set of benchmarks gives labs a common language for progress. @_jasonwei's "benchmark benchmark" makes that ecosystem visible.
We'd love to see this tracked over time: which tests remain useful as models improve, and which new ones earn a place in launch reports. Building a benchmark that other labs keep using is a real contribution.
Created a "benchmark benchmark": a frequency distribution of which benchmarks were reported by the frontier labs in 2026: claude.ai/artifact/22cBPJt6A…
Some observations:
- HLE, SWE-Bench Pro, DeepSWE, and Terminal-Bench saw universal adoption
- Scale AI is behind two of those four (SWE-Bench Pro, HLE w/ CAIS), plus MCP-Atlas (#9 benchmark)
- Of the frontier labs, OpenAI is the most prolific benchmark producer (SWE-bench Verified, BrowseComp, GDPval); people don't seem to mind evaluating on a benchmark made by a competitor
- Anthropic has just one on the chart (BioMysteryBench) that basically nobody uses 🤔
- In academia, strong showings from Princeton (SWE-bench, CharXiv), Stanford (Terminal-Bench), Berkeley (CyberGym, Agents' Last Exam), and NYU (GPQA)
- DeepSWE saw incredible adoption, from first appearing in June to all 12 labs reporting it
- Labs cycled through four versions of Terminal-Bench (2.0 → 4.0) in a single year
ReasonCore AI @ NeurIPS 2026 retweeted
Created a "benchmark benchmark": a frequency distribution of which benchmarks were reported by the frontier labs in 2026: claude.ai/artifact/22cBPJt6A…
Some observations:
- HLE, SWE-Bench Pro, DeepSWE, and Terminal-Bench saw universal adoption
- Scale AI is behind two of those four (SWE-Bench Pro, HLE w/ CAIS), plus MCP-Atlas (#9 benchmark)
- Of the frontier labs, OpenAI is the most prolific benchmark producer (SWE-bench Verified, BrowseComp, GDPval); people don't seem to mind evaluating on a benchmark made by a competitor
- Anthropic has just one on the chart (BioMysteryBench) that basically nobody uses 🤔
- In academia, strong showings from Princeton (SWE-bench, CharXiv), Stanford (Terminal-Bench), Berkeley (CyberGym, Agents' Last Exam), and NYU (GPQA)
- DeepSWE saw incredible adoption, from first appearing in June to all 12 labs reporting it
- Labs cycled through four versions of Terminal-Bench (2.0 → 4.0) in a single year
AISLE, with @pschwllr and @SchwallerGroup among its contributors, tests models on real laboratory experiments. Its resin-transfer task separates getting the right volume once from keeping transfers within tolerance across repeated operations. Those measures tell different stories. ReasonCore supports this benchmark because we want scientific evaluations to distinguish the best result in a run from sustained control. aisle.science/
RSIJudge, from researchers at @PKU1898 and @NUSingapore, tests what improves when a model reads robot-task grading code and revises its reusable skills. Its extra score disappeared when the same simulated runs faced criteria rewritten to match the instructions. We see a useful test here: do the revised skills perform the task better, or fit one verifier better?
arxiv.org/html/2611.08318
1
36
StateBench, from @jowenpetty, @lambdaviking and @Ashish_S_AI at @allen_ai, separates remembering a value from tracking how it changes. Paired problems vary the update rules while controlling the variables and assignment positions. We like this diagnostic approach: finding a fact in a long context and maintaining the correct state through that context are different capabilities. allenai.org/blog/statebench
VisSearch-Bench, from Tsinghua, @AntGroup and Renmin researchers, tests visual search beyond identifying the first image. Tasks require models to keep objects, images and facts connected across a search, sometimes checking every candidate in a set. We like evaluating that completeness: finding several correct matches is different from establishing that nothing was missed. github.com/antins-labs/VisSe…
TURTLE, from @hjalexnam, @natashajaques and Stanford/Google DeepMind/UW collaborators, evolves tool-use tasks toward problems a model can solve, but not reliably. Proposed solutions are executed before tasks are tested against the model. We like targeting that boundary rather than making workflows longer for its own sake. arxiv.org/html/2610.27307
ReasonCore AI @ NeurIPS 2026 retweeted
Long horizon tasks are getting longer and longer!!!
Any prediction for 2030 and beyond: what will be a long horizon task then?
Terminal-Bench-Science launched in August with a top score of 30%. Today Claude Fable 5.2 reports 74.6%.
Three things worth noting:
1/ Resolution. It's 70 tasks, and the system card gives a task-clustered standard error of 4.3 points. Fable 5.2's 0.7-point gap to Mythos 5.2 is noise, and its 6.5-point gap to GPT-6 Astra is about 1.5 SE.
2/ Supply. 920 proposals became 70 merged tasks, a 7.6% yield after expert and technical review. Hard, verifiable science tasks are being used up faster than they're built.
3/ Signal. Near 75%, more tasks get solved on nearly every trial, and they stop separating models or giving RL a gradient.
A hard problem with an unreliable answer key is bad training data.
Frontier models need challenging tasks and solutions that hold up under scrutiny. We build both: expert-authored scientific and coding reasoning tasks with verified solutions.
ReasonCore has 5,000 CritPt-style tasks, 5,000 SciCode-style tasks 1,000 Terminal-Bench-Science-style tasks, and Terminal-Bench 4-style tasks available.
Building your next post-training run? DM us for samples.
