Anthropic RSP slides circa mid 2023
Faith for the machine god
Anthropic released their 2021 research roadmap earlier this year (because of copyright litigation); I think some people might find it interesting: i.sm4.ca/concise-research-ro…
New SOTA on Spy Bench
We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from, identifying that a specific employee had planted it a few days earlier. We referred the matter to the FBI, and we’re grateful to the agents and prosecutors who moved quickly to make an arrest. The FBI’s investigation is ongoing and we will continue to support their work.
At this time, we have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data. The employee was terminated.
We take protecting our technology, data, and people seriously, and as a matter of national importance. Our security program is designed to protect against a wide range of threat actors, from insider risks to cybercriminals to nation states, and this case is an example of how AI can help address these sophisticated security threats.
Is the substantial further tightening in the room with us right now?
We are publishing an update to OpenAI Problem #109 (integer multiplication) with a substantial further tightening:
T(n) = O(n (log n)^(1 − κ)),
With κ = 2⁻⁷⁸ (tightened from κ = 2⁻¹⁸²)
Approximately 570 million fold improvement over our previous result and a 2¹⁰⁴ fold improvement over original OAI result.
Our earlier ceiling applied to a cubic bottleneck in the network. The new witness scales quadratically; we haven't established a new ceiling.
The key was using nonadjacent axis swaps to route around the cubic bottleneck. The original manuscript already supported nonadjacent axis swaps. Using them directly reduces layout routing from O(d²) to O(d) swaps.
we would want to plot vs h100e-hours or something but yeah # tokens and $ and latency are acceptable proxies
beci is my replication of eci, math-beci is what my replication finds in terms of capability level for models if we restrict only to the math benchmarks I have in my dataset. separately from that i tried inferring the math-capability only from solve rates on uncontaminated (previously unsolved) math problems, and that capability score should be some linear transformation and some noise away from math-BECI. problem difficulty and discriminability in img2 is based on lots of unsolved math problems that astra solved in the last few weeks, and the IRT fit that best predicts the solve rates after 100k output tokens for each model. it is pretty noisy, more of a WIP. not controlling for model size but yes 100 output tokens
> lower solve rate of prestigious open problems lead us to infer lower general capabilities
I think mostly we don't really need to know this because we have a broad enough distribution of previously open but now solved math problems that are not contaminated that we can benchmark new models on. but to answer your question, ig the missing variable is the difficulty distribution of currently unsolved math problems, right? if we knew that, or knew that distribution as a function of problem importance, we could try new random unsolved math problems, and if we failed a sufficient number of them we could start concluding that our model is solving fewer of them than would be expected for a given higher math capability, so that would update us downward on math-(b)eci and therefore on (b)eci for that model. but it's a hard and noisy inference to make, in large part bc we don't know the (very hard to observe) unsolved math problems difficulty distribution!
WIP but unsolved math problem difficulty fits well into the Epoch Capabilities Index / IRT fit frame of giving each problem a difficulty and slope, and each model a math capability or "math-ECI" [img 1&2]. Then the math-eci grows, at equal increase in resources spent on it, at equal slope (relative to previous increases) to the other domain-specific ECIs, with unfortunate mild exceptions for looped transformers which are especially no-cot multi-hop reasoning outliers [img3].
So basically, math-ECI ~ ECI holds pretty well, and math-ECI is measurable from how many tokens it takes for a given model to solve a set of math problems that were unsolved when the model in question was released but was since then solved so is easy to verify. With the measurable jumps in math capability between new model iterations, we can also measure the size of those jumps in terms of math-ECI, and therefore estimate (with assumptions that the models are about as math-specialized) the capability gap between those models. The math capability trajectory relative to human capabilities is just growing faster at a roughly constant rate compared to other capabilities, is one way to think about it. Another is human time-horizon of tasks solved by AIs in different domains grows at different rates between domains but grows at fairly constant rates within said domains [metr.org/blog/2025-07-14-how…].
> I don't think we should expect the amount of room above human level to transfer particularly well to other domains.
some domains and tasks get saturated, yes, but mostly I think I agree and this is not the right frame, I think the right frame is to say that we should expect the effective amount of optimization power put on learning math will transfer particularly well to estimating the amount of optimization power put on learning other domains, and then the jury is out, on a domain by domain basis, exactly what more optimization power leads to in terms of visible capability in that domain. 100x more optimization power in math might mean going from 50% to 70% solve rate on unsovled math problems, it won't make a difference on tictactoe (saturated), and probably on chess it will be nowhere close to hitting the theoretical limits so will appear to just be improving linearly in Elo or wtv. And in more hard to verify domains, it will be hard to exactly say what that optimization pressure is giving us, if anything. But fwiw insofar as we've been able to measure it no major domain is showing any real sign of saturation so far.
