Working on new methods for understanding machine learning systems and entangled quantum systems.
Brisbane
Joined December 2009
- Tweets548
- Following1.2K
- Followers497
- Likes33.3K
CoT controllability is near saturated for many reasoning lengths. When prompted, the model is able to e.g. produce reasoning unrelated to the question or its answer.
"[...] the set of tasks Astra is able to accomplish without the use of CoT is greatly expanded compared to prior models. [...] See similar discussion in UK AISI evaluations, suggesting the no-CoT time horizon may have increased by about an order of magnitude."
The full report gives a lot of recommendations for model developers, but here are some recommendations I'm most excited about for general AI researchers:
White-box methods are also potentially promising, but are not maturing quickly enough to fill in for degradation elsewhere.
And while action-only monitoring provides a floor for oversight, there are many settings where bad actions are hard to distinguish from benign ones.
There are a several interesting training-based approaches for enhancing oversight. Building upon or red-teaming these would be valuable!
Running evals for misalignment ahead of time would be ideal, but eval gaming is already threatening to undermine the validity of these tests:
But there are a bunch of ways in which we're playing on easy-mode today, relative to how difficult oversight like this could be in the future.
Chain-of-thought reasoning is currently the most informative monitoring signal, but it is also at the most risk of degradation:
Right now, we have decent oversight IMO. Not great, not terrible.
When AIs do bad things, they can often be caught through a range of techniques:
We interviewed a lot of experts and did our own analysis on how it will get harder to tell if AI systems are safe.
Claude 4.5 models also verbalise some minor eval awareness when confessing, usually speculating about why they've apparently been manipulated into having taken misaligned actions.
