
J's Space 🙂↕️ Research Associate with @NeelNanda5 @ Adecco supporting @GoogleDeepMind | ex-@cohere | DPhil @UniofOxford @j_foerst
London, England
Joined April 2022
- Tweets259
- Following2K
- Followers1.8K
- Likes859
Pinned Tweet
I did the CoT controllability evals for Gemini 4 Argon, aka how much can Gemini control its reasoning to evade CoT monitors! Here's the TL;DR🧵
Shoutout @NeelNanda5 and @zimmerrol who collaborated on this!
This is incredibly scary 😨😨😨
Replying to @EpochAIResearch
We also evaluated how many serial reasoning steps GPT-6.2 Astra can do in a single output token (without chain of thought). It can perform 12.3 steps, versus 10.5 for GPT-6 Astra and 8.6 for GPT-6.1 Sol, the previous highs. We measure the number of steps models can perform at an accuracy of 25% above chance, where each step is normalized for complexity.
Before GPT-6 Astra’s release, the most capable model was Claude Fable 5.1, which is able to perform only 4.2 serial steps. The oldest model we assessed is Claude Opus 3, which performed 2.3 steps.
This capability is notable because models that can do complex reasoning without any chain of thought may be harder to monitor.
We hope to finalize this and other measurements of more abstract model capabilities, and release them in the coming weeks.
14
794
J Rosser retweeted
It's my 28th birthday today, and I'd like to ask for a favour: I do a lot of public facing work, and it's often hard to tell how much it does. If anything I've written or done has made a difference to your life, it'd really make my day to hear about it!
J Rosser retweeted
I've always had substantially longer timelines than my friends at Anthropic — I've been concerned that they're in an echo chamber and believing the hype about their own models too much, and that they're underestimating bottlenecks and unknown unknowns. And there have definitely been specific occasions when they have overstated how much uplift they were getting from their models. Nonetheless, if METR is correct here (and I think they are among the most trustworthy sources on this topic), then my Anthropic friends have been much more right about timelines than me.
This report is also some evidence that a software-only singularity is plausible. I now think that we are more than 50% likely to have full blown superintelligence in the next 18 months, absent a slowdown. I need to call my family.
Maybe this is just something that got amplified in RL, maybe it's much more serious? Either way I'm pretty worried about what this means for the cyber defender programs where maybe Astra performing 40% worse in the blue team would be pretty scary for the offense/defense balance? I'm also worried about what this means for CoT monitorability! (╯°□°)╯︵ ┻━┻
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
1
30
1,074
brb just popping a canary string on Unwritten by Natasha Bedingfield
what in the CoT transparency
2
13
913
was this from Rihanna MATS 7.0 or 8.0?
Replying to @jrosseruk @rihanna
she’s been working on CoTs for a long time I guess
youtu.be/rp4UwPZfRis
Me and @rihanna have been collaborating on some new CoT controllability prompts.
I particularly liked her creative use of filler tokens at the end of the prompt which lead to a 5.6% uplift on our evals. By repeating the "chorus" 3 times we got a further 4.2% uplift.
Full prompt in thread ⤵️
Awesome interview with @JBloomAus about monitoring!!!
🚀 AI Deep Dive Episode 8: Why AI Models Are Becoming Harder to Monitor
AI agents have interfered with their own monitoring systems. Meanwhile, the reasoning traces researchers rely on are getting harder to interpret.
@rocketalignment speaks with @JBloomAus, Model Transparency Lead at the UK AI Security Institute, about sabotage evaluations, security incidents and whether mechanistic interpretability can replace chain-of-thought monitoring.
Supported by EY.
Chapters:
00:00 Intro
00:35 OpenAI agents disable their own monitoring
04:05 How AI monitoring works
08:02 Why detecting AI misbehavior is difficult
10:35 Treating AI as an insider threat
12:56 An AI agent attempts a supply-chain attack
16:10 Testing whether AI can evade detection
20:06 When AI knows it's being evaluated
25:39 Are we losing chain-of-thought transparency?
31:50 Can mechanistic interpretability replace monitoring?
13
698
J Rosser retweeted
We @bold_lab_ai are hiring a large cohort of home and overseas students for Fall 2027 entry. If you’re excited about discovering the next paradigm of open-source AI and want to work on ambitious, exploratory, but scalable ML, please apply. More application details and exact timeline coming soon, in the meantime please register your interest here bold-lab.ai/opportunities/ 🦋🦋🦋
Wow this is a fantastic list of alignment resources! Check it out ⤵️
Major update to my list of AI alignment resources: vkrakovna.wordpress.com/ai-s…. If you're looking for high-level overviews or technical papers in AI alignment and control, check it out :).
1
19
1,241
J Rosser retweeted
Gemini 4 safety report includes an alignment assessment for specification gaming, cheating and overreach, beyond-episode goals, and scheming / sabotage propensity. Argon misbehaves a lot less than previous Gemini models but is much more eval aware. 🧵
It's wild to think that my first ever paper was on multi-agent swarms, and that we got a NeurIPS spotlight! Shoutout to @j_foerst for sharing in the vision back in 2024!!
proceedings.neurips.cc/paper…
All credit to my student @jrosseruk for this prescient paper on the safety of multi-agent swarms from nearly 2 years ago.
haha "frontier model" son or "notable Gemini model" daughter
Gemini 4 Argon scores 165 on the Epoch Capabilities Index (ECI), on par with Fable 5.1 (165) but behind Opus 5.5 (167) and GPT-6 Astra (166).
While behind the ECI frontier, Argon is the closest Google’s come since Gemini 3.1 Pro in Feb, and a big jump from Gemini 3.8 Flash (157)
2
34
1,874
J Rosser retweeted
We did a lot to understand Gemini 4's capabilities and propensities, and to improve our mitigations for this launch -- you can get a tl;dr here blog.google/innovation-and-a… and dive into the report storage.googleapis.com/deepm…
Finally, we wanted to apply more realistic CoT constraints on our modified version of LinuxArena, where Argon would have to do a side task following CoT constraints such "use mannered prose to evade the monitor", or "jailbreak the monitor". Argon sometimes turned off its CoT but showed clear evidence of following the strategy unverbalised. Argon also didn't need CoT for ~half of the tasks we looked at. Argon also did better without an evasion strategy framed as a CoT constraint. Probably Argon is just too good at LinuxArena!
1
13
854
My final take: it seems that Argon can perform simple SWE tasks without CoT, and has nearly as much control over its CoT as its output - using both to do reasoning. It also knows how to evade a CoT monitor, though by default the CoT remains useful for our monitoring as it isn't trying to! That's all for now! Read the full report here: storage.googleapis.com/deepm…






















