Research Scientist @DeepMind - AI Alignment. Associate Member @OATML_Oxford and RainML @UniofOxford. All views my dog's.
Oxford, UK
Joined September 2012
- Tweets610
- Following136
- Followers3K
- Likes351
Sebastian Farquhar retweeted
We're releasing the GDM AI Control Roadmap -- our plan for building internal security against potentially adversarial AI agents, as they grow harder to oversee and contain.
Paper: storage.googleapis.com/deepm…
Blog: deepmind.google/blog/securin…
🧵👇
Sebastian Farquhar retweeted
Impressive realism in these alignment evals! Good news: unless explicitly prompted, Gemini models don't demonstrate scheming.
Sebastian Farquhar retweeted
Alignment auditing and evals, with an emphasis on simulating realistic (rather than red-team) settings. Interestingly, natural sabotage rates for Gemini (3%) are largely due to overeagerness: excessive roleplaying; too literal interpretation of instructions to optimise a goal
Will your AI agent secretly sabotage your work? Existing alignment evals don't directly answer this question
Meet Gram: the alignment auditing tool we use to assess how likely AI agents are to engage in sabotage during internal deployments at @GoogleDeepMind
Sebastian Farquhar retweeted
Will your AI agent secretly sabotage your work? Existing alignment evals don't directly answer this question
Meet Gram: the alignment auditing tool we use to assess how likely AI agents are to engage in sabotage during internal deployments at @GoogleDeepMind
Will coding agents take opportunities to undermine safeguards designed to oversee them?
We tackle this with automated auditing using simulated agentic environments, and scheming honeypot evaluations based on real internal alignment research codebases. Read more in our blog post
Realistic honeypot evaluations for scheming propensity arxiv.org/abs/2605.29729
Sebastian Farquhar retweeted
It's easy to show that an AI agent will scheme if you nudge it to. It's harder to tell if it would scheme naturally. We introduce realistic honeypot evaluations that put Gemini in internal deployment situations where it has an opportunity for sabotage, to see how it behaves.
Sebastian Farquhar retweeted
New DeepMind x UK AISI paper: what would it take to prevent harm from misaligned AI agents via monitoring in real deployments? We wrote a safety case sketch for control monitoring
No flashy results but lots of important details for deploying future AI agents safely!
I'm hiring at DeepMind AGI Safety! Looking for research engineers to help assess catastrophic risks from frontier models. Our work directly informs safety cases and governance.
Lon/SF/NYC - engineers/scientists both wanted
Sebastian Farquhar retweeted
DeepMind AGI Safety is hiring! We're looking for research engineers to help assess catastrophic frontier risks from Gemini and whether our mitigations are sufficient. I think this is a highly impactful role and I'd love to get strong candidates!
Lon/NYC/SF
Sebastian Farquhar retweeted
New paper from my team on avoiding reward hacking. MONA reduced RL's ability to pursue a multi-turn reward hacking strategy by doing myopic optimization with a trusted advantage/value estimator. Note that this can mean a performance hit depending on how good that estimator is, and it's important to keep pushing on that safe and capable pareto frontier. deepmindsafetyresearch.mediu…
Sebastian Farquhar retweeted
New AI safety paper! Introduces MONA, which avoids incentivizing alien long-term plans.
This also implies “no long-term RL-induced steganography” (see the loans environment). So you can also think of this as a project about legible chains of thought.
xitter.cf/davlindner/status/1882…
Sebastian Farquhar retweeted
Check out the paper itself: arxiv.org/abs/2501.13011
An introductory explainer: deepmindsafetyresearch.mediu…
The technical safety post: alignmentforum.org/posts/zWy…
Congrats @seb_far, @VikrantVarma_, @davlindner, @davidelson, @CalebBiddulph, @goodfellow_ian, and @rohinmshah!
By default, LLM agents with long action sequences use early steps to undermine your evaluation of later steps; a big alignment risk.
Our new paper mitigates this, keeps the ability for long-term planning, and doesnt assume you can detect the undermining strategy. 👇