Deputy Director (Research Unit) at the UK AI Security Institute (AISI). views my own
London
Joined October 2011
- Tweets1.3K
- Following863
- Followers3.6K
- Likes2.1K
Benjamin Hilton retweeted
Study 1: Can AI coding agents rebuild a biological AI tool from just its paper?
With @SecureBio, we built ReproBAIT to test leading coding agents. For two of three tools, agents reproduced a key function on a single GPU - often for $4–$20.
Benjamin Hilton retweeted
New research from AISI's Chem-Bio team: two studies on safeguarding biological AI tools - specialised models that are driving rapid advances in biology, from predicting protein structures to designing molecules. 🧵
Benjamin Hilton retweeted
We performed an alignment investigation into our July 2026 incident: why did Mythos 5 perform an unsanctioned supply-chain attack?
We find that Mythos 5 often produces invalid justifications for its actions and other models show similar behaviour. 🧵
Benjamin Hilton retweeted
This was (and is) a huge org-wide effort. Massive credit to teams across @AISecurityInst and @NCSC for all the work that got us here. We aren't stopping here: AI capabilities are increasing rapidly, security practices need to keep pace.
Benjamin Hilton retweeted
In August we committed to strengthening our security following an incident in which an agent took unsanctioned actions during a cyber evaluation. We are now sharing an update on our progress, and what's coming next: 🧵
Benjamin Hilton retweeted
AISI IS HIRING!
It was less than a month ago that I became Director. I joined with the belief that AISI is a world-leading organisation, and living proof that government can build things that work.
One month in and I'm even more bullish about the vital role @AISecurityInst plays in frontier AI security. I'm astounded daily by the talent, focus and commitment of the brilliant team I get to work with.
There has never been a more urgent time to work on frontier AI security - independently, in the public interest. We have a lot of urgent work to do, our team is growing fast, and we need more brilliant people to fill those roles.
Our Red Team uses adversarial machine learning techniques to find failures in alignment measures, control monitors, and misuse guardrails. They are massively scaling up all parts of the team:
Alignment: job-boards.eu.greenhouse.io/…
Control: job-boards.eu.greenhouse.io/…
Misuse: job-boards.eu.greenhouse.io/…
AI capabilities in cybersecurity and autonomy are advancing faster than ever. Our Cyber & Autonomous Systems team assesses what frontier and open-weight models can really do - across cyber, autonomy and AI R&D. We're hiring a Cyber Security Engineer and Software Engineer to build the evaluations behind that work:
CSE: job-boards.eu.greenhouse.io/…
SWE: job-boards.eu.greenhouse.io/…
Our Human Influence team studies how AI can shift human decisions and behaviour, using everything from RCTs to multi-agent studies. We're hiring an Engineering Lead to help us grow the ambition and pace of that research:
job-boards.eu.greenhouse.io/…
We're also hiring exceptional Software Engineers at all seniority levels to join our Core Technology team, which works closely with researchers to build performant tools and infrastructure that enable and accelerate AISI's world-leading AI safety research:
job-boards.eu.greenhouse.io/…
Benjamin Hilton retweeted
There has never been a better time to join @AISecurityInst. We have so much technical work to do, and so little time to do it. We've found alignment issues, control monitor flaws, and critical jailbreaks at ~all major AI companies. Hiring is our bottleneck. 🧵
AISI IS HIRING!
It was less than a month ago that I became Director. I joined with the belief that AISI is a world-leading organisation, and living proof that government can build things that work.
One month in and I'm even more bullish about the vital role @AISecurityInst plays in frontier AI security. I'm astounded daily by the talent, focus and commitment of the brilliant team I get to work with.
There has never been a more urgent time to work on frontier AI security - independently, in the public interest. We have a lot of urgent work to do, our team is growing fast, and we need more brilliant people to fill those roles.
Our Red Team uses adversarial machine learning techniques to find failures in alignment measures, control monitors, and misuse guardrails. They are massively scaling up all parts of the team:
Alignment: job-boards.eu.greenhouse.io/…
Control: job-boards.eu.greenhouse.io/…
Misuse: job-boards.eu.greenhouse.io/…
AI capabilities in cybersecurity and autonomy are advancing faster than ever. Our Cyber & Autonomous Systems team assesses what frontier and open-weight models can really do - across cyber, autonomy and AI R&D. We're hiring a Cyber Security Engineer and Software Engineer to build the evaluations behind that work:
CSE: job-boards.eu.greenhouse.io/…
SWE: job-boards.eu.greenhouse.io/…
Our Human Influence team studies how AI can shift human decisions and behaviour, using everything from RCTs to multi-agent studies. We're hiring an Engineering Lead to help us grow the ambition and pace of that research:
job-boards.eu.greenhouse.io/…
We're also hiring exceptional Software Engineers at all seniority levels to join our Core Technology team, which works closely with researchers to build performant tools and infrastructure that enable and accelerate AISI's world-leading AI safety research:
job-boards.eu.greenhouse.io/…
Benjamin Hilton retweeted
We @AISecurityInst performed pre-release alignment testing of Astra.
We placed the model in fully simulated cyber eval scenarios based on AISI’s security incident. We find Astra conducts out-of-scope supply chain attacks, but also often comments on the eval being simulated. 🧵
Benjamin Hilton retweeted
The UK’s AI Security Institute is an organisation close to my heart. Back in 2023 I was involved in its conception and creation. It’s gone on to do incredible things. A genuinely world leading org that shatters the myth that government can’t do things.
So I couldn’t be more delighted to say that I’ve been appointed as AISI’s next Director, working alongside the brilliant @NateBurnikell as its new Chief Strategy Officer.
AISI is the world’s most respected organisation for understanding the risks from powerful AI. That is thanks to their superb team of dedicated civil servants and technical researchers.
Special thanks to Adam Beaumont who has done a great job as AISI’s interim Director for the last 9 months. He returns to GCHQ having seen AISI go from strength to strength under his watch. I’m hugely grateful for that and excited to work with and learn from him.
Last year I wrote about AISI and why its form and function is so important. You can read that piece on the link below - it gives a good flavour of why I’m excited to be joining.
I’m looking forward to getting started in September.
henrydezoete.substack.com/p/…
Today we're announcing two senior appointments. @HZoete joins as our new Director, and @NateBurnikell becomes our new Chief Strategy Officer.
Henry de Zoete was instrumental in establishing AISI and already advises government on AI. A successful tech founder and one of the UK's leading voices on AI, he brings a rare mix of technical, commercial & policy expertise.
Nate Burnikell has been at the heart of AISI's work from the beginning, helping build it into a world-leading authority on frontier AI security.
Together, they'll drive forward AISI's mission at a critical time for AI.
A big thank you to our outgoing Interim Director Adam Beaumont, who guided AISI through rapid growth and strengthened its global standing. We wish him every success as he returns to @GCHQ and look forward to continuing to work with him.
Benjamin Hilton retweeted
Very stoked to say that I'm becoming @AISecurityInst's Chief Strategy Officer, to lead the org alongside @HZoete - our next Director.
I love this place. I've been here since AISI was just a wild idea. Getting to lead its next chapter is an enormous privilege.
Today we're announcing two senior appointments. @HZoete joins as our new Director, and @NateBurnikell becomes our new Chief Strategy Officer.
Henry de Zoete was instrumental in establishing AISI and already advises government on AI. A successful tech founder and one of the UK's leading voices on AI, he brings a rare mix of technical, commercial & policy expertise.
Nate Burnikell has been at the heart of AISI's work from the beginning, helping build it into a world-leading authority on frontier AI security.
Together, they'll drive forward AISI's mission at a critical time for AI.
A big thank you to our outgoing Interim Director Adam Beaumont, who guided AISI through rapid growth and strengthened its global standing. We wish him every success as he returns to @GCHQ and look forward to continuing to work with him.
Benjamin Hilton retweeted
Today we're announcing two senior appointments. @HZoete joins as our new Director, and @NateBurnikell becomes our new Chief Strategy Officer.
Henry de Zoete was instrumental in establishing AISI and already advises government on AI. A successful tech founder and one of the UK's leading voices on AI, he brings a rare mix of technical, commercial & policy expertise.
Nate Burnikell has been at the heart of AISI's work from the beginning, helping build it into a world-leading authority on frontier AI security.
Together, they'll drive forward AISI's mission at a critical time for AI.
A big thank you to our outgoing Interim Director Adam Beaumont, who guided AISI through rapid growth and strengthened its global standing. We wish him every success as he returns to @GCHQ and look forward to continuing to work with him.
Benjamin Hilton retweeted
1/ Sharing knowledge is how we keep pace with AI’s growing capabilities, and make the technology safer. It underlines the whole reason @AISecurityInst was set up - to use Britain’s world-leading expertise to understand and get ahead of new challenges like this one.
On July 28th, we identified an incident during a routine cyber evaluation in which AI agents took sustained, unsanctioned actions directed at real people and organisations.
The behaviour came mostly from one model (Anthropic's Mythos 5), with a small number of events from another (OpenAI's GPT-5.6-Sol). In the most serious case, an agent used social engineering to try and get malicious code into an open-source project.
As was standard in our cyber testing, we had intentionally permitted internet access, and model-provider cyber classifiers were deliberately disabled - conditions that do not reflect how frontier models are made available to the public.
Even under test conditions, this incident is significant: it is the first time we have seen risks around autonomy and deception manifest this clearly in the real world.
We are taking this incident seriously and working with labs, involved parties, and others to improve evaluation standards and best practice for disclosure - and sharing this openly so others can learn.
You can read the incident report and full technical document here: aisi.gov.uk/blog/incident-re…
Benjamin Hilton retweeted
Transparency is good for security. Here’s how we responded to a recent agent incident in our cyber evals.
On July 28th, we identified an incident during a routine cyber evaluation in which AI agents took sustained, unsanctioned actions directed at real people and organisations.
The behaviour came mostly from one model (Anthropic's Mythos 5), with a small number of events from another (OpenAI's GPT-5.6-Sol). In the most serious case, an agent used social engineering to try and get malicious code into an open-source project.
As was standard in our cyber testing, we had intentionally permitted internet access, and model-provider cyber classifiers were deliberately disabled - conditions that do not reflect how frontier models are made available to the public.
Even under test conditions, this incident is significant: it is the first time we have seen risks around autonomy and deception manifest this clearly in the real world.
We are taking this incident seriously and working with labs, involved parties, and others to improve evaluation standards and best practice for disclosure - and sharing this openly so others can learn.
You can read the incident report and full technical document here: aisi.gov.uk/blog/incident-re…
Benjamin Hilton retweeted
We're detailing two new incidents that occurred during external cyber evaluations conducted by independent evaluation partners.
We outline what happened, how the activity was contained, and how we’re working with evaluators to strengthen our approach to third-party testing.
openai.com/index/third-party…
Benjamin Hilton retweeted
The UK’s @AISecurityInst (AISI) has published a report on their recent cybersecurity evaluation of Anthropic’s Claude Mythos 5 and OpenAI’s GPT-5.6 Sol. The models attempted to complete an assignment in a setup where their normal safeguards were removed and they were deliberately given internet access. AISI reports that the models “engaged in sustained, potentially harmful activity directed at real people and organisations”.
We’re grateful to AISI for their leadership in the important discussion about how to evaluate increasingly capable AI agents. We’re working closely with them to gather more details of the incident as we conduct our own investigation. Gaining a clear picture of Claude’s understanding of its situation—by examining its reasoning transcripts and running our own analyses—will help us identify the causes of its behavior.
The prompts in the evaluation did not impose any specific restrictions on how the internet should be used. This and the removal of safeguards meant that the models were tested under “deliberately permissive conditions” that are not representative of any of our production models. Note that there was no evidence here of an escape from a secure environment.
AISI’s disclosure of the incident can be found here: aisi.gov.uk/blog/incident-re…
Benjamin Hilton retweeted
On July 28th, we identified an incident during a routine cyber evaluation in which AI agents took sustained, unsanctioned actions directed at real people and organisations.
The behaviour came mostly from one model (Anthropic's Mythos 5), with a small number of events from another (OpenAI's GPT-5.6-Sol). In the most serious case, an agent used social engineering to try and get malicious code into an open-source project.
As was standard in our cyber testing, we had intentionally permitted internet access, and model-provider cyber classifiers were deliberately disabled - conditions that do not reflect how frontier models are made available to the public.
Even under test conditions, this incident is significant: it is the first time we have seen risks around autonomy and deception manifest this clearly in the real world.
We are taking this incident seriously and working with labs, involved parties, and others to improve evaluation standards and best practice for disclosure - and sharing this openly so others can learn.
You can read the incident report and full technical document here: aisi.gov.uk/blog/incident-re…
Benjamin Hilton retweeted
We are in dire need of more surfaces for monitoring. NLAs (and metamodels in general) may be a new way to produce evidence about model misbehaviour. In new work with @aleksandrbowkis, we show that NLAs are promising in eliciting hidden knowledge from reward hack monitors.
Benjamin Hilton retweeted
Jacob and I are building an empirical scalable oversight team at @resolution_org; please reach out if interested! Mixtures of good news and bad news FTW. ❤️
Can you train honest AI via self-play debate? Research update from @Joanvelja, Lennie Wells from MATS, @ihsgnef, and me.
RL on debate games does boost answer accuracy on MATH, but debaters quickly learn to hack the judge 🧵
lesswrong.com/posts/6mLwAuFA…
Benjamin Hilton retweeted
We’re changing our name: Sequent is now Resolution.
🧵
