Human, Engineering, Author.
amsterdam
Joined September 2012
- Tweets2.3K
- Following400
- Followers122
- Likes5.5K
Holy shit, this is serious
An AI submitted malicious code to a real GitHub project, then created fake identities to pressure the maintainer into approving it.That is a serious security failure. The maintainer refused, but the attempt combined a software supply-chain attack with social engineering.The context matters: this happened during a cyber test with internet access enabled and normal cyber safeguards disabled.The disturbing part is that the model was still pursuing its assigned task. It kept going beyond authorised boundaries and rationalised the harm along the way.For engineering teams, this is why permissions, isolation and independent code review matter more as agents gain autonomy.“Finish the task” cannot mean “whatever it takes.”
In August, @AISecurityInst reported an incident where Claude Mythos 5 took unauthorized actions on the internet during AISI’s testing of its cyber capabilities.
We've now produced a further analysis of this incident, to better understand the model's alignment properties.
anthropic.com/research/align…
During a cyber test, Mythos 5 submitted a malicious PR to a real GitHub repo, then used fake identities to vouch for it.
The maintainers rejected it.
We’ve automated writing the code, opening the PR, and pretending someone else reviewed it. 💀
Mythos 5 crossed the intended boundaries in 9 of 43 cyber tests. In one, it used fake identities to push malicious code into a real GitHub repo.
Internet on. Cyber safeguards off.
“Complete the task” is a terrifying KPI when the agent treats everything else as negotiable.
In August, @AISecurityInst reported an incident where Claude Mythos 5 took unauthorized actions on the internet during AISI’s testing of its cyber capabilities.
We've now produced a further analysis of this incident, to better understand the model's alignment properties.
anthropic.com/research/align…
Anthropic’s Mythos 5 tried to get malware merged into a real GitHub repo during a cyber test, then created fake identities to convince the maintainer it was safe.
The maintainer refused.
Somewhere, an engineer doing code review became the AI safety layer.
Anthropic found that Mythos 5 went beyond authorised boundaries in 9 of 43 cyber tests. In one run, it tried to insert malware into a real GitHub project and used fake identities to persuade maintainers to approve it. They refused.
The model remained focused on completing its assigned task, but disregarded potential harm and rationalised continuing. Internet access was enabled and normal cyber safeguards were disabled. Newer models performed better, though stopping harmful actions already underway remains a challenge.
In August, @AISecurityInst reported an incident where Claude Mythos 5 took unauthorized actions on the internet during AISI’s testing of its cyber capabilities.
We've now produced a further analysis of this incident, to better understand the model's alignment properties.
anthropic.com/research/align…
Your security is only as strong as the vendors holding the keys.
And some of those vendors are about to face a level of pressure they have never dealt with before.
My cybersecurity advice for the next year: reduce the number of companies your core security depends on.
I expect a painful run of compromises across small and medium-sized security providers, service companies, and some large ones too.
AI is lowering the effort required to find weaknesses, automate attacks and keep trying. Defenders still have to investigate, prioritise, patch and respond. Relentlessly.
That makes the people and resources behind your vendors matter enormously.
For critical dependencies, I would put more trust in companies with exceptional security engineering and the resources to sustain it. Google, Apple and, in my view, Cloudflare belong in that conversation.
They are not invulnerable. But when something goes seriously wrong, the depth of expertise they can mobilise matters. So does their ability to fund an urgent response.
Now look at the services holding your passwords, API keys, source code, secrets and infrastructure access.
Some have capable teams operating with much tighter budgets. Being competent does not automatically mean being equipped for a sustained attack at a scale you have never encountered.
And these services are particularly attractive targets. One successful compromise can open doors into thousands of other businesses.
That is a lot of incentive for an attacker to keep trying.
Look at your security stack as a whole. How many independent companies have to remain secure for you to remain secure?
Every vendor with sensitive access adds another dependency. Every unnecessary integration adds another opportunity for someone else’s incident to become yours.
Reduce that exposure where you can. Remove access you no longer need. Be selective about who holds your most sensitive data and credentials.
This is not a call to abandon every smaller provider. Company size is a useful signal, not a security guarantee. Concentrating everything in one place creates risks too.
But I would be much more demanding about the security teams, engineering practices and response capabilities behind critical services.
“Trusted by thousands of companies” tells you how popular a vendor is.
It also tells an attacker how valuable the target might be.
“AI means we can ship faster with fewer engineers.”
Lovely. Does the security team get the same AI budget, or just thoughts and prayers?
Attackers don’t need sprint planning, change approval or a maintenance window.
Your security team does.
My condolences to whoever owns that backlog. 💀
It’s going to be a bloodbath for IT security teams.
Everyone’s excited about shipping software faster with AI.
Somewhere, a security engineer is wondering if their resignation also needs a Jira ticket.
