It’s clear that we must put an end to the dangerous race to superintelligent AI. This should be accomplished by banning its development through an international trust-but-verify regime.

The inability of developers to understand or control the behavior of the AIs they’re building, combined with their rapid pursuit of superintelligent AI, is why top AI scientists are warning that humanity is faced with a risk of human extinction.

This has some effect, but as we’ve seen, they can still be misused. In the case of open-weight AIs, developers have essentially no ability to apply such mitigations, as they can be, and are, trivially removed.

AI developers don’t want cybercriminals to take their models and use them to cause damage, and in fact, they take measures specifically to try to prevent this: putting AIs through further training in an effort to make them less likely to assist with harmful requests.

One would be that of AIs escaping control, the other of bad guys controlling AIs. But these issues actually have the same root, which is that developers can’t control their systems.

The second piece here is that AI developers don’t know how to control or ensure the safety of the systems they’re building. It is tempting to understand the problem of rogue AIs and the use of powerful AI agents by cybercriminals as two distinct issues.

As these AIs become more capable, new abilities, like the ability to find flaws in computer systems, fall out in unpredictable ways. The AI companies aren’t especially interested in developing nation-state-level cyberweapons; it’s happening as a side effect of the AIs improving.

There are two main things to understand here. The first is that AI companies are in a race to develop ever more powerful AIs, with the goal being to create superintelligent AI, fully autonomous AI systems smarter than any human and acting without human direction.

Importantly, the flaws found in PHP aren’t an isolated case. Just eyeball the graph: the number of such flaws being found in software is staggering. This is increasingly being driven by AI.

Cloudflare, an internet infrastructure and security company which around 28% of the web flows through, has already recorded millions of attempts against its customers to exploit the PHP vulnerabilities disclosed this week.