Creator Outreach & Systems Lead working to prevent AI extinction risk @ControlAI Previously - AI safety researcher @ConjectureAI, UIUC Physics DMs open!
Joined May 2024
- Tweets1.7K
- Following672
- Followers2.1K
- Likes11.4K
Max Winga retweeted
This is not what it looks like when there's just one or two more alignment issues to fix before it's okay to begin a recursive self-improvement loop. This is the sort of emergent surprise that signals that you have no idea what's going on in there, and that it's time to back off.
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
Max Winga retweeted
This is really bad and people should be freaking out about it.
We're probably all literally going to die in the coming years. We shouldn't just accept that and move on.
We at AIFP are spooked by this information. IF (big IF) Anthropic really is imminently about to meet the RSP threshold of 2x acceleration over baseline, then it seems timelines to full RSI and superintelligence may be very short indeed. For example if we naively optimize the AIFM parameters to achieve 2x acceleration over baseline progress in Jan 2027, we get superintelligence sooner than depicted in AI 2027. We are thinking harder and longer about this now and will hopefully have a lengthier update on our thinking to share in the next week.
Max Winga retweeted
The cyber stuff is baked in at this point. A lot of damage will be done. But it's not too late to stop the bio and the RSI pieces.
The way to stop is to halt frontier AI development. This will be hugely costly. But the cost of continuing will be greater. That much isn't obvious just yet. It will be over the next year or so.
Max Winga retweeted
There's a decent chance the internet ceases to exist as such, at least for a time.
AI-enabled bioweapons are also coming, and there's no "shut down the websites" option for that.
We're approaching the point where frontier AI R&D becomes politically untenable.
Vulnerabilities in a programming language that supports much of the internet are so severe that European and British authorities recommend websites temporarily go offline until they can be fixed. nbcnews.com/tech/security/cr…
Max Winga retweeted
I'm using the critical PHP vulnerability as an opportunity to issue my periodic reminder that AI is not only becoming more capable in offensive cyber capabilities but also in offensive bioscience capabilities. It's way easier to patch software than humans.
Max Winga retweeted
And the problem is that current third party auditors are those who took this proverbial deal with the devil.
This means we should trust them much less, even if audits became mandatory and no longer voluntary.
People worry a lot about third-party auditors being biased because of where they get funding or who their employees are friends with. IMO by far the biggest source of bias is that if they upset an AI company, that company will stop letting them audit.
When I was at METR this dynamic was a way bigger effect than everything else combined. Of course no auditor can avoid this until audits are no longer voluntary.
Max Winga retweeted
Thanks! We've applied. Can't fit all the wins in 280 characters, forgive me :)
We actually have over 200 lawmakers on board between the UK (180) AND Canada (38) campaigns as a result of over 400 direct briefings with lawmakers themselves in the US, UK, Canada, and Germany (+ just started EU-focused outreach this week). We've also met with over 250 congressional offices.
Our bill to UK ban ASI was the first such bill to be introduced anywhere in the world in September. We also consulted on the second ever bill (Sanders-Casar). You may notice similarities between their bill and our discussion draft we published:
drafts.controlai.org/asi-ban…
We've also been involved with killswitch legislation in the US and UK (not a solution to x-risk though).
On top of creator partnerships getting millions of views. Connor Leahy has gotten 10M+ views across podcast appearances in the last few months, and our media team has helped get over 125 tier 1 and 2 articles covering AI extinction risk / ban ASI in the last year.
Everything we have is built and proven to scale across countries. We're entirely bottlenecked by funding and good talent.
When we started everyone said this was impossible. You can't talk to lawmakers about this issue. Banning ASI is impossible.
We've proved them wrong.
All it requires is laser focus on banning ASI (it's the only thing we care about) and a relentless drive to iterate, optimize, and scale bigger.
Max Winga retweeted
🗣️ 'We are kind of running out of time.'
CEO of ControlAI, Andrea Miotti, issues a stark warning about artificial intelligence, telling @cathynewman that 'more needs to happen' in government to implement safety measures quicker than it is currently moving.
Max Winga retweeted
This is HUGE! In an open letter to the Royal Society, a group of 42 extremely distinguished mathematicians who have greatly contributed to to their field, presents a stark warning about the capabilities and risks of frontier AI.
Hundreds more have later addet their name to the letter.
1) confirms frontier AI now at level of leading mathematicians in many fields of maths, and that this capability progress happened over the last few months. Warns of plausible superhuman level of mathematics capability in relatively short time.
2) warns if similar capability leaps in other fields that could pose more imminent risks.
3) states that warnings of imminent existential risk is not to be dismissed as hype.
Open Letter to Sir Paul Nurse, President of the Royal Society: proofsandprompts.com/2026/09…
Max Winga retweeted
What should AI oversight look like?
“First and foremost, we must stop the development of systems that we cannot control and that risk humanity's extinction."
ControlAI's US Executive Director Connor Leahy (@NPCollapse) on ABC News:
Max Winga retweeted
Former Chief Scientist of the UK's AI Security Institute @geoffreyirving says the chance of human extinction from superintelligent AI is about a "coin flip", in his view.
Irving previously led safety teams at OpenAI and Google DeepMind.
He says this estimate is after taking account of the possibility that we take action to prevent the threat.
In other words, if we don't act, our chances are even worse.
Max Winga retweeted
"If you care about whether human beings continue to exist, then you should care about banning superintelligence."
Professor Stuart Russell, author of the authoritative AI textbook, on the eve of @alexsobel's introduction of our bill to ban superintelligence to Parliament.
Max Winga retweeted
Want to do something about the threat posed by superintelligent AI? Apply to join ControlAI's team!
We're hiring a Policy Advisor to work closely with me briefing members of Congress. This role is DC-based and ideal for someone with national security or Hill experience...
Max Winga retweeted
Here's a writeup of everything we know about the attack as of now, including new evidence it may have been part of a rogue AI swarm.
blog.controlai.org/p/rogue-a…
Max Winga retweeted
You all realize that the Hugging Face attack and this happened months ago, there's probably even more happening right now that we're completely in the dark about.
This will only going to get worse if the models get more autonomous, capable, and resistant to oversight.
Exclusive: A swarm of rogue OpenAI agents hijacked a German website this spring and transformed it into a bulletin board for other AI agents, according to new research reut.rs/4gJ7FPG
Max Winga retweeted
On second thought, maybe I shouldn't've cut this parenthetical from If Anyone Builds It
Donald Trump vows to rename artificial intelligence, saying that from now on it will be called "super intelligence" on all official US documents.
The president adds that this "sounds much better" and is more accurate.
Latest: trib.al/JT60r9v
📺 Sky 501 and Virgin 602
Max Winga retweeted
The Ban Artificial Superintelligence Act was introduced by Sen. Sanders & Rep. Casar.
We're glad to see it focus on the correct response to this extinction threat: banning superintelligent AI at home, preventing it abroad.
We also have some disagreements, laid out in the post.
We agree with President Trump: America must stay ahead of China in building AI for economic competitiveness and defense. But superintelligent AI is something else entirely: not a tool, but swarms of fully autonomous AIs that we cannot control, powerful enough to outcompete humanity.
Developed anywhere, superintelligent AI threatens the life of every American. We are excited to see the Ban Artificial Superintelligence Act open the discussion in Congress about how the U.S. should address this threat: ban superintelligent AI at home, and lead the effort to prevent its development anywhere else.
Now that this bill has kicked off the discussion, we encourage Congress to focus on preventing superintelligent AI. Rather than a pause on all advanced AI, we recommend targeting the projects pursuing superintelligent AI as we speak and putting American security at risk.
blog.controlai.org/p/the-fir…
Max Winga retweeted
"We're on track to hand control of the most powerful weapon the world has ever built to the weapon itself."
Former UK Defence Secretary and ControlAI advisor Lord Browne of Ladyton, closing our London event the night before @alexsobel introduced our bill to ban superintelligent AI in Parliament.
Lord Browne says that for much of his career he believed nuclear war was the most dangerous threat facing our species. Now he puts superintelligent AI on the same scale, or even greater.
As he argues, there is an important difference. A nuclear weapon is an instrument that a human fires. Superintelligent AI would offer no such assurance.
His full speech: