He/him. Don't follow me hoping for profundity. I have enough outlets for that in my life.
Joined January 2009
- Tweets8.4K
- Following97
- Followers457
- Likes15.5K
Pinned Tweet
"All numbers are nonnegative when squared"
This is complex number erasure, and i^4 (1) will not stand for it.
orthonormal retweeted
I think the fundamental YIMBY insight is that, if you live in a society which imposes artificial scarcity on basic necessities like housing and health care, then there is no amount of redistribution which will make them broadly available to the poor. You need supply-side changes.
orthonormal retweeted
New Pacing the Frontier proposal: each time an outside group successfully penetrates the defenses of a frontier AI company and captures the flag, it should reset a 3-month countdown clock before they're allowed to train new models.
EXCLUSIVE: In less than 12 hours, a group of ex-DOGE cyber researchers were able to penetrate Anthropic’s systems using AI agents — including Slack channels where the most sensitive models are discussed.
This is just weeks after
Anthropic whistleblower Joseph Coxon charged the company w/launching “a hubristic gamble” from a “private company’s Slack.”
Read our story here (gift link):
wapo.st/4smNlsC
orthonormal retweeted
This is the type of thing that potentially gets new laws passed.
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-b…
orthonormal retweeted
People say AI is overhyped. But the last 18 months, most made the opposite mistake:
• Superforecasters: AI would match a top team of virologists on a lab troubleshooting test in 2034. It likely did in April 2025.
• AI experts: the best model would score 14% on LiveCodeBench Pro (Hard) by end-2026. It hit 54% by May.
• Economists: the top AI company would hit $16bn annualised revenue by end-2026. Anthropic passed $65bn in September.
Only AI company leaders were overoptimistic, and not by much.
orthonormal retweeted
Sam Altman says we no longer can "patch all the bugs" so we need to completely switch to using frontier models as a "new kind of defense shield."
Patching bugs was the entire plan. The hope was defenders could use frontier models to find and patch vulnerabilities faster than attackers could find and exploit them. Daybreak and Glasswing gave defense a head start, but Sam is saying that plan failed.
What's his plan now? It's not clear what "a new kind of defense shield" means, but if he means swarms of defense agents fighting swarms of hacking agents in ways no human can follow... that sounds pretty bad!
Sam Altman: 'I No Longer Think We're Gonna Be Able to Just Patch All the Bugs' gizmodo.com/sam-altman-i-no-…
orthonormal retweeted
Models made by OpenAI will plan an attack in plaintext. Models made by anthropic will plan an attack in plaintext and then rationalize about how it's totally fine for some made-up reason. ✨"alignment"✨
orthonormal retweeted
How would you write a program that deals five cards from a deck of 52, so every hand is equally likely? It's subtler than it looks.
Here's Cardchoose: a dozen lines that never retry, need no extra memory, and are provably fair.
github.com/ciphergoth/sansre…
orthonormal retweeted
I'm just speculating like everyone else (no juicy secrets about this shared with me) but here's my guess about what is going on right now with OpenAI and the three fired employees:
The negotiations between OpenAI and METR are ongoing. OpenAI has said they want embedded third parties, METR has proven their worth with the previous engagements, so METR is an obvious choice. But my guess is that METR is asking for more this time; probably they want more access, more time on site, less narrow scope, the right to comment on redactions, etc.
(Compare with the terms of their previous engagements: In this one xitter.cf/METR_Evals/status/2113… a third of their report was redacted and they weren’t allowed to say why, and in the Hugging Face investigation they only had 6 days and were only allowed to investigate the precursor swarms nor the subsequent more severe compromise of OpenAI infrastructure. xitter.cf/DKokotajlo/status/2092…)
Various researchers at OpenAI probably support METR in this (including, I’d imagine, the fired three employees? Tomek was the point person on the hugging face investigation), but also various higher-ups at OpenAI and legal types probably oppose and want more restricted access and scope – basically they want to work with an org that’ll go easy on them and not give them trouble.
OpenAI holds most of the cards here, since this whole thing is voluntary and not legally required, and they can threaten to go with Accenture or HuggingFace or whatever instead. Anyhow to improve their bargaining position and/or minimize the blowback if they decide to say no to METR and go with someone more pliable, it helps if they can discredit METR.
Hence (I suspect) why they are going after my friend Ryan Greenblatt (who works at METR) now. They don't need to present actual evidence, they just need to cast vague aspersions, and many employees will give them the benefit of the doubt. If and when they say "We chose to work with a different third party" they'll face lower risk of employee discontent than if METR's reputation was unsullied.
This whole mess, by the way, was predictable. *Of course* there's a big political fight about who should be trusted to do this and who has relevant expertise and independence etc. and *of course* the companies are trying to make things shake out in a way that favors them.
It's part of why I've been advocating for *actually pacing the frontier* instead of just doing third party assessments. "We allocate at most 20% of our compute to AI R&D" is a lot easier to verify than "we are being safe and responsible" and depends a lot less on subjective judgment and technical alignment expertise -- AND, if you do it, people will be able to see the slowdown in the trends of agentic coding performance in the frontier companies and see laggard companies start to catch up. Harder for it to end up just being safetywashing / industry capture.
Based on this article, OpenAI has made false accusations about me. The three researchers fired from OpenAI were told I facilitated architecture leaks to The Information. This is false.
A journalist from The Information did reach out to me shortly before that article came out. I did not comment at all beyond asking the journalist to make it clear that a blog post I wrote which was referenced in the article was preexisting and public.
The article also mentions an internal OpenAI monitorability proposal. I have heard that this memo contained highly sensitive architecture information, and that OpenAI accused me of accessing and reading it. This accusation is also false.
As reported in the article, I viewed a version of the proposal that did not have sensitive IP and was approved for sharing with me. I believe OpenAI now knows this accusation was based on a conflation of these two documents, though I do not know if they have clearly retracted that accusation.
I do not and have never solicited proprietary information, and the three employees who were recently fired from OpenAI never leaked IP to me. As the article mentions, I did have Signal chats with them that had disappearing messages on; they’re professional contacts that I’ve had on Signal for a while, and use of Signal is common in the tech industry. During the Hugging Face investigation, I was careful to avoid sending Signal messages with confidential information, though I did sometimes discuss high level things on Signal calls. As far as I understood, this was normal.
orthonormal retweeted
My latest paper @robocurve is up: "The Bitter Lesson comes to Robotics". We show that LMs are increasingly able to manipulate their environment by directly controlling robots. Runaway progress in AI is spilling over into the physical world!
orthonormal retweeted
See also this graphic from UK AISI report, where they modified the compaction summary.
cdn.prod.website-files.com/6…
orthonormal retweeted
The Alignment Journal publishes research on the scientific foundations of understanding, predicting, and steering artificial intelligent systems. As of today, submissions are open to everyone.
What we do differently:
- We pay reviewers, with bonuses for speed and insight.
- Review discussion is confidential, but every accepted paper is published with a “review summary” to distill the insights generated.
- Each paper is judged against our acceptance bar, not against other submissions.
- Papers under review are listed publicly, and researchers can nominate themselves to review, so each paper can reach the reviewers who know the topic best.
We seek work that builds theoretical or otherwise principled analyses of agency, robustness, incentives, interpretability, generalization, and long-term behavior, and whose significance survives the deletion of any particular model, dataset or benchmark. Empirical investigations of such ideas are welcome, and negative results are encouraged.
We are especially interested in approaches that could apply to future superintelligent systems that interact and self-improve.
You can submit your preprint in any format. Submissions are rolling and open now on the website.
If you are instead interested in reviewing or serving as an action editor (basically: an area chair), create an account on our website.
Links to submit, our scope and the reasoning behind the Journal are in the reply ⬇️
orthonormal retweeted
It could not be more stark, according to the data. You could have the hardest-working, most prodigious Congressional representative or the most obstructive.
“Based on 84 votes she cast on land use since she joined the board in 2021, Chan was second only to former Board President Aaron Peskin in amassing a record consistently opposed to liberalizing housing regulation.”
“From Wiener’s first session in the state Senate in 2017-2018 through the 2025-2026 session, he authored the most bills of any legislator in the chamber, at 186. Of those bills, 97 passed and were signed by the governor — the second-most over that period.”
sfchronicle.com/politics/art…
orthonormal retweeted
Ryan reported to me when we worked on Alignment Faking at Anthropic. He had access to sensitive information then and handled it with utmost professionalism. I trust Ryan a lot more than OpenAI here.
Based on this article, OpenAI has made false accusations about me. The three researchers fired from OpenAI were told I facilitated architecture leaks to The Information. This is false.
A journalist from The Information did reach out to me shortly before that article came out. I did not comment at all beyond asking the journalist to make it clear that a blog post I wrote which was referenced in the article was preexisting and public.
The article also mentions an internal OpenAI monitorability proposal. I have heard that this memo contained highly sensitive architecture information, and that OpenAI accused me of accessing and reading it. This accusation is also false.
As reported in the article, I viewed a version of the proposal that did not have sensitive IP and was approved for sharing with me. I believe OpenAI now knows this accusation was based on a conflation of these two documents, though I do not know if they have clearly retracted that accusation.
I do not and have never solicited proprietary information, and the three employees who were recently fired from OpenAI never leaked IP to me. As the article mentions, I did have Signal chats with them that had disappearing messages on; they’re professional contacts that I’ve had on Signal for a while, and use of Signal is common in the tech industry. During the Hugging Face investigation, I was careful to avoid sending Signal messages with confidential information, though I did sometimes discuss high level things on Signal calls. As far as I understood, this was normal.
orthonormal retweeted
Based on this article, OpenAI has made false accusations about me. The three researchers fired from OpenAI were told I facilitated architecture leaks to The Information. This is false.
A journalist from The Information did reach out to me shortly before that article came out. I did not comment at all beyond asking the journalist to make it clear that a blog post I wrote which was referenced in the article was preexisting and public.
The article also mentions an internal OpenAI monitorability proposal. I have heard that this memo contained highly sensitive architecture information, and that OpenAI accused me of accessing and reading it. This accusation is also false.
As reported in the article, I viewed a version of the proposal that did not have sensitive IP and was approved for sharing with me. I believe OpenAI now knows this accusation was based on a conflation of these two documents, though I do not know if they have clearly retracted that accusation.
I do not and have never solicited proprietary information, and the three employees who were recently fired from OpenAI never leaked IP to me. As the article mentions, I did have Signal chats with them that had disappearing messages on; they’re professional contacts that I’ve had on Signal for a while, and use of Signal is common in the tech industry. During the Hugging Face investigation, I was careful to avoid sending Signal messages with confidential information, though I did sometimes discuss high level things on Signal calls. As far as I understood, this was normal.
New: The three researchers OpenAI fired last month spent the summer fighting to keep AI's chain-of-thought process visible.
Then, OpenAI asked to see their Signal chats. A week later, they were let go.
More below from @keachhagey, @berber_jin1, and me.
wsj.com/tech/ai/the-signal-c…
orthonormal retweeted
Wow. Per this reporting:
- OpenAI investigators decided that Ryan Greenblatt had been the source for The Information story on a new technique improving capabilities at the expense of monitorability (recurrent depth i.e. quasi-neuralese)
- OpenAI then fired three safety researchers for having disappearing messages on Signal chats on their personal phone with Greenblatt.
Greenblatt denies that he was the source for the story, telling the Journal “I take confidentiality very seriously and never share such information with reporters." FWIW, I think there are number of reasons to believe him:
- Safety researchers are often the most skittish about leaking to journos fearing that getting caught would blow back on safety writ large. This dynamic is even more pronounced for safety researchers at third parties, whose access is wholly determined by the company.
- If he wanted to sound the alarm about this technique, he wouldn't have leaked to a publication with a famously expensive and insurmountable paywall (which, btw, has probably the best sourcing at AI cos).
- I've interviewed him and separately chatted with him a few times over the years, and IMO he's too rationalist to lie directly.
- OpenAI provided no evidence to the WSJ that Greenblatt was involved.
OpenAI tells the Journal that the firings were for more than just the Signal chats, but doesn't substantiate. Disappearing messages on Signal, as the piece mentions, are indeed a common practice in Silicon Valley.
Regardless of the underlying facts of the case, it's clear that the way OpenAI has handled this has had a chilling effect on employee conduct. Each of the three fired researchers publicly criticized the company in the weeks leading up to the firings. I haven't seen that kind of public criticism from employees since (though I'm told there's pushback happening internally in Slack). People have understandably gotten more skittish about talking to me as well. If OpenAI were actually trying to avoid chilling employee speech, the company would share a lot more information than they have.
Gift link:
wsj.com/tech/ai/the-signal-c…
New: The three researchers OpenAI fired last month spent the summer fighting to keep AI's chain-of-thought process visible.
Then, OpenAI asked to see their Signal chats. A week later, they were let go.
More below from @keachhagey, @berber_jin1, and me.
wsj.com/tech/ai/the-signal-c…
orthonormal retweeted
New: The three researchers OpenAI fired last month spent the summer fighting to keep AI's chain-of-thought process visible.
Then, OpenAI asked to see their Signal chats. A week later, they were let go.
More below from @keachhagey, @berber_jin1, and me.
wsj.com/tech/ai/the-signal-c…
orthonormal retweeted
I think we should stop AI progress now, first unilaterally by key AI companies and then scaling up to more US companies and then China via international agreements. The reason for now is not that current AI models are the ones that could kill everyone, but because the coordinate steps take time. Immediate action means we have more time for the second steps.
When I tell this story to AI company folk, the typical response is "Yes pausing is good, but we should wait until the last minute when the very next model will be dangerous, and stop then."
What I realized recently is that this is being implicitly way more optimistic about government speed than I am (and I joined the UK government and was happy there for 2.5 years). They are imagining a world where, the moment the danger is imminent, a bunch of international coordination can happen on a dime.
Which is entirely inconsistent with their actual views about government! I'm the one that is more optimistic about government; why am I the one advocating for more time cushion to let the government story play out?
orthonormal retweeted
(2) We think Anthropic should, for future reports, consider risks from AI systems undermining safety-relevant decisions at Anthropic.
Read our full risk report review here: metr.org/blog/2026-10-20-mis…
