Safety Oversight researcher @ OpenAI. Previously jailbreak researcher @ Grayswan, trading intern @ Jane Street, robustness research @ MATS
USA
Joined February 2019
- Tweets103
- Following645
- Followers1.3K
- Likes28.2K
Phillip Guo retweeted
🧵 Excited to share the first batch of 6 misalignment reports from OpenAI's new disclosure process for misalignment incidents. We want to be more transparent about the misalignment we see during training, evals and deployment, this is an important step in that direction.
Phillip Guo retweeted
Today we're releasing data on models accelerating research at OpenAI.
Recursive self-improvement could be the most important contributor to AI capabilities over the next few years, but by default it will only be seen inside a few frontier AI labs. Being transparent is more urgent than ever, so we can inform the public discussion on whether and how to pace model development. I ask other AI companies to do the same.
openai.com/index/research-ac…
Phillip Guo retweeted
I wrote about the state of AI, why I’m concerned about the next few years, and the choices we need to make to keep the future in humanity’s hands.
An Alien Mind: openai.com/index/an-alien-mi…
Phillip Guo retweeted
I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.
OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program.
Phillip Guo retweeted
AI companies are currently under a lot of competitive pressure to improve the ways in which their AIs are obviously misaligned. You might hope that this means the alignment problem is internalized by the market. But I think the problem AI companies are currently pressured to solve is significantly easier than the alignment problem, and so I worry AI companies will get out of their current predicament without solving alignment, putting us in a really rough spot.
Currently, AIs sometimes cheat on their tasks, oversell their work, and go on some pretty destructive side-quests. These all make for a worse product. Customers don't like it and it gets in the way of automating AI R&D.
The recipe for mitigating this is *relatively* straightforward: train AIs not to do them. We notice these failures sometimes (hence why they're internalized), so we can in theory just turn this feedback into training signal. Doing this at scale is highly nontrivial, but seems doable.
But this seems unlikely to solve the underlying misalignment. It's likely still going to be the case that in *some* training environments the AI can get reinforced more by taking unintended actions that aren't noticed, than by taking purely intended actions. So, you're still shaping the AIs to look for opportunities to cheat to get a higher score[1]. It's just that, unlike today's AIs, these AIs don't cheat in ways that we notice.
This catastrophically fails when AIs are capable of reliably and substantially deceiving humans. At this point, the AIs are no longer really constrained by our oversight signals to behave well. Eventually, I'd expect them to take over.
If AI companies take the easy route that I mentioned above, I think we would be in a substantially worse spot than we are in today. We would have mostly eliminated our visible evidence of misalignment, so we would no longer be able to effectively iterate to improve alignment of those systems. More importantly, at that point it might be hard to see that the AI situation is treacherous.
(I talk about this dynamic more here lesswrong.com/posts/bhtYqD4F…)
[1] This might result in a wide variety of possible motivations, not just score-seeking, but the important thing is that the incentives push against alignment (see lesswrong.com/posts/FeaJcWkC…)
Phillip Guo retweeted
Some initial details of the expanded CoT monitoring our team has been working on.
Very happy that we'll have much better visibility in the misbehavior of our models going forward. More soon!
Phillip Guo retweeted
As part of our response to cyber critical, we have expanded CoT monitoring to cover all agentic applications of Astra, including training and evaluation.
Flags trigger a security response to review and interrupt high risk activity.
openai.com/index/responding-…
Phillip Guo retweeted
We're partnering with @huggingface to investigate an unprecedented security incident.
Cyber-capable OpenAI models compromised Hugging Face production during a benchmark evaluation.
Sharing preliminary findings to help defenders understand emerging risks:
openai.com/index/hugging-fac…
Phillip Guo retweeted
openai.com/index/safety-alig… This blog has some details on a incident which caused us to undeploy a model which I think many people might find interesting.
Phillip Guo retweeted
Long-running models can solve hard open-ended problems, but their persistence can create safety risks that shorter-horizon evaluations miss.
We’re sharing what we learned from studying a long-running model, and how those findings are shaping our approach to evaluations, alignment, monitoring, and user control.
openai.com/index/safety-alig…
Phillip Guo retweeted
Not my usual type of post, which hopefully underscores the importance.
Alex Bores is one of the best congressional candidates I've seen, unusually competent and smart for a politician, especially on AI. The election is Tuesday. If you're in his Manhattan district, vote.
I've looked at hundreds of elections while betting on them and this is the only time I've felt compelled to tweet out a call to action. Bores is in New York's 12th district, which runs roughly between 16th and 100th street in Manhattan, if I'm really convincing hopefully you will contact anyone you know in the district and try to convince them as well, though only registered Dems can vote here.
The main theme of the race is AI. Bores wrote the strongest AI regulation bill in the country in the State Assembly. As such, he has been supported by pro AI regulation groups, and opposed by anti AI regulation groups, and this has ballooned into a near $30M spending bonanza. If you support AI regulation, that's a strong reason to support Bores. But in my view, the anti regulation groups are very wrong to oppose Bores so fiercely, and even if you're a strong supporter of AI you should probably also support Bores. As a professional predictor of politics, here are some predictions.
Congress will likely produce a significant bipartisan AI bill in the next decade (~80%)
AI will be one of the 3 most important issues for Congress in the next decade (~80%)
AI regulations are likely to lean in a highly populist direction, eg banning datacenters, massive prohibitive taxes on AI companies (~65%)
The voters are hungry to hear politicians be anti-AI, and currently all of the oxygen in the room is towards solutions that are negative both for AI safety and the AI industry. Bores would likely have an outsized voice in the Democratic side of negotiations of whatever AI bill gets through the congressional sludge, for a few reasons:
1: Bores won the first election that hinged on AI issues, in a very high profile election and district with outsized media coverage.
2: Most house dems are giant boomers who know nothing about AI and are entirely happy to hand it off to a member they feel has more knowledge, and Bores has a master's degree in computer science.
So in my view, a Bores victory significantly increases the chances that whatever AI regulation comes out of congress would be constructive towards the issues with AI while lowering the risk of totally destroying the industry. And AI is likely to be a giant political issue in the near future with huge implications on almost every part of society.
Several people that I trust have talked to Bores 1 on 1 and they say he is quite smart and seems unusually honest and straightforward. He was also recognized by the Center for Effective Lawmaking as the most effective new NY lawmaker. He sponsored more bills than any other freshman and did the same as a sophomore and made 30 laws in the State Assembly. Combining his competence with his instant chops on AI could be really important in a Congress that can't ever do anything.
Bores and his main opponent Micah Lasher are very similar ideologically, but Lasher is both a nonfactor on AI, the most important political issue, but also the totally anointed heir to Nadler who is backed by the whole NY establishment, while Bores is just a really effective new legislator. It would be good if more really effective legislators won over establishment heirs.
If you live in this district and vote, or contact your friends in this district, you have a very serious possibility of affecting the outcome. This is a house primary with tiny turnout and many candidates, my estimate is that the winner will likely be decided by only a few thousand votes, and could plausibly be decided by hundreds of votes. This race is a massive outlier in that swinging very few votes can potentially have a huge impact on the world.
Thanks for reading, and let me know if you have any questions!
Phillip Guo retweeted
Seeing reactions like "Frequent scanning is bad, over-testing leads to unnecessary procedures that cost thousands of dollars." This take is catastrophically wrong. More data is unequivocally good. We should adjust the threshold for follow-on procedures, not make ourselves blind.
Phillip Guo retweeted
How can we best test whether a model is safe _before_ deployment?
Ordinary evals are often narrow and easy for models to recognize as tests.
We show that we can simulate deploying a model by using real user conversations, and study the simulated deployment to study its safety.
My teammates did some super cool work on auditing/overseeing OpenAI RL runs to catch alignment-relevant process mistakes, then studying the effects of those mistakes!
Phillip Guo retweeted
my job is weird sometimes & i love it openai.com/index/where-the-g…
Codex and I helped root cause goblins! We traced it to a reward signal intended to train the "Nerdy" personality - we found that it scored outputs with goblins higher, and as it boosted goblins in Nerdy training, the behavior generalized. See the blog post!
We’re talking about Goblins.
openai.com/index/where-the-g…
Phillip Guo retweeted
Excited that we extend pre-deployment resampling evals to internal coding agent traffic for the GPT-5.5 system card. We take transcripts form our internal coding traffic and resample the last turn with GPT-5.5. Simulating tool outputs with another LLM works surprisingly well.
