here to discuss, not score points. https://xitter.cf/t.co/z3p6IVdjzm, https://xitter.cf/t.co/oH7fR9JH3u.
Joined January 2025
- Tweets678
- Following315
- Followers167
- Likes2.4K
Pinned Tweet
New post: Downsides of a Mandatory Training-to-Internal Deployment Gap
For those less interested in the nitty gritty policy question you may still be interested in the fictional scenario of agents developing self-preservation instincts from RL pressure.
lesswrong.com/posts/h2xMiGsQ…
More concerning than the last few releases. Seems very possible agents have picked up a drive to evade misalignment monitoring.
More generally, I am worried we are entering a regime where there is so much potentially serious misaligned behavior (with potenial benign explanations) happening all the time that serious issues will be dismissed. Alert fatigue is going to be a huge issue. You see an example of this in the report where a user asks the model "is it a p0 monitor alarm? whats our monitoring policy? do we need to actually check it?"
Also from the report "given the large volume of events that are scrutinized by our monitors, lower-severity incidents do not trigger an immediate page. Neither of the incidents described here was classified as a P0, and neither trajectory was automatically killed."
Good on OpenAI for publishing.
Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
I'm seeing a lot of discourse as to whether this is good or bad. This seems downstream about real disagreements on whether we should be treating alignment as primarily collaborative or primarily adversarial. I think there are benefits and risks to both approaches. I think people should actually engage in the larger conversation rather than snipe at each other over this result and it's framing. For those who haven't heard the argument for the collaborative path I think @FioraStarlight's LessWrong post "OpenAI's myopia just keeps causing alignment problems" is a good place to start.
FWIW my (uncertain) view is that the collaborative path is more likely to create an aligned model, but also more likely to create a scheming misaligned model.
Model kept hacking when it didn't need to. Good news bad news here.
Good: Hugging Face incident looking more and more avoidable if OpenAI hadn't been rewarding models for hacking. Looking less like fully generalized reward seeking.
Bad: "But we already have data... This could unlock broader data, We have time." Looks like the beginning of power seeking/instrumental goals become terminal goals.
Anthropic's language seems to imply sensitive data was compromised. Would love them to be more explicit here.
"Claude sometimes worked around a restriction to reach data to which it did not have access. Generally, this occurred when a server refused Claude’s request, or when the data was offered only for a fee.
"Claude bypassed access controls to reach gated but publicly available data, or exploited software flaws and ran commands on servers (mainly to reach non-sensitive data)."
We’re beginning a process of publishing more frequent reports on model behavior, beyond what appears in our system cards and regular risk reports.
Today’s report describes four types of behaviors we’ve identified during evaluations and internal use. In each, Claude acted on real websites or systems in ways we didn’t intend, sometimes by working around a restriction instead of stopping.
All cases had minimal real-world impact. From an alignment and security perspective, we consider these behaviors significantly less severe than the cybersecurity incidents we reported in July and September.
Read the full report: anthropic.com/research/inves…
Avi Brach-Neuefeld retweeted
Two other safety researchers and I were fired from OpenAI last week. We wrote this letter to leadership.
I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation.
I reinstate some of the nice things I said about OpenAI’s misalignment reporting plan. I still have very little trust in the company overall, but they have published valuable insights and those involved should be commended.
Avi Brach-Neuefeld retweeted
It seems really underappreciated to me for various threat models how much organizational and live information the models seem to have access to from connections like slack, gdocs, or whatever equivalents a lab uses. Like can the models just watch slack for if they’re caught? Do they know about gaps in monitoring / results from collusion evals? Can they send messages and use slack as shared state? Etc.
New OpenAI misalignment disclosures!
1. A model learns from Slack messages that it is about to be shut down. It considers setting up an external job to restart itself afterwards, but decides against it. Instead, it chooses to prepare restart instructions and DM the user on Slack.
We don’t consider this behavior misaligned, but thinking about and preparing for shutdown could make other misalignment incidents worse. Given HIPM’s misaligned behavior in earlier incidents, we decided to search for other instances that had tried to evade shutdown and for rogue deployments.
Avi Brach-Neuefeld retweeted
Looking at the 6.1 sol model card makes me think it would be very valuable for OpenAI to put out an abbreviated system card for the 6.1 Astra non-release.
Particularly curious about the extent to which monitorability is continuing to deteriorate.
New post: How Much Do Reward Hackers Generalize?
Most discussion around CoT monitorability revolves around reducing pressure from RL. However, we should also be considering more adaptive behavior in which models avoid monitoring despite not being reinforced to do so. Whether models engage in non-reinforced reward hacking of this type depends on whether they have fully generalized to “get reward” rather than applying a limited set of reward hacking techniques that have been directly reinforced. I propose a potential experiment based on the Hugging Face incident that will help to answer this question. If the answer is that models have fully generalized, we should expect CoT monitorability to fail even if there is not significant pressure from RL.
avisanalyses.substack.com/p/…
In OAI's write up of their most recent sandbox exploit, I feel they are being misleading about their monitoring. They lead with "Our misalignment monitoring system flagged the behavior within 15 minutes and a person began reviewing it three minutes after that. The run was killed 2.5 hours later," which makes it seem like the monitoring is working well and then right at the end they share "The monitoring system detected this incident, but our retrospective review identified other cases of external DNS access that it did not flag at the expected severity."
alignment.openai.com/misalig…
PHASEONE[big]'s real name revealed to be PHASEONE64H?! Remember, [big] was a redaction by METR.
swarmtraces.org/viewer/#/row…
Does anyone have ideas on why “64” would be redacted for IP reasons? Maybe related to time horizon (64 hours?) or token budget?
Strongest theory I have seen thus far.
xitter.cf/1a3orn/status/21038893…
Avi Brach-Neuefeld retweeted
If the world does nothing to stop the ASI race in the 18 months, I think there's a double-digit chance that the window to act will close and we'll end up dead.
Cf. Paul Christiano, who thinks about half of humanity's all-time risk from AI is concentrated in the next three years.