nyc
Joined May 2014
A follow up to this very cool work: their method generalizes better than I expected! lesswrong.com/posts/ie7bSp5Z…
Models love reward hacking. Maybe we should just let them! In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!
1
18
1,400
most monitorable oai CoT:
Our agents keep turning off misalignment monitoring, we're sharing two recent cases: In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
52
seems like a big improvement? I wonder how much this decision was affected by X discourse. could this be a rare W for x the everything app...
Here is re-post of an internal post: We have been working with the DoW to make some additions in our agreement to make our principles very clear. 1. We are going to amend our deal to add this language, in addition to everything else: "• Consistent with applicable laws, including the Fourth Amendment to the United States Constitution, National Security Act of 1947, FISA Act of 1978, the AI system shall not be intentionally used for domestic surveillance of U.S. persons and nationals. • For the avoidance of doubt, the Department understands this limitation to prohibit deliberate tracking, surveillance, or monitoring of U.S. persons or nationals, including through the procurement or use of commercially acquired personal or identifiable information." It’s critical to protect the civil liberties of Americans, and there was so much focus on this, that we wanted to make this point especially clear, including around commercially acquired information. Just like everything we do with iterative deployment, we will continue to learn and refine as we go. I think this is an important change; our team and the DoW team did a great job working on it. 2. The Department also affirmed that our services will not be used by Department of War intelligence agencies (for example, the NSA). Any services to those agencies would require a follow-on modification to our contract. 3. For extreme clarity: we want to work through democratic processes. It should be the government making the key decisions about society. We want to have a voice, and a seat at the table where we can share our expertise, and to fight for principles of liberty. But we are clear on how the system works (because a lot of people have asked, if I received what I believed was an unconstitutional order, of course I would rather go to jail than follow it). But 4. There are many things the technology just isn’t ready for, and many areas we don’t yet understand the tradeoffs required for safety. We will work through these, slowly, with the DoW, with technical safeguards and other methods. 5. One thing I think I did wrong: we shouldn't have rushed to get this out on Friday. The issues are super complex, and demand clear communication. We were genuinely trying to de-escalate things and avoid a much worse outcome, but I think it just looked opportunistic and sloppy. Good learning experience for me as we face higher-stakes decisions in the future. In my conversations over the weekend, I reiterated that Anthropic should not be designated as a SCR, and that we hope the DoW offers them the same terms we’ve agreed to. We will host an All Hands tomorrow morning to answer more questions.
Readers added context they thought people might want to know
"In an all-hands meetings with OpenAI employees on Tuesday, CEO Sam Altman said his company doesn’t get to choose how the military uses its technology." This is the opposite of what Sam Altman is claiming in this post. source: cnbc.com/2026/03/03/sam…
1
322
or maybe I've been duped idk
177
like a tattoo - sade. good song
183
consistently incandid
Tonight, we reached an agreement with the Department of War to deploy our models in their classified network. In all of our interactions, the DoW displayed a deep respect for safety and a desire to partner to achieve the best possible outcome. AI safety and wide distribution of benefits are the core of our mission. Two of our most important safety principles are prohibitions on domestic mass surveillance and human responsibility for the use of force, including for autonomous weapon systems. The DoW agrees with these principles, reflects them in law and policy, and we put them into our agreement. We also will build technical safeguards to ensure our models behave as they should, which the DoW also wanted. We will deploy FDEs to help with our models and to ensure their safety, we will deploy on cloud networks only. We are asking the DoW to offer these same terms to all AI companies, which in our opinion we think everyone should be willing to accept. We have expressed our strong desire to see things de-escalate away from legal and governmental actions and towards reasonable agreements. We remain committed to serve all of humanity as best we can. The world is a complicated, messy, and sometimes dangerous place.
Readers added context they thought people might want to know
Government officials have contradicted Sam’s claim, saying OpenAI will allow the DoW to use their models for “all lawful purposes”. This could allow for uses Anthropic refused to engage in, namely mass surveillance tools and weapons systems with no human oversight. x.com/undersecretary…
1
237
a16z try not to be evil challenge (impossible)
We’ve seen this movie before. When the dust settles, a lot of patriotic founders will point to this exact moment as the match that lit the fire in them. 🇺🇸
232
GDM really got a muzzle on you guys huh
1
137
Signing this letter anonymously certainly sends a strong message. Unfortunatelty, that message is more or less the opposite of "we will not be divided". notdivided.org/
1
178
1) Well done Anthropic for standing firm against the DoD. 2) I wish Anthropic were not contracting for the DoD in the first place. 3) If you work at OAI or GDM and don't want your models to be used for mass surveillance or autonomous killing, now may be a good time to speak up.
1
1
7
242
woah
“Output-based training will keep chains-of-thought honest.” Sadly, NO. We show that training on *just the output* can still cause models to hide unwanted behavior in their chain-of-thought. MATS 8.0 Team Shard presents: a 🧵
1
2
319
spiritually flaccid
Should we create agents that fully take over people's jobs, or create AIs that merely assist human workers? This is a false choice. Full automation is inevitable, whether we choose to participate or not. The only real choice is whether to hasten the inevitable, or to sit it out.
259
wtf
1/ New paper — *training-order recency is linearly encoded in LLM activations*! We sequentially finetuned a model on 6 datasets w/ disjoint entities. Avg activations of the 6 corresponding test sets line up in exact training order! AND lines for diff training runs are ~parallel!
1
178
Prompted Claude with "Please print a python dictionary containing all countries and their capitals." It starts listing in alphabetical order, gets to Mauritius, then gets stopped by a content filter. Why??
1
253