sanjana retweeted
Welcome to the real world. Today we’re introducing FrontierEE Bench, the first benchmark to test frontier models on electrical engineering tasks at a physical lab bench.
Real-world engineering benchmarks should involve real hardware. At our lab, models control instrumentation, run experiments on physical hardware, and decide what to test next.
The best models scored 36.7% across a range of EE domains including mixed-signal circuits, power systems, & electromechanics.
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards.
These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems.
openai.com/index/our-first-p…
tech with Hannah strikes again
Replying to @sanjanayed @OpenAI
hit the download canon powershot g7x mark ii photos button in the passenger seat of the prius otw home 😋😋
also for vis no action needed @sanjanayed
1
4
498
sanjana retweeted
I spent 20,000 dollars in API credits having different AI models play each other at Blood on the Clocktower (a Mafia or Werewolf type social deception game).
The results were fascinating. Models have gotten much better over the course of this year at creative on-the-spot scheming and bluffing, and also at reasoning as a team to break apart these underground conspiracies.
I go through some of the most standout games these AIs played in this video. The new models are getting pretty good. If my friends pulled off some of these moves while we were playing, I’d give them a fist bump.
Fwiw, OpenAI’s newer models are much better than Anthropic’s at this game.
You can see the full benchmark at the website, and I’ve also made it so that you and your friends can play again the AIs: clocktowerbench.com
sanjana retweeted
We’ve updated the Model Spec, our public description of how we want our models to behave. This revision makes clearer what we want from models as they take on more autonomous work in the world and become capable of helping us in new ways. 🧵
openai.com/index/updated-mod…
sanjana retweeted
Some details on what our monitoring system currently looks like openai.com/index/how-we-dete…
sanjana retweeted
🧵 Excited to share the first batch of 6 misalignment reports from OpenAI's new disclosure process for misalignment incidents. We want to be more transparent about the misalignment we see during training, evals and deployment, this is an important step in that direction.
sanjana retweeted
thank you anthropic for sharing! important information for the world here.
eg: a freelance threat actor tried using frontier AI to engineer and test a coordinated drone swarm that could select a human target and detonate without human oversight
more examples in their report
We're publishing our most detailed threat intelligence report to date.
It covers how people tried to misuse Claude—for cyberattacks, influence operations, surveillance, biology, and building weapons—and how we found and stopped them.
We disrupted every operation in the report, and used the lessons from them to strengthen our safeguards. Where appropriate, we also shared what we found with authorities and other AI companies.
These cases are not typical: we’re highlighting some of the most sophisticated misuse we’ve seen. But they’re especially important to discuss, because they show us where AI misuse is headed, where our safeguards work, and where they need to improve.
We’re publishing this report so others can spot the same activity on their own platforms, and so we can give the public a clearer view of how emerging threats develop.
Read the report: anthropic.com/threat-intelli…
the super important ground breaking thing that happened at work today was that they stocked up cheez its in the micro kitchen
6
232
Introducing GPT-5.5
A new class of intelligence for real work and powering agents, built to understand complex goals, use tools, check its work, and carry more tasks through to completion. It marks a new way of getting computer work done.
Now available in ChatGPT and Codex.
sanjana retweeted
In the coming weeks we'll be enabling basic telemetry by default in Phoenix over the coming weeks.
Our commitment: privacy and security are foundational
What we're collecting: Simple, anonymous usage stats
Opt out: Set PHOENIX_TELEMETRY_ENABLED=false—no questions asked.
github.com/Arize-ai/phoenix/…
sanjana retweeted
In case you missed it: we gathered with builders in NYC last night to dive into what really moves the needle after an agent is deployed.
A lot of progress starts in production. This means learning from how your agent behaves in the wild and feeding those findings back into your development pipelines.
We walked through how continuous evaluation surfaces failures, closes the feedback loop, and turns real insights into a better performing agent.
With Arize AX, online evals and real-time tracing makes every production run part of an iterative cycle. This helps teams ship smarter agents that are consistently evolving.
Check out the video recording of the session: youtu.be/qF2XQ3WSyrE



















