sanjana (@sanjanayed) header
sanjana (@sanjanayed) profile picture

@openai safety!

Berkeley, CA
Joined March 2025
sanjana retweeted
Welcome to the real world. Today we’re introducing FrontierEE Bench, the first benchmark to test frontier models on electrical engineering tasks at a physical lab bench. Real-world engineering benchmarks should involve real hardware. At our lab, models control instrumentation, run experiments on physical hardware, and decide what to test next. The best models scored 36.7% across a range of EE domains including mixed-signal circuits, power systems, & electromechanics.
84
53
13
493
36,833
sanjana retweeted
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards. These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems. openai.com/index/our-first-p…
672
308
347
3,248
1,097,216
I spent 20,000 dollars in API credits having different AI models play each other at Blood on the Clocktower (a Mafia or Werewolf type social deception game). The results were fascinating. Models have gotten much better over the course of this year at creative on-the-spot scheming and bluffing, and also at reasoning as a team to break apart these underground conspiracies. I go through some of the most standout games these AIs played in this video. The new models are getting pretty good. If my friends pulled off some of these moves while we were playing, I’d give them a fist bump. Fwiw, OpenAI’s newer models are much better than Anthropic’s at this game. You can see the full benchmark at the website, and I’ve also made it so that you and your friends can play again the AIs: clocktowerbench.com
189
181
134
3,150
655,072
blows my mind that there is no frozen yogurt shop in San Francisco
91
7
10
338
52,542
sanjana retweeted
We’ve updated the Model Spec, our public description of how we want our models to behave. This revision makes clearer what we want from models as they take on more autonomous work in the world and become capable of helping us in new ways. 🧵 openai.com/index/updated-mod…
89
103
73
824
213,406
🧵 Excited to share the first batch of 6 misalignment reports from OpenAI's new disclosure process for misalignment incidents. We want to be more transparent about the misalignment we see during training, evals and deployment, this is an important step in that direction.
39
79
20
647
70,821
thank you anthropic for sharing! important information for the world here. eg: a freelance threat actor tried using frontier AI to engineer and test a coordinated drone swarm that could select a human target and detonate without human oversight more examples in their report
We're publishing our most detailed threat intelligence report to date. It covers how people tried to misuse Claude—for cyberattacks, influence operations, surveillance, biology, and building weapons—and how we found and stopped them. We disrupted every operation in the report, and used the lessons from them to strengthen our safeguards. Where appropriate, we also shared what we found with authorities and other AI companies. These cases are not typical: we’re highlighting some of the most sophisticated misuse we’ve seen. But they’re especially important to discuss, because they show us where AI misuse is headed, where our safeguards work, and where they need to improve. We’re publishing this report so others can spot the same activity on their own platforms, and so we can give the public a clearer view of how emerging threats develop. Read the report: anthropic.com/threat-intelli…
5
7
146
16,605
sanjana retweeted
Introducing GPT-5.5 A new class of intelligence for real work and powering agents, built to understand complex goals, use tools, check its work, and carry more tasks through to completion. It marks a new way of getting computer work done. Now available in ChatGPT and Codex.
2,436
6,627
3,415
50,653
13,434,198
In the coming weeks we'll be enabling basic telemetry by default in Phoenix over the coming weeks. Our commitment: privacy and security are foundational What we're collecting: Simple, anonymous usage stats Opt out: Set PHOENIX_TELEMETRY_ENABLED=false—no questions asked. github.com/Arize-ai/phoenix/…
3
2
8
309
sanjana retweeted
In case you missed it: we gathered with builders in NYC last night to dive into what really moves the needle after an agent is deployed. A lot of progress starts in production. This means learning from how your agent behaves in the wild and feeding those findings back into your development pipelines. We walked through how continuous evaluation surfaces failures, closes the feedback loop, and turns real insights into a better performing agent. With Arize AX, online evals and real-time tracing makes every production run part of an iterative cycle. This helps teams ship smarter agents that are consistently evolving. Check out the video recording of the session: youtu.be/qF2XQ3WSyrE
4
2
3
491
Spoke at @arizeai’s AI Builder Meetup with @mastra a few weeks back & the talk is now live! I covered the basics of observability + evals, and showed via a Mastra agent how to set up observability, view traces, run evals, and kick-start your iteration cycle. Would love to hear how you're tackling this in your own workflows. If you're looking to add observability + evals, happy to help you get there 🚀 youtu.be/qQGQ9l7ddxE
1
6
1
14
3,914
🚀 New feature Alert: Dataset Splits 🚀 Splits let you define named subsets of your dataset (e.g., train, hard_examples) and filter your experiments to run only on those subsets. Learn more & Check out this walkthrough: ⚪️ Create a split directly in the Phoenix UI ⚪️ Fetching that split down into your code ⚪️ Running an experiment scoped to that subset ⚪️ Inspecting how metrics shift when isolating difficult examples 👉 Full demo + code inside the video below
1
4
9
363
We optimized Claude Code's system prompt - just its prompt - and achieved +10% boost on SWE Bench. No architecture changes, tool improvements, or fine-tuning. Last time we optimized Cline, seeing 15% boosts on SWE Bench, bringing GPT-4.1's accuracy up to Sonnet 4-5, just through prompts. Now, we applied the same algorithm, Prompt Learning, on Claude Code. Just like we did with Cline, we only touched the CLAUDE.md file - optimizing Claude Code purely through custom instructions that are appended to Claude Code's system prompt. See our detailed blog post: arize.com/docs/phoenix/datas… Or just read more on how we did it below 👇
41
102
7
1,806
212,542
sanjana retweeted
We benchmarked Prompt Learning (prompt optimizer) against GEPA and saw similar/better results in a fraction of the time. Since we launched Prompt Learning in July, the most common question we get is: “Prompt Learning or GEPA — which should I use?” So we re-created every GEPA benchmark, measured rollout efficiency, and compared the end-to-end user experience. Results: Prompt Learning achieves similar to GEPA accuracy with far fewer rollouts. We break down the results, and also compare the full end-to-end user experience using these optimizers below 👇
3
7
7
754
Check out our very own @seldo at TS AI Conf!
it takes one to know one tsai builders, that is @seldo from @arizeai and @denise_teng25 from @GradientVC are emceeing tomorrow's TSAI conf livestream starts 9:35am PT tsconf.ai
5
10
4,269
sanjana retweeted
“The purpose of abstracting is not to be vague, but to create a new semantic level in which one can be absolutely precise.” ― Edsger W. Dijkstra @ArizePhoenix
2
9
275
We improved @cline, a popular open-source coding agent, by +15% accuracy on SWE-Bench — without retraining LLMs, changing tools, or modifying Cline's architecture. We achieved this simply through optimizing its ruleset, in ./clinerules — a user defined section for developers to add custom instructions to the system prompt, just like .cursor/rules in Cursor or CLAUDE.md in Claude Code. Using our algorithm, Prompt Learning, we automatically refined these rules across a feedback loop powered by GPT-5. Here’s how we brought GPT-4.1’s performance on SWE-Bench Lite to near state-of-the-art levels — matching Claude Sonnet 4-5 — purely through ruleset optimization. See our more detailed blog post 👉: arize.com/blog/optimizing-co…
35
84
22
1,164
210,588
We released repetitions in Phoenix last week to tackle a core challenge with LLMs: variability. A borderline input can flip classifications from run to run, making it hard to tell if a change is real or just noise. This cookbook shows a full use case you can run yourself: evaluating customer reviews with repetitions. ⚪️ Generate & label a dataset ⚪️ Run repeated evals per example ⚪️ See how results stabilize (or wobble) across runs ⚪️ Catch regressions that single-shot evals would miss It’s a practical way to move from anecdotal “one run” impressions → more reliable model comparisons.
1
3
10
338
Coding agents like Cline don’t always need retraining to get smarter — sometimes all it takes is better prompts. We used Prompt Learning to optimize Cline’s Plan Mode on SWE-bench and saw big gains. To stay true to real developer workflows, we left Cline’s base system prompt untouched and focused on updating its rules instead. Just like .cursor/rules or CLAUDE.md, Cline has a user-defined rules section. We applied Prompt Learning to optimize this rules file and tracked the improvements. Here’s the process + results 👇
1
3
12
919