
Evals Researcher @Meta Superintelligence Labs. Past: @Google, @Predibase, CS+Music @Columbia
San Francisco, California
Joined October 2013
- Tweets336
- Following463
- Followers461
- Likes3.1K
Pinned Tweet
Excited to share our new paper, Jagged Judges: Epistemic Stability Under Silence, Pressure, and Persistence
LLMs are increasingly used as judges to evaluate, grade, and analyze model behavior. We usually validate them against expert-vetted examples and call them trustworthy when their accuracy is good enough.
But accuracy doesn't tell us anything about how much conviction the judge actually has.
Would the LLM's verdict survive if it was asked again, questioned, or questioned repeatedly? How does one measure an LLM's confidence, really? 🧵
The writing styles of AI models can change a lot from one version to another.
If you noticed Claude Fable 5.2 feels way punchier than Fable 5.1, Artificial Analysis just quantified the exact personality shift on AA-PersonalityBench across 30+ frontier models.
Compared to Fable 5.1, Anthropic trained Fable 5.2 to aggressively strip out AI writing tells:
- Em-dashes dropped 99%: from 6.41 em-dashes per response in Fable 5.1 down to 0.05 (just 1 em-dash every 20 replies vs 7.4 per reply on GLM-5.3).
- Response length dropped 43%: from 576 words per reply down to 329 words (vs 1,030 words on Qwen3.8-27B).
- Slop index fell from 24 down to 7, and 'Answer First' jumped from the 36th to the 86th percentile.
The catch? Because Fable 5.2 is so terse, its Informativeness score plunged 49 percentile points (from 96 down to 47), and 'Flags Nuances' fell from 98 to 48.
1
41
Justin Zhao retweeted
i keep thinking about this plot from metr.org/blog/2026-11-05-ant…. (A)ECI has a very neat methodology introduced in arxiv.org/abs/2512.00193 that reduces all benchmark scores to a single number. also the two lines look convincing at first glance, suggesting ~2x acceleration...
1/4
Justin Zhao retweeted
absolute fanfic. AI has real, immediate, tangible dangers that we need to focus on, for instance the impact on the job market and making sure random script kiddie can't use it for cyber offense.
But this "oh once it's superintelligent it will take over and ensave us, maybe kill us all!" is pure sci-fi fanfic and distracts from the real (and difficult) issues, which linear probing won't solve.
It's worth repeating the LLMs don't just "decide" to do something and then just "do it" in a void. They generate tokens based on our prompts, only as long as we call `model.forward()`, and on giant clusters with terabytes of weights, that's impossible to "quickly self-replicate".
We also need to regularly remind that the OAI/HF incident was specifically a _guardrail-free_ cyber eval where the model is _instructed_ to hack, to find out how good it is at it.
Some recent personal event made me realize there needs to be more push-back on this doomsday nonsense, and we need to re-focus the discussion on real dangers/issues. I won't argue much in the replies though, not interested in peer-reviewing Terminator fanfic.
I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years).
I’ll say a bit more about each of those claims.
It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: blog.aifutures.org/p/q35-tim…
I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see teapawt.net/dwarkesh_sp/status/209… and youtube.com/watch?v=-Ef0ocEo…
If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. blog.redwoodresearch.org/p/n…)
Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.)
I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon.
Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything.
I don’t know what you should do.
I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon.
(It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.)
I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.
Justin Zhao retweeted
Our first version of Muse Ember 🔥
Ember aims to maximize intelligence per GW of inference to help bring personal superintelligence to everyone.
We would love all the feedback to keep hill-climbing. Please send it our way. We will keep hill-climbing on efficiency, user experience, and better approach to train with internal frontier!
More to come 🫡
Really cool work! I've started feeling that the best agent isn't the most capable one, but the one that keeps you engaged and critical about your work.
If an agent is too perfect and smart, it's too easy to default to endless chains of "sounds good, keep going" and lose track of progress.
There should be a mode where once every 10-20 turns, the agent gives a deliberately worse response. And if you as the agent's operator miss too many of these, the system tells you that you need to pay more attention.
Replying to @AlexTamkin
What can we do about this?
I think we need a much better understanding of how agents shape people's decisions, in and out of work.
I also think there are ways we can design both model behavior and AI products to better support human agency.
1
118
Justin Zhao retweeted
Happy Pythagorean Theorem Day!
10/24/26
10² + 24² = 26²
This is the last one we’ll experience in our lifetime.
Unit tests written by coding agents are actually so bad. PSA tell your agents to read up on
- testing.googleblog.com/2013/…
- testing.googleblog.com/2017/…
68
This is how you train dogs to stop barking.
First, teach them how to bark on command by rewarding them when they bark in relation to a specific word or gesture. Make it a game!
Once that’s mastered, teaching them a different command for how to stay quiet is a lot easier.
Models love reward hacking. Maybe we should just let them!
In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!
3
267
Justin Zhao retweeted
evals are the cornerstone for both selling data to frontier labs and building your own self-improving models. building high-end evals is extremely valuable and important
Justin Zhao retweeted
Muse Spark Open was literally as good as the best model we released just two months ago.
Now it’s open-weight. Enjoy!
research.meta.ai/blog/muse-s…
The relative value of human understanding depends on multiple factors.
Even in software, some code is worth deep, human ownership, while other code is more valuable simply because it’s usable and it works, whether anyone reads it or not.
AI discoveries can be the same. Deep human understanding is obviously important for being informed about future direction, but it’s totally possible for the immediate value of a discovery to outweigh that.
This is a slide from Terence Tao's 'Math 2.0' public lecture yesterday evening at Caltech. I am highly confident that, if the cure actually worked, almost any Stage 4 cancer patient would tell you that human comprehension of its mechanism is completely irrelevant to them.
5
204
Every idea is derivative. Pre-existence means nothing.
Applying the idea at the right time and marketing/communicating it effectively is way more important.
Fun fact about (some) AI researchers / academics: the more senior you get, the more convinced you become that every new AI idea is something you already invented 20 years ago. Or, even better: wait until something actually works and then try convincing everyone that this is exactly what you’ve been saying all along.
I was at a conference where a speaker was presenting their research. One academic stood up and said: what you’re doing is actually something I did a few years ago. The speaker was very polite and said thank you for pointing this out.
Then a second researcher stood up and said, well I already did it this 10 years ago, you should cite my work. Then a third person said, I already did all of this about 20 years ago.
It was the funniest thing ever.
1
73
One of the most passionate data visualizers out there!
My friend @justinxzhao made this very cool app inspired by one of my tennis data visualizations 🎾📈
You can see the points from victory for both players at every point of any 2026 US Open match!
🔗: justinxzhao.com/demos/points…
4
187




























