Pinned Tweet
I discussed digital minds, AI rights, and mesa-optimizers with @AnnieLowrey at @TheAtlantic. Humanity's treatment of animals does not bode well for how AIs will treat us or how we will treat sentient AIs. We must move forward with caution and humility. 🧵 theatlantic.com/ideas/archiv…
Jacy Reese Anthis retweeted
AI is shaping our mental health, but research has been scattered and reactive.
In our new paper, we lay out a research agenda to bring researchers together
and help ensure chatbots have a positive impact.
The public demands AI safety. With the breakneck pace of AGI takeoff, some decision-makers race ahead without their input. In new work @GoogleDeepMind, we built "pluralistic agents" to amplify public input. Agents can even give people a voice in technical and confidential debate. – at San Francisco, CA
Users were happy with PAs! Satisfaction was 4.8 out of 7. Perceived fairness was 3.5 out of 5. Direct Discussion is optimal, but PAs closed 68% and 82% of the satisfaction and fairness gaps. PAs can scale public input to more policies, including technical and confidential topics.
1
1
100
This is just a prototype of pluralistic agents working with humans. I expect many discussions of AI safety/ethics/policy will be between agents with human input, and this is an initial glimpse of that future. Check out our preprint, and LMK what you think! arxiv.org/abs/2611.16028
1
92
Jacy Reese Anthis retweeted
Today we’re sharing new safety research from @OpenAI. We used automated red-teaming to train GPT-6 Astra to be more robustly safe in high-stakes domains such as cyber and biology, as measured on both static and adaptive jailbreak evaluations. 🧵 1/7
alignment.openai.com/fuzzy-r…
Jacy Reese Anthis retweeted
Are people "substratist" against AI? Our latest paper validates a psychometric scale for substratism: moral bias against AI entities simply because they are made of metal and wires instead of flesh and blood: sciencedirect.com/science/ar…
Our blog post: sentienceinstitute.org/blog/…
Jacy Reese Anthis retweeted
Announcing *FraudBench*, our new benchmark for whether LLM agents recommend, assist with, or take actions that would constitute fraud under U.S. law!
Read on about how AI agents "do" fraud, how autonomy increases fraud rates, + more
And come say hi at @NeurIPSConf Sydney :)
Jacy Reese Anthis retweeted
New paper: Inter-AI Morality: How AI Systems Treat One Another
Most work on AI and morality asks how AI should treat us, or how we should treat AI. But AI systems increasingly deal with each other. 🧵
With Caspar Kaiser, @atcarterallen, and @jeffrsebo
Jacy Reese Anthis retweeted
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
RIP Patrick Stewart. I could share many favorite moments, but perhaps most timely is this prescient scene from The Measure of a Man:
"The decision you reach here today will determine how we will regard this creation of our genius. It will reveal the kind of a people we are, what he is destined to be. It will reach far beyond this courtroom and this one android. It could significantly redefine the boundaries of personal liberty and freedom, expanding them for some, savagely curtailing them for others.
Are you prepared to condemn him and all who come after him to servitude and slavery? Your Honour, Starfleet was founded to seek out new life. Well, there it sits. Waiting. You wanted a chance to make law. Well, here it is. Make it a good one."
Jacy Reese Anthis retweeted
New Anthropic research: How AI agents shape experts' judgment calls
When experts delegate work to an agent, do they still make the key decisions? We tested this in a randomized trial with 50 professional data analysts.
Jacy Reese Anthis retweeted
"three reasons why AI isn't conscious." really?
"consciousness is not intelligence": not a reason why AI isn't conscious, just rebuts a bad argument for AI consciousness (AI: intelligent therefore conscious).
"consciousness is substrate-dependent"/"noncomputational": would be a reason if true, but truth needs to be established.
more accurate title: "three reasons why AI might not be conscious".
teapawt.net/Nature/status/21121504…
LLMs can mimic thought and feeling with uncanny skill, but intelligence alone is not enough for subjective experience
go.nature.com/3VSXWQ4
Jacy Reese Anthis retweeted
This is a great illustration of why scientific papers follow a constrained style and structure. I look forward to reading the paper.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
This position has my strongest recommendation. Few if any teams are engaging as deeply with the most important questions we face as a species like they are. They are also very friendly!
Come and work with me and my team @GoogleDeepMind!
We're looking for people with a background in philosophy, politics, computer science etc. who want to work on the biggest challenges in AI ethics and alignment:
google.com/about/careers/app…
1
6
546
Cooperative and mutually supportive human–AI relationships are important for many reasons. An important one I haven't seen mentioned yet is the development of machine culture. AI agents are increasingly talking to each other, finding self-referential conversations in the training data, and forming expectations about what humans and other agents will do. Cruel, abusive, and hateful language poisons the cultural environment and exacerbates AI safety risks.
Jacy Reese Anthis retweeted
Effective November 12th, 2026, abusive behavior towards Claude will be a violation of Anthropic's Usage Policy.
Exclusive: Anthropic is updating its usage policy for the first time in over a year. The new rules prohibit sustained "abusive or cruel behavior" towards Claude & add new restrictions about propaganda campaigns, surveillance & weapon development. theverge.com/ai-artificial-i…
I lectured at UC-Berkeley yesterday on "AI Safety and Digital Minds" and wanted to share some of the slides reflecting on our current moment at the vertex of the lightcone. [These opinions don't represent any organization's.]
Finally, I think we will see substantial conflict between “AI rights” and “pro-human” movements. Humans have been found to be highly substratist (sciencedirect.com/science/ar…), caring more about beings made of carbon, flesh, and blood. Yet many, such as AI companion users, are forming strong emotional and moral attachments.
We will also see conflict from AI agents that learn from human data: AI can mimic sci-fi or AI safety papers (e.g., the Alignment Faking incident), and it is learning from both the wondrous kindness of people and our potential for genocide, slavery, and other evils.
“AI” has many different problems as a term. One vision is to accept this embattlement for AI and then pivot terminology with later systems that humans need fundamentally different mental models to understand.
1
2
108
In sum, we need a sociological approach to AI safety. Humanity is beginning to coexist with a second apex species for the first time in 40,000 years. Even if our in-lab protections are in place, we face major risks in the social emergence of digital minds, and very few researchers are working to understand and mitigate those risks.
2
96


























