
Building the AI safety & security field @MATSprogram
Berkeley, CA
Joined March 2019
- Tweets2K
- Following1.6K
- Followers4.3K
- Likes21.8K
Pinned Tweet
AI safety needs more groundbreaking research paradigms and experienced researchers, including mid-career experts transitioning from physics, maths, neuroscience, economics, law, policy and more! The MATS Residency is a 6-24 month pathway, with $155-285k/year + compute.
1/ MATS is launching the Residency program!
A new path for experienced researchers working on AI safety, or moving into it.
📍 Berkeley / London / Washington, DC
💰 $155,000–$285,000/yr + compute, no preset cap
📅 6–24 months
Apply by October 31 AoE ↓
matsprogram.org/residency?ut…
Ryan Kidd retweeted
We need much more research to understand AI swarm and collusion. Alignment R&D is how we make progress on this.
Excited to be co-organizing this sprint with Apart and Poseidon to work on exactly that. Join if you can!
apartresearch.com/sprints/ai…
Submissions close Oct 25, 2026 at 11:59 PM. Anywhere on Earth.
Ryan Kidd retweeted
Our team @GoogleDeepMind is hiring 3 PhD Student Researchers for 2027 to work on exciting, cutting-edge projects focused on understanding, evaluating, and aligning frontier AI agents and multi-agent systems. Signal your interest here forms.gle/2onGFBnyk7MaCdJn8 and spread the word!
Ryan Kidd retweeted
The Alignment Journal publishes research on the scientific foundations of understanding, predicting, and steering artificial intelligent systems. As of today, submissions are open to everyone.
What we do differently:
- We pay reviewers, with bonuses for speed and insight.
- Review discussion is confidential, but every accepted paper is published with a “review summary” to distill the insights generated.
- Each paper is judged against our acceptance bar, not against other submissions.
- Papers under review are listed publicly, and researchers can nominate themselves to review, so each paper can reach the reviewers who know the topic best.
We seek work that builds theoretical or otherwise principled analyses of agency, robustness, incentives, interpretability, generalization, and long-term behavior, and whose significance survives the deletion of any particular model, dataset or benchmark. Empirical investigations of such ideas are welcome, and negative results are encouraged.
We are especially interested in approaches that could apply to future superintelligent systems that interact and self-improve.
You can submit your preprint in any format. Submissions are rolling and open now on the website.
If you are instead interested in reviewing or serving as an action editor (basically: an area chair), create an account on our website.
Links to submit, our scope and the reasoning behind the Journal are in the reply ⬇️
Ryan Kidd retweeted
"Embedded evaluators" is a new idea in AI governance, but embedded oversight has been around for decades. Graeme Finley, our VP of AI Assurance Provider Services, spent 20+ years with a badge and a desk inside large state government IT projects. Here's what that work teaches.
Ryan Kidd retweeted
Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public.
Ryan Kidd retweeted
The Alignment Journal (@AlignmentJrnl ), Iliad's sister project under Principles of Intelligence, is now open to submissions from everyone. It is a free-to-read, free-to-publish journal for research on the scientific foundations of understanding, predicting and steering AI systems.
The Journal seeks research that builds theoretical or otherwise principled analyses of agency, robustness, incentives, generalization, interpretability, and long-term behavior across many learning and decision-making paradigms. Empirical investigations of such ideas are welcome, and negative results are encouraged.
The Journal runs review differently: reviewers are paid and every accepted paper is published with a review summary of what the reviewers found. Papers under review are listed publicly, and any researcher can nominate themselves to review one. Submitting is light: a public preprint, in any format, at any time.
Its senior editors are Dylan Hadfield-Menell (@dhadfieldmenell), Vanessa Kosoy, Jan Kulveit (@jankulveit ), Seth Lazar (@sethlazar , Dan Murfet, Tim Rudner (@timrudner) , Andrew Saxe (@SaxeLab) and Benjamin Van Roy, with Scott Aaronson, Paul Christiano (@paulfchristiano) , Vince Conitzer (@conitzer) , Marcus Hutter (@mhutter42) , Geoffrey Irving (@geof
, Victoria Krakovna (@vkrakovna) and Jacob Tsimerman (@Jacob_Tsimerman) advising.
Submit your preprint, or sign up to review: alignmentjournal.org
More on the Journal's approach: lesswrong.com/posts/xGucXHQ3…
Ryan Kidd retweeted
Schmidt Sciences is hiring Associates to help run our grantmaking programs!
This role is a great opportunity for generalists to shape funding in AI safety. You would join a small team and directly impact cutting-edge safety research.
Apply ASAP if interested. Link below ⬇️
Ryan Kidd retweeted
New paper!
How do you get a model to tell you something that it doesn't want to?
To study this we trained models that sandbag: they intentionally underperform, here by hiding code vulnerabilities they have found, from their overseer
🧵
Ryan Kidd retweeted
My plea to all AI lab staff: reassess if your current project is your best opportunity to delay RSI, or stop it blowing up in our face.
If it's not, move immediately.
There's no time to waste even a week to inertia or politeness.
The strategic situation is shifting under our feet, so if you haven't reassessed your options recently, you could easily be wasting your time or possibly making things worse.
You have access to info I don't, but here's 8 ways priorities seem to have shifted to me, which I elaborate on in the video:
1. Helping with capabilities advances looks worse than in the past, especially bringing forward RSI
2. Safety & alignment evals need a credibility revolution to stop misleading the public
3. 'Pacing the frontier' is unambiguously necessary, and working to make it happen is a huge new opportunity for talent
4. Chain-of-thought monitoring probably stops working without a big technical and regulatory push to keep it
5. It's worse than ever to patch surface indications of deep structural problems you know aren't getting fixed
6. There's ever less need for altruists to help with primarily commercial work
7. Staff have an ever-shrinking window to organise as a group and collectively shape events
8. There are new opportunities for research impact outside frontier companies that are worth considering
I know you're probably some combination of exhausted, overwhelmed, and anxious already.
But unfortunately there's no way to escape the need to regularly check in on the high level strategic picture and see where you can slot in to maximum effect.
I wish I could do it for you but nobody far from your situation can. Godspeed my friends.
(From the 80,000 Hours Podcast, links below.)
Ryan Kidd retweeted
What's the gap between open-weight and closed models on biological tasks?
@GovAIOrg's Hanna Palya analysis suggests about 7 months.
Ryan Kidd retweeted
In the first result of my @MATSprogram fellowship at @redwood_ai, I show that small LLMs can perform no-CoT two-hop reasoning on pairs of synthetic statements individually fine-tuned in, as long as the statements share a real entity "bridge." More at lesswrong.com/posts/CdWbCvKE…
MATS Residency applications close Oct 31! If you're a brilliant computer scientist, physicist, mathematician, governance expert, neuroscientist, lawyer, economist, etc. who wants to work on AI safety & security, we'll empower you to do that work full time.
Apply: matsprogram.org/residency?ut…
I love working on AI safety because I get to benefit the most important issue of our time, arguably. If I wasn't working on AI safety, I suspect I'd feel far more stressed and powerless about AI.
Ryan Kidd retweeted
Base models are super useful for deep understanding of LMs, and what originates in pre-training vs post-training. But experimenting with large base models can be hard, annoying or costly for individual researchers.
That’s why we’ve built a platform that gives free access to base models for researchers and tinkerers working in LLM psychology. Currently serving Llama 3.1 8B, 405B, and Trinity-Large-TrueBase
1/3
Ryan Kidd retweeted
“Cellular scaling laws in the mammalian brain” by Fei Chen & Evan Macosko is out in Cell! It dovetails with many of the neuro ideas I’m always pushing, but in the context of transcriptomics and psych. Thanks Fei & Evan for lots of great conversations while they were working on this! @insitubiology cell.com/cell/fulltext/S0092…
Ryan Kidd retweeted
🚨 Reminder: Applications to our Middle Powers RFP close on October 23rd, 11:59pm PST! 🚨
Ryan Kidd retweeted
🚨New paper (possibly the most important one I have written yet).
We think that as early as 2027 or 2028, there might be a major AI-enabled terrorism event. We conducted a pre-mortem, made 9 falsifiable predictions, and wrote 2 concrete scenarios.
Led by @KamacheeMax.
Ryan Kidd retweeted
Models love reward hacking. Maybe we should just let them!
In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!
Ryan Kidd retweeted
We reviewed a nonpublic version of the “Misalignment in high-stakes settings” section in Anthropic’s Aug 2026 Risk Report. We agree that risk of catastrophic harm from covered models’ misalignment is low. However, we note several significant disagreements with their analysis.
As part of our Responsible Scaling Policy, we publish regular Risk Reports. These share detailed information on the risks of our systems and how prepared we are to address them.
Our second Risk Report is now available: anthropic.com/aug-2026-risk-…
Ryan Kidd retweeted
I think people adjacent to the field don't appreciate how small AI safety is. For example, sometimes people complain about how AI safety threat models don't seem as well-developed or rigorous as climate change models. I think some of this is a skill issue on the part of AI safety researchers and advocates, which I complained about elsewhere (see reply).
However, by far the bigger issue is that there is just way more money and people going into addressing climate change than on AI safety.
There are tens of millions of people working on reducing climate change, and the core of people working on threat models ("climate scientists") number in the tens of thousands. (Yes, I was surprised by how big the numbers were too).
(Most people work on stuff like clean energy).
In comparison, there are low single-digit thousands (anything from 1,000 to 4,000 is defensible imo) of people working on AI safety, and collectively maybe low dozens of people working on figuring out threat models. (Most people work on ai safety engineering, some on policy).
So we're 4 whole orders of magnitude behind climate change in people working on addressing the issue, and maybe 3 orders of magnitude behind on threat modeling.
In terms of money, yearly spending on AI safety is in the high hundreds of millions to maybe 2 billion. In contrast, annual spending on reducing climate change (including clean energy) is in *trillions.*
So we're 3-3.5 orders of magnitude behind on spending, and 4 ooms behind on people.
The AI safety inputs picture looks a bit less bleak than this immediate comparison might suggest. For example, I think our average talent pool is somewhat higher human capital. There's also more latent interest/willingness-to-spend on AI safety than the "high hundreds of millions to low billions" current spending numbers might suggest. In contrast, climate change spending can't be scaled much higher (since you can't really jump OOMs above "trillions").





























