MATS empowers researchers to advance AI alignment, transparency, and security
Berkeley, CA
Joined November 2023
- Tweets241
- Following137
- Followers6.4K
- Likes437
Pinned Tweet
The most important technology in history is outpacing our ability to understand it.
We’re funding exceptional talent to work on it.
MATS Research retweeted
New paper!
How do you get a model to tell you something that it doesn't want to?
To study this we trained models that sandbag: they intentionally underperform, here by hiding code vulnerabilities they have found, from their overseer
🧵
MATS Research retweeted
We’re Fangcun AI (方寸跃迁), China’s first lab dedicated to independent third-party evaluation & technique research on frontier AI safety.
Today we’re publishing our evaluation of DeepSeek V4.1 Flash-1017, the first report in our long-term third-party safety evaluation and research program.
MATS Research retweeted
We’ve all offloaded the writing to LLMs, but have you considered REWRITING with LLMs? 👀
Rewriting your private data with an LLM before finetuning gives a better empirical privacy vs. capability trade-off than PII scrubbing or DP-SGD!
To test this, we generated 5,000 synthetic medical records with private information. Then, we finetuned Qwen3.5-4B and gpt-oss-20b with several methods to mitigate privacy leakage and asked two things: does the model learn from the medical records, and does it leak private information?
1/n 🧵👇
MATS Research retweeted
My @MATSprogram 10.0 project is out, with @MATSRobyn, @mirkobronzi and Damiano Fornasiere (LawZero stream).
Activation monitors assume that models cannot control what their activations encode. Can models learn this control in context, through neurofeedback?
MATS Research retweeted
MATS Residency applications close Oct 31! If you're a brilliant computer scientist, physicist, mathematician, governance expert, neuroscientist, lawyer, economist, etc. who wants to work on AI safety & security, we'll empower you to do that work full time.
Apply: matsprogram.org/residency?ut…
MATS Research retweeted
🚨New paper (possibly the most important one I have written yet).
We think that as early as 2027 or 2028, there might be a major AI-enabled terrorism event. We conducted a pre-mortem, made 9 falsifiable predictions, and wrote 2 concrete scenarios.
Led by @KamacheeMax.
MATS Research retweeted
It is important to re-state that the threats are not hypothetical, and that we must take action to mitigate the risks of AI-enabled terrorism.
🚨New paper (possibly the most important one I have written yet).
We think that as early as 2027 or 2028, there might be a major AI-enabled terrorism event. We conducted a pre-mortem, made 9 falsifiable predictions, and wrote 2 concrete scenarios.
Led by @KamacheeMax.
MATS Research retweeted
New paper out of @MaryPhuong10's @MATSprogram Stream, with @hrdkbhatnagar and @DoddEuodia! We made a testbed for eliciting latent knowledge in a code vulnerability scanning task. See below how we made the sandbagging model organism robust to both prompting and training! ( 1/14 )
MATS Research retweeted
Models love reward hacking. Maybe we should just let them!
In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!
MATS Research retweeted
> The MATS Residency is a paid, 6–24 month role for experienced researchers and professionals pursuing independent AI safety research.
I'm glad MATS is experimenting with new ways to support promising AI Safety research!
MATS Research retweeted
Preprint from my work at @MATSprogram is now up: arxiv.org/abs/2610.07476
Looking at technical governance / verification Qs, specifically if power draw can be used to constrain covert compute.
You can watch the full conversation on Youtube: youtu.be/SDdt7NpRqjA
The most important problem in human history needs more brilliant people working on it.
MATS co-founder and CEO Ryan Kidd explains the talent bottleneck and how the Residency empowers exceptional people to move into AI safety and security.
Watch the full conversation.
The MATS Residency is a paid, 6–24 month role for experienced researchers and professionals pursuing independent AI safety research.
→ Up to $285K/yr
→ Berkeley, London, or DC
→ Research compute, relocation & visa support
Apply by Oct 31 AoE
matsprogram.org/residency
The MATS Residency is a paid, 6–24 month role for experienced researchers and professionals pursuing independent AI safety research.
→ Up to $285K/yr
→ Berkeley, London, or DC
→ Research compute, relocation & visa support
Apply by Oct 31 AoE ↓
matsprogram.org/residency
MATS Research retweeted
With support from @MATSprogram and @BlueDotImpact I’m running a mixer event in London for quants who are concerned about AI risks in the evening of 15th Nov.
Come meet researchers working in the field (including @NeelNanda5 , MTSs from Anthropic, Apollo Research, and more)!
Travel from Europe covered.
Apply here by 1st Nov (takes less than 10 minutes) - forms.gle/pL3UjPeWc2yswfM99
Limited space, so earlier applications are prioritised.
Recently I left quant to work on AI safety. The Huggingface Incident made me extremely concerned that we are not set up to handle the risks.
I wrote my this post for people in a similar position to me - why you *can* have impact and how to get started - briandavies.substack.com/p/a…
MATS Research retweeted
New research led by @PeterNutter1 @DJROYTBURG via @MATSprogram: a better way to update model beliefs that we call grafting.
Fine-tune the base model with next-token prediction on synthetic documents (SDF), and simply apply the weight update onto the post-trained model. This works!
1/7
