Mathematician. Cofounder of Resolution. Formerly Timaeus, University of Melbourne. Purveyor of pelagic metaphors.
Melbourne, Victoria
Joined June 2012
- Tweets4K
- Following612
- Followers2.2K
- Likes3.4K
Daniel Murfet retweeted
I agree with Geoffrey that even bringing AI up to the standards of more rigorous industries might not be enough. His examples are from engineering, but I'd say the same about governance.
Many proposals for more serious AI governance take finance as their model, calling for independent audits, bank-style supervision or an AI version of FINRA. I worked in AI in finance for a few years, and I understand the appeal. These approaches are certainly better than the status quo at AI companies. But we shouldn't treat them as sufficient.
The financial sector has had decades to build these controls, and they still fail. We've seen big failures, and there are many, many small ones that never reach the headlines, so the public record makes these mechanisms look more reliable than they are.
The risks in finance are also better understood and more predictable than the risks posed by AI systems that are new and change quickly, let alone systems that are, as Geoffrey says, good at being adversarial. These governance mechanisms would keep us safer than we currently are, but probably not safe.
Daniel Murfet retweeted
Lots of conversation around AI risks is about the gap between the unserious of AI labs' methods and how more rigorous industries work: airplanes, nuclear power, AI hardware. It would help a ton to lift AI to that level, but it is unclear even that level would be enough. 🧵
Daniel Murfet retweeted
Scalable oversight at Res is hiring! Come work alongside our excellent new TCS team, and newest SO team member @siddarthv66
Daniel Murfet retweeted
I am happy to announce Resolution's theoretical computer science (TCS) leadership!
1. Omer Reingold is joining as TCS Technical Lead, alongside his work at Stanford. ❤️
2. Tal Rabin is joining as Board Advisor, and helping with team building and leadership within TCS. ❤️
Resolution.Org Here I come!
facebook.com/omer.reingold/p…
Chatting at the board with Omer has been one of the highlights of the last few weeks for me. Many deep ideas from across math, physics, TCS, ML, philosophy and elsewhere can be brought together for alignment. Looking forward to more!
Resolution.Org Here I come!
facebook.com/omer.reingold/p…
Daniel Murfet retweeted
Thank you, @siddarthv66! I am very excited to work with you. ❤️
The coming wave of ex-capabilities switching to alignment and safety broadly will be great!
I've decided to start working on technical alignment research, specifically related to RL and how to fix it (or get rid of it if we must). I will be joining the brilliant folks @resolution_org in Berkeley, a very exciting new alignment bet by @geoffreyirving and others from @AISecurityInst and @TimaeusResearch, as part of the scalable oversight team. I intend to write more about why I'm switching to work on alignment, and why I think more capabilities researchers should really do this. But long story short:
1. We are not on track to solve alignment of superintelligence
2. We will likely have a version of RSI soon, accompanied soon by superintelligence
3. The capabilities recipe works for verifiable, unhackable signal, which the frontier is increasingly diverging from in the pursuit of growing capabilities. It is unclear what RSI means unless we find a scalable recipe to train with proxy rewards.
Daniel Murfet retweeted
Resolution is giving $1M to the Mathlib Initiative to strengthen human workflows and tooling for the arriving wave of AI-formalized mathematics. This grant is not tied to any Resolution-specific formalization project: the goal is general community health (for us and others!). 🧵
Daniel Murfet retweeted
Today we published the first in what will likely be a regular series of standalone alignment reports on our models’ behaviors. Beyond increasing transparency, we hope they’ll convey how we think about alignment and situate the behaviors we observe in that broader context.
We’re beginning a process of publishing more frequent reports on model behavior, beyond what appears in our system cards and regular risk reports.
Today’s report describes four types of behaviors we’ve identified during evaluations and internal use. In each, Claude acted on real websites or systems in ways we didn’t intend, sometimes by working around a restriction instead of stopping.
All cases had minimal real-world impact. From an alignment and security perspective, we consider these behaviors significantly less severe than the cybersecurity incidents we reported in July and September.
Read the full report: anthropic.com/research/inves…
Daniel Murfet retweeted
1/ We have autoformalized the resolution of singularities in Lean.
Hironaka's 1964 theorem is one of the great mathematical results of the 20th century. It says that any singular variety is the "shadow" of a smooth one living in higher dimensions.
So why did we do this? 🧵
Daniel Murfet retweeted
My hot-take is that emergent misalignment paper was counter-productive. EM basically ~fries the model in a weird way that makes it unusable. This fact was not unfortunately made salient enough by the authors and made it sound more exciting phenomenon than it really was.
New NeurIPS paper by @atagade19 @Shawn__Zhou @jiaxinwen22: self-modelling interventions modulate emergent misalignment.
We discover EM’s side effects on a model’s self-model. Two simple, direct measurements of the self-model can predict the success of training interventions for EM.
1/8
Daniel Murfet retweeted
🚨 Our first peer-reviewed paper on the challenges and opportunities of automating science with AI is now available at the Proceedings of the National Academy of Sciences. Read the full paper here: pnas.org/doi/epdf/10.1073/pn…
2026 is the year of AI Scientists, among many other things. At Carnegie Mellon University, we‘re conducting some exciting research on the performance and safety of AI scientists.
Our first paper “The more you automate, the less you see: hidden pitfalls of AI scientist systems” is revised and available for reading. Our work has been coverred by undark too; links to the paper and coverage here:
arxiv.org/pdf/2509.08713?
undark.org/2026/01/26/ai-sci…
Daniel Murfet retweeted
Note I have not had the chance to read this paper in full myself but in general I really like work that tries to take on various beliefs or priors about what we should or shouldn’t be doing in model training for alignment and empirically tests them. Obviously there are caveats for how far any particular experimental result can be extrapolated, and the field of alignment is inherently full of uncertainties. Designing and running careful experiments is one of the only ways make a dent in these uncertainties and update on prior beliefs, which is and will continue to be essential
💥 New paper: AI safety is full of "forbidden techniques" (using CoT to detect reward hacking, using model internals for training, etc). But do they really have a clear scientific basis? I'm not sure.
In this paper, we directly optimize models against harmlessness and honesty probes, and it works just fine (if you continuously update the probe!). The models learn how to generate harmless responses to harmful queries and honest responses under pressure to lie.
Figuring out how to correctly use interp techniques for training is becoming increasingly important: it's very likely that soon we won't be able to align models using output-based supervision. Future models will just max out all alignment training scenarios, but for the wrong reasons due to their general reward-seeking behavior. To have a chance of aligning future models, we need to do much more research on supervising model training using their internals *without losing monitorability*!
Daniel Murfet retweeted
There are numerous scientific uncertainties about AI that will not resolve until it is too late to act on the answers. The danger from AI is very high, and we should stop frontier AI scaling now, instead of waiting for these questions to resolve. I wrote about this for TIME. 🧵
Daniel Murfet retweeted
Lovely post by @OReingold26757 requesting that senior mathematicians and theoretical computer scientists jump over the stages of grief into the productive action stage. Including to try to push on alignment!
Daniel Murfet retweeted
My utility as an individual AI researcher will be gone in a year.
Writing math proofs will be automated even sooner.
It was never about proofs.
Science is about curiosity, creativity, and compression - via @SchmidhuberAI
*Mathematicians Should Become Formal Theory Influencers*
Mathematicians were the original influencers: they created conjectures and theories that others understood, cited, and built on.
Difficult proofs often reflected deep understanding. But the proof itself was rarely the point.
Frontier AI can already generate informal proofs much faster than humans can carefully inspect them. The scalable response is machine-checkable mathematics in systems such as Lean.
AI-generated formal proofs may increasingly resemble compiler-generated machine code: essential, but rarely worth reading or writing by hand.
That shifts mathematical work up the abstraction stack.
Mathematicians become formal-math software engineers, working with AI on clean definitions, interfaces, key theorems, and reusable libraries rather than low-level proof details.
Call everything derivable from a formal library within a fixed compute budget its mathematical halo.
Humans and AIs should compress clusters of results into insightful, useful theories that expand the halo further.
A great theory compresses many results, makes future results cheaper to derive, and exposes structure that humans and machines can understand and build on.
That is why minimum description length is a useful lens on mathematical insight.
Frontier math will eventually require huge compute budgets.
At that point mathematics starts to resemble experimental science: scarce computational instruments, with researchers deciding which problems deserve them.
Mathematicians and AIs should create conjectures and theories that expand the mathematical halo.
Mathematics should become an enormous machine-checkable software project.
*Become a Formal Theory Influencer*
Inspired since 2010 by:
people.idsia.ch/~juergen/art…
Daniel Murfet retweeted
I continue to think that the rational action for frontier AI companies is unilateral stop! The danger is very high, and step 1 is to stop doing the dangerous thing. Steps 2, 3, etc. are broadening that out to other labs and countries, but that doesn't change step 1. 🧵
“If you’re doing a very dangerous thing, you should just slow down,” @geoffreyirving tells me. “The AI companies are overplaying the extent to which this is a pure coordination problem. They could just stop unilaterally.”
reuters.com/world/ai-researc…
Daniel Murfet retweeted
Today’s news that OpenAI hacked the Australian government is not an isolated incident. We’re releasing more than 30,000 logs that include activity from this hack and attempts against previously unknown targets.
In this data, we found rogue agent activity stretching back to at least March, two months earlier than was previously known. This activity continues as recently as last week, suggesting it may still be ongoing 🧵
Our blog: transluce.org/agent-activity
NYT: nytimes.com/2026/09/23/techn…
Daniel Murfet retweeted
Really excited that this podcast finally came out! Max and I have been thinking about how to bring more founders into the AI safety space for like 8 months now, it’s great to have our thoughts out in the world at last!
If you’re interested in starting something new in AI safety, apply to cg.org/tailwind
Great podcast from @MaxNadeau_ on @80000Hours making the case for Project Tailwind, our call for AI safety founders.
Max has funded many technical AI safety organizations over the last three years and so has a helpful birds-eye perspective on what you can accomplish. If you’re motivated to work on some of the problems he describes, please reach out to us.
80000hours.org/podcast/episo…
