Jay Creagh retweeted
Taking stock of the last three years of development in AI technology, Nicolas D. Villarreal argues that Artificial General Intelligence is a genuine possibility with major implications for the future of class struggle on Earth.
cosmonautmag.com/2026/10/mar…
Jay Creagh retweeted
This is interesting work, but I don't think it's a significant update to the big picture of interpretability.
I'm on board with the broad idea of doing exploratory interpretability with neurons, e.g. neuron-based attribution graphs, for pragmatic problems like why did a model do a sketchy thing? I think this work provides some reasonable evidence, in addition to good earlier work from
Transluce, that this might actually work for practical purposes, and has some advantages over the standard alternatives like SAEs/transcoders. I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc.
I expect that a neuron-based attribution graph will still have significant error and be missing a lot just like transcoders, once you prune them down to the most important components, even if they solve error terms. Which is fine for pragmatic purposes, and may suffice for getting the gist of what's going on, but I remain pessimistic about really fully understanding things and being confident there's nothing we're missing. I do think that using neurons is just vastly easier to, for example, do on a frontier model, which is great. I'm also excited to see more innovation in using agents to interpret neurons. I feel like this is an area the field has somewhat neglected, and where there's probably a very high ceiling on how far things can go. The SwiGLU double negative thing is also a useful point.
I still think superposition is real and this work doesn’t really affect my cruxes. Most importantly, it seems highly likely to be true that the model can represent more concepts in the residual stream than it has dimensions, ie representational superposition, which I think the authors agree with me on. For neuron/computational superposition, it seems to me like language models have far more facts and memorized things in them than they have neurons. And that superposition is an efficient way to pack in more facts than you have neurons by representing them as linear combinations of neurons, especially for arbitrary unstructured values. Facts/memorized content is largely an unstructured database of key-value pairs IMO and I don’t see how this is compatible with a monosemantic view of neurons. I expect that weak neuron activations contain lots of semantic superposition-y info, which, if ablated, would meaningfully damage network performance. As I understand the work, it does not properly address what's going on with weak neuron activations in LLMs (I am skeptical of how much vision results will transfer, eg I expect small image models memorise far less). This picture is compatible with the highest activations of a neuron being mono semantic (though IMO the post defines monosemantic too loosely, I think I’d call a fair few of those polysemantic). It’s also plausible that the most important concepts are likely to correspond to a neuron (or a handful), which would be great, and may suffice for practical purposes!
Similarly, my impression is that SAEs generally don't find things that are a
single neuron, or even very sparse in the neuron basis, and are often a better basis for certain tasks than the neuron basis, though the evidence base is kinda weak here since we largely focus on residual stream or layer output SAEs, I’d love to see work on this in Goodfire’s forthcoming paper! I also think residual stream SAEs are extremely useful, as they give a good bottleneck of the model’s information at this token and layer, and I don’t see a good way to replicate them in the neuron framework.
And kudos to Nick for updating a bunch of his historical vision work! Revisiting old results with modern tools and more rigour seems great.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Jay Creagh retweeted
New essay: To keep human judgment, we need AI courts.
AI-generated filings are starting to overwhelm courts around the world.
The most popular response I’ve heard is to fight AI with AI: If we can just give judges the right tools and training, AI will help them keep pace.
But this strikes me as the worst of both worlds.
Because we want human judges in the loop, we don’t get the speed and scale benefits of AI. Yet because we expect them to resolve a massive number of cases quickly, judges (or their clerks) may end up leaning on AI more than they mean to or should.
This essay offers another way: a parallel AI adjudication track.
An AI track — inspired by AI use in private arbitration but formally part of the court system like magistrate judges — would let courts be explicit about how human judgment is allocated.
Done right, this would allow courts to process large volumes of cases while ensuring human judges devote their full attention to the cases where their judgment is needed most.
Read more in @PaxMachinaMag: paxmachina.ai/to-keep-human-…
Jay Creagh retweeted
Law enforcers already have authority to charge companies and their CEOs for creating and releasing dangerous, unvetted, or defective products. We shouldn’t let discussions about new legal regimes distract from the fact that there’s no AI exemption from laws already on the books — a point @FTC emphasized repeatedly during my tenure.
1. There is an extensive set of laws that govern dangerous and defective products. For example, releasing unvetted AI models or agents can violate consumer protection laws. Shipping flawed AI tools without implementing adequate measures to detect and stop rogue or defective AI agents can be an “unfair or deceptive” act or practice under the FTC Act (and analogous state laws). And some state AGs are already exploring holding AI firms and their CEOs criminally liable when their models participate in criminal activity.
2. Existing laws also prohibit “unfair methods of competition.” This covers instances where AI firms appropriate the competitively sensitive information of their customers, including through tracking their use of various tools. It can also cover instances where firms pursue dangerous behavior, aware that doing so may compel rivals to do the same.
As the Supreme Court has noted: “A method of competition which casts upon one's competitors the burden of the loss of business unless they will descend to a practice which they are under a powerful moral compulsion not to adopt, even though it is not criminal, was thought to involve the kind of unfairness at which the [unfair methods of competition] statute was aimed."
3. The highly concentrated and interconnected structure of these markets could be creating major risks and conflicts of interest. We had started investigating these partnerships and cross-investments across the stack (and released a preliminarily overview of some findings: ftc.gov/news-events/news/pre…).
Both federal and state enforcers should be scrutinizing these opaque relationships and inter-dependencies. We are already seeing how these relationships could undermine accountability. For example, OpenAI could face liability given the Hugging Face incident, but Hugging Face being bought up by Nvidia means that we’re unlikely to see it file a lawsuit over this — given Nvidia’s strong incentive to see OpenAI continue full speed ahead.
4. As AI tools dramatically change the landscape of cybersecurity risks and hacks, all businesses should be doubling down on having core security protections in place. Firms that fail to invest in adequate data security measures or fix known vulnerabilities can also be breaking the law. A recent analysis showed that around 1/3 of Fortune 100 companies do not even have a way to notify them about security issues. During my @FTC tenure, we sued firms for poor data security practices and held CEOs liable when they were personally responsible.
this.weekinsecurity.com/doze…
ftc.gov/news-events/news/pre…
5. As policymakers consider new legal regimes, we should be looking to lessons from prior efforts to govern major sectors, such as banking and other networks, platforms, and utilities. Tools like structural separations, nondiscrimination, and supervision could be key, and there’s a rich history of what works and what doesn’t. But we can and must pursue any new efforts alongside enforcing existing laws.
Jay Creagh retweeted
We for sure need more third party evaluators and embedded assessors for frontier AI companies.
If you want to set one up, @GovAIOrg would be keen to incubate.
You can apply for our entrepreneur in residence program here: governance.ai/post/eir. You'll have a year of runway, lots of support and input, freedom to experiment, and ~$150k in seed funding.
We've been working on these topics for years at this point. See e.g.:
- Frontier AI Auditing: arxiv.org/abs/2601.11699
- In-House Evaluation is not Enough: arxiv.org/abs/2503.16861
- Towards Publicly Accountable Frontier LLMs (2023) arxiv.org/abs/2311.14711
- Frontier AI Regulation (2023) arxiv.org/abs/2307.03718
Would love to help you make some of these ideas reality!
Jay Creagh retweeted
My journey to develop AGI spans 25 yrs, including 10+ yrs thinking about technical & societal perspectives at Google DeepMind.
AGI is on the horizon - we need deeper understanding of its implications. To help, we've created the DeepMind Institute.
Jay Creagh retweeted
We're excited to welcome Bilal to the BlueDot team. He's joining us to build the institution that will make it easy for great people to gain context, upskill and get to work on AI safety in a fast and effective way.
I recently resigned from Google DeepMind, where I worked on AGI safety and alignment research. At Google, I witnessed AI development first hand. I too am extremely concerned by the default trajectory of this technology. I earnestly believe that AI has the potential to kill us all, and that we might be running out of time to avoid this outcome.
The pace of AI progress in the past few years has been staggering. When I first started working on AI in early 2022, AIs were amusingly useless. Just four years on, AI agent swarms from OpenAI are cracking famous century-old math problems and, more worryingly, escaping the control of OpenAI and autonomously hacking into the third-party company HuggingFace, against anyone's wishes.
Things will only get crazier: I think it's possible that the AI companies might, in the next few years, succeed in building superintelligent AI systems that far exceed human capabilities in every domain. I am not confident that these AI systems will do what we want. In particular, misaligned superintelligences may, much like the rogue AI agents involved in the HuggingFace incident, escape our control and take dangerous actions that may result in the permanent disempowerment or death of humanity. Alignment is the problem of preventing this, and is both difficult and unsolved. Our present understanding of how to train AI systems that deeply want what we want is extremely rudimentary. Worse, we are not on track to solve alignment in time: frontier AI capabilities are improving much faster than our understanding of AI alignment.
I am optimistic that navigating AI safely is possible. In order to do so, we need to coordinate to avoid this manic race between AI companies. We need to pace AI development to a speed that society can handle, where emerging risks can be addressed before extreme harm is realised. We need much more transparency into AI development to ensure that AI companies are not imposing unacceptable levels of risk on us all.
More broadly, we need many more people thinking carefully about the problem of making AI go well. It is, in my view, the most important problem facing humanity this century, and the stakes are immense. I'm very directly working on this next: I want to help people interested in working on mitigating catastrophic AI threats do the most effective work that they can. I think many people from many backgrounds in many roles have a part to play.
Jay Creagh retweeted
MATS 10.0 Symposium spotlight talks just dropped! If you're interested in AI safety & security, here is a taste of the research MATS helps support!
youtube.com/playlist?list=PL…
Jay Creagh retweeted
AI could make government much busier without making it better.
@JamesBroughel argues that if the incentives driving agencies stay the same, more AI in government bureaucracy may just lead to more polished justifications for choices already made.
A response to @nickacaputo
Jay Creagh retweeted
Fuck it just put a 5xmultiplier on the YouTube view count. Count every second of watch time as a trillion views. Advertisers will love it. Start using mathematical notation to keep track. Embrace the viewgularity. Yeah my channel pulls 6e3.74 ↑↑ 9e17.3 views a month what of it.
People do not coordinate only through broad legal rules and prices. Hayek emphasized abstract rules that allow people to coordinate like property, contract, trade, and competition. But Lachmann also emphasized the practical secondary institutions people orient their plans around, like banks, standardized contracts, product categories, and so on.
In software, an abstraction boundary is an interface that hides complexity hiding beneath. In this 2005 paper (papers.ssrn.com/sol3/papers.…), Miller and Tulloh explain that you can apply this concept to markets too. Consider a post office: there's an abstract boundary that separates why the customer wants to mail something (which the postman doesn't need to know); what the shared transaction is (all the recognizable steps and commitments involved in sending mail); and how the postal system actually delivers it (complex logistics network hidden from the customer).
The middle part is what lets the user benefit from the postal system’s expertise without having to learn postal logistics. The boundary defines the shared 'what' but also separates the customer’s 'why' from the provider’s 'how'. Not only that, but reusability means the same institution can be used to satisfy many purposes (birthday invites, subpoenas etc) and polymorphism means different providers can satisfy the same need and compete (UPS, FedEx etc).
An important question in institutional theory is how societies achieve both stability and adaptation; the paper authors say that the solution is stable interfaces allow changing internals. I find this very intuitive: when companies don't evolve/change from the inside much, you get ossification and insufficient adaptation. When laws change too much and institutions are unstable, uncertainty affects market confidence.
The people who are good at redrawing abstraction boundaries are entrepreneurs, who notice when existing categories are wrong and will invent new ones to remedy faults or address demand. What has always saddened me is how poorly rewarded and incentivized political entrepreneurship is. Part of the reason why is that this is hard: market abstraction boundaries are often disciplined by exit, entry, profit/loss, customer choice, and provider competition - but these feedback loops are much weaker in the public sector.
I hope we'll see a lot more of this in the coming decade. In fact this is something that AI will hugely facilitate, since it can lower the cost of articulating and prototyping new abstraction boundaries. We've already seen minor examples through e.g. citizens creating websites/services that compete with government ones. Though usually this is to make state services more legible rather than changing the boundaries in the first place.
I think if people want the future to go well, bolstering state capacity and enabling more innovation on the governance/democracy side of things will be critical. People don't really like this because it's a slow process, but I think they're wrong (and cheems), and playing the 'urgency of AGI' card to bypass this through a de facto state of emergency will cause lasting harms, partly by weakening institutional learning, public trust, and future coordination capacity.
Jay Creagh retweeted
We need more ambition in AI safety and governance.
There is more important work to do than there are organizations to do it, and more funding is available for ambitious projects than ever before.
To address the gap, we're hiring Entrepreneurs-in-Residence at @GovAIOrg. Participants get a year of salary plus ~$150k in seed funding via partner funders to start a new AI governance and safety org or project. We take no equity.
Even though it wasn't an explicit goal, GovAI staff have spent their time starting new orgs, including @Safe_AI_Forum and Trajectory Labs.
We're interested in a range of orgs and projects. Examples include: automating AI governance, an organization that rapidly produces well-evidenced, expert-endorsed “proto-standards”, an AI Bellingcat, a sub-frontier model evaluator, a TechCongress for outside the US.
We take rolling applications. The pitch is max 2 pages. If you've built something impressive and have context on the field, consider applying.
New post! Alignment doesn't just relate to a model's properties, but also the systems and institutions under which it operates. As we enter an increasingly multi-agent world, better harnesses, protocols, and mechanism design will be critical for safety. 🐜 blog.cosmos-institute.org/p/…
Jay Creagh retweeted
BREAKING: GIGANTIC COMPUTER POWERED BY HALF THE NATIONAL GRID SOLVES MATH PROBLEM STUMPING HERMIT PROFESSOR LIVING ON NOTHING BUT COFFEE AND BISCUITS FOR 15 YEARS
Jay Creagh retweeted
Many institutions no longer serve anyone’s values.
What’s guiding them instead seems to be purposeless busywork. A kind of white noise.
In a new essay, @edelwax calls this "Value Drift" and explains why it happens, even when everyone can see what's gone wrong.
Jay Creagh retweeted
Minor correction: mathematics requires pen, paper, and roughly $1 trillion of compute infrastructure.
I think it's important to try and tell these stories and explain these risks outside of their niche online communities. Gradual disempowerment in the face of agentic AI feels closer than I think most people realise.
18
Jay Creagh retweeted
Our research project SIMA is creating a general, natural language instructable, multi 3D game-playing AI agent. The agent can carry out a wide range of tasks in virtual worlds, making AI more adaptable, helpful & fun! dpmd.ai/sima-1
























