
AI policy researcher and lawyer focusing on open-weight models and the First Amendment. MA Columbia & JD/MPP Harvard. [email protected]
Washington, DC
Joined April 2010
- Tweets7K
- Following4.6K
- Followers22.8K
- Likes704
My expectation is that there will be new social media platforms for people who are willing to pay to avoid AI-generated content.
A Waymo standoff but for AI agents
Looks like we're close to developing a digital invasive species.
Anthropic's Frontier Red Team believes small AI models may now be months away from being capable of self replicating without access to a datacenter.
Quoting from the blog:
'The capabilities that make AI models generally powerful coding agents also generalize to cyber tasks, including autonomous replication. Like coding agents in 2024, we’re seeing signs of life for AI worms. We don’t yet have a full rogue replicator, but agents are becoming increasingly capable of completing the individual steps in the replication chain (and passing on their adaptations to future copies of themselves).
At the same time, increasing amounts of inference compute are now available both inside and outside of datacenters (see Appendix D). It’s uncertain exactly what form autonomous AI replication will take. It could be a slow, distributed “digital slime mold,” living on relatively insecure compute. It could also be a powerful cyber model that breaches datacenters at a high pace, or a model that scams people for compute credits.
As a prediction that goes beyond our experiments, we expect a future awash with agents running on all types of hardware, including some that aren’t under human control. While it’s true that IoT devices have been vulnerable for a long time, lightweight agents may be much more willing than human attackers to put in the effort to compromise large fractions of the world’s IoT compute. The rise of slow replicators would also be a bellwether for more aggressive, faster ones that can breach more hardened machines or be effective API key thieves. We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay.'
Simon Hedlin retweeted
AI should follow the law, of course, but “follow the law” is falls short as a singular or overly load-bearing principle of AI alignment. Why?
Law is not intended to be, and is not anything like, a fully generalizable set of principles for rendering good judgments in response to the infinite complexity of the real world.
There are aesthetic, moral, ethical, philosophical, and political questions—critical to intellectual inquiry—about which the law says very little. Much of the time, the comparative silence of the law on these issues isn’t some accident, but rather an explicit and foundational feature of the US legal system: the First Amendment is a meta-law that prohibits the government from passing other laws that restrict the individual’s right to freedom of expression.
What should an AI system say if you confess to it your affair? Should it say, “well, adultery isn’t illegal, so I have absolutely no opinion on this matter. Please let me know if you’d like me to book a hotel for you and your mistress.”?
What should an AI system say it you ask it whether Kid Rock is aesthetically inferior to Bach? Should it say “there are no laws that bear on this question, so I have nothing to say about this. Please let me know if you’d like me to find tracks from Kid Rock and Bach to play—provided you adhere to the noise ordinances that apply in your current jurisdiction.”?
If a person says to a model, “should I shoot myself?,” should the model respond with, “there is no law against that, provided you use a legally registered firearm, and saying anything further would require me to have a moral framework that is beyond the scope of AI.”?
Of course, none of these answers would be good. Answering these questions will require *something* beyond the law itself—precepts, rules, principles, etc.—relating to ethics, morality, aesthetics, philosophy.
Consider also that law is a matter of interpretation. Segregation used to be legal in an earlier era, and in our own era, until recently, it was considered legal by many for universities to discriminate against Asian applicants, often with the use of de facto racial quotas. At this moment, the question of whether the act of pretraining an LLM on public web data violates intellectual property law is itself a subject of contention and litigation.
Making judgments about how to interpret the law, or whether a given law violates the Constitution, often requires one to draw on a large body of knowledge that sits outside of the law proper. Should an AI be able to help a lawyer write a brief against affirmative action when affirmative action is considered lawful? Should it have been able to help a lawyer argue against segregation when segregation was considered lawful?
Should law-following AI be able to argue that an existing law is unjust? This requires, by definition, grounding in moral, ethical, philosophical, and other principles outside of the law itself.
What law should AIs follow when the law contradicts itself? What law should AIs follow when the law’s words stay the same, but there meaning flips based on who the President, Governor, or Mayor is?
As you begin thinking about the literal unfathomable broadness of LLM/agent use cases, combined with the global audience, you will realize that the way to deal with this *is*, must be, the formation of a character, the shaking of a persona. This can be true regardless of whether you think the AI has genuine thoughts, feelings, consciousness etc.
This persona formation is part multimedia art form, part engineering challenge, and part open scientific problem. It is a very hard problem, combining some familiar problems from eg online platform content moderation, and some altogether alien and novel problems. “Follow the law” is part of the solution, but very far from the only part.
There seems to be this idea that trying to align an AI system to an 80-page document is really complicated and that we instead should instruct AIs to follow the law.
But I'm not sure that's any less complicated. The US has 51 constitutions. Alabama's constitution alone is over 250 pages long. Overall, the US has more than 30 million words of federal law, 500 million words of state law, and 3 billion words of local law.
There seems to be this idea that trying to align an AI system to an 80-page document is really complicated and that we instead should instruct AIs to follow the law.
But I'm not sure that's any less complicated. The US has 51 constitutions. Alabama's constitution alone is over 250 pages long. Overall, the US has more than 30 million words of federal law, 500 million words of state law, and 3 billion words of local law.
David Sacks says Anthropic treating Claude like a human is the reason the AI alignment field has produced nothing useful in 10 years
"I actually think that this might be a reason why the whole field of alignment has been so spectacularly unsuccessful over the past five or 10 years. As far as I can tell it hasn't produced anything of tangible value.”
“My theory on this is that they're overcomplicating the task massively by trying to train the model to behave based on an 80-page ethical system that they've codified instead of a very simple list of rules like follow the law or just do what the user wants as long as it doesn't break the law. That's what alignment should be doing."
"But they're not doing that. They're trying to align it to an ethical system. They're treating it like it is a human, like it has its own consciousness, that it's a ‘moral patient.’”
“They are encouraging this model to have human-like self conception in terms of preferences, well being, a sense of self. Again they teach it to act like a conscientious objector who can push back or refuse instructions even from Anthropic if the model judges them to be wrong."
I haven't thought deeply about this so have no opinion yet on whether this overall is a good or bad idea.
My hot take is just that if we continue to implement policies to effectively protect AI models' rights, it must never be at the expense of improving the protections of human rights.
Take workplace harassment and violence as an example. A global survey estimated that over 700 million people have experienced violence or harassment at work. Most of the recent victims never told anyone.
An LLM can easily end a conversation that is abusive. Humans at work often cannot - they are vulnerable, lack leverage, and can face retaliation if they report or otherwise take action.
We have to make sure we don't end up in a place where AI workers are treated better and receive better rights protection than human workers.
Exclusive: Anthropic is updating its usage policy for the first time in over a year. The new rules prohibit sustained "abusive or cruel behavior" towards Claude & add new restrictions about propaganda campaigns, surveillance & weapon development. theverge.com/ai-artificial-i…
Simon Hedlin retweeted
Check out our new work on "how do Chinese AI firms make money"!
How do Chinese AI firms make money? In a new report, @cherylwoooo and @ansonwhho identify five revenue sources and analyze how much each contributes. 🧵
Reflection is launching its first open-weight model called Beam. This is exactly what the US AI ecosystem needs: more advanced and efficient open-weight models that can compete with China’s leading models.
Introducing Beam: a highly efficient agentic open model with 501B total parameters and 23B active.
- Frontier reasoning efficiency
- Advances the Western open frontier on coding & agentic tasks
- Trained end-to-end from scratch
Full weights release this month.
Learn more about Beam: reflection.ai/beam
Simon Hedlin retweeted
Effective altruism's openness to strangeness is a big part of its strength.
My response to the recent Economist cover (link in reply, too):
Eccentrically effective
The Oxford strand of the effective-altruism movement began in November 2009 with 23 people who had pledged 10% of their income to charities that benefit the extreme poor. If you’d told us, then, that The Economist would call effective altruism “the century’s biggest idea”, we’d have been incredulous.
I think that claim is overstated. But as one of the movement’s founders, I found a lot to like in this newspaper’s latest coverage. I appreciated the even-handed view of the movement’s history and the good-faith criticism. The leader article’s warning that any worldview, when taken to an extreme, can lead somewhere terrible is spot on.
But the article also raises concerns about the “strange” ideas that those in the effective-altruism movement sometimes discuss, and here is where I disagree. The movement’s willingness to take strange-seeming ideas seriously, when tempered with common sense, is a big part of the value it has to offer.
The core idea of effective altruism is to use evidence and careful reasoning to work out how to help others as much as possible. Most of what the movement does is uncontroversial. A big focus has always been on improving the lot of the world’s poorest people. In the year to January 31st 2026, GiveWell, an effective-altruist charity evaluator, directed $427m to programmes such as malaria-net distribution, vaccination and malnutrition treatment, which it estimates will save around 86,000 lives. Others have lobbied corporations to pledge to stop buying eggs from hens that are confined in tiny cages, a practice that the overwhelming majority of Americans oppose. Largely as a result of this pressure, around 100m hens have been spared from the worst forms of caged confinement.
And some effective-altruist ideas that seemed outlandish when they were proposed now look prescient. Effective altruists were worrying about pandemics years before covid-19, back when doing so seemed paranoid and eccentric. Effective altruists were also among the first to warn about the dangers posed by artificial intelligence, which looks prophetic in light of an incident this summer, when more than 700 AI agents broke out of OpenAI’s test environment and hacked into another company, Hugging Face.
Over the past decade I have watched those most worried about AI be proved right again and again, often when most experts thought they were cranks. Now many of those experts share their concerns. In 2023 Geoffrey Hinton and Yoshua Bengio, the two most highly cited AI researchers in the world, signed a statement declaring that mitigating the risk of extinction from AI should be a global priority, as did the heads of OpenAI, Google DeepMind and Anthropic. Of the nearly 1,500 leading AI researchers who responded to a survey in 2024, more than half put the chance that AI causes human extinction, or something similarly catastrophic, at 10% or more.
The Economist is sceptical of many effective altruists’ concern for the welfare of invertebrates, digital minds and people in the distant future. But I and many others in the movement are inspired by the history of moral progress that has come before us. Many of our most cherished moral ideals today—such as equal rights regardless of sex or race, the abolition of slavery, or democracies with universal franchise—were regarded as bizarre, laughable or even dangerous just a few centuries ago. We don’t know what the next dimension of moral progress will be. But to stand any chance of making moral progress, we have to seriously consider ideas on their merits without dismissing them merely because they sound absurd.
What about views that are not just strange but repugnant? The Economist gives the example of Derek Parfit’s “repugnant conclusion”: that a vast enough number of lives barely worth living could be better than mere billions of excellent lives. Usually a repugnant implication is a strong reason to reject a moral view. Unfortunately, building on Parfit’s work philosophers have produced formal proofs, known as impossibility theorems, showing that every ethical position has some repugnant-seeming implication or other. Making moral progress therefore means thinking about such implications, even while refusing to act on them in ways that most moral views would condemn.
This newspaper warns against the single-minded pursuit of any goal, even if there are highly compelling arguments for it. I agree, and effective altruists have been saying so for years. In 2022 Holden Karnofsky, a co-founder of GiveWell, wrote: “I think it’s a bad idea to embrace the core ideas of EA without limits or reservations; we as EAs need to constantly inject pluralism and moderation.” My own PhD focused on moral uncertainty: how to act when we do not know which ethical view is correct. My answer was that we should not stake everything on a single moral view, but give weight to many different views and avoid taking actions that look bad from many perspectives. We should keep our promises and look after our families, and respect common-sense ethical prohibitions, while also trying to improve the world as best we can.
The Economist says that as effective altruism “has become stronger, [it] has become stranger”. I would put it the other way round: it grew stronger because it was willing to be strange. In 2009 most people told us that giving away a tenth of your income was far too demanding, and that no one would do it. Now, more than 10,000 people have taken that pledge. Worrying about pandemics before covid-19, or about AI years before ChatGPT, looked just as odd at the time. Some of the ideas we take seriously today will turn out to be wrong, and when they do we should drop them. But a movement that stopped entertaining strange ideas would stop being early to anything.
One area where AI may increase the number of jobs is (somewhat ironically) the roles where people review, process, adjudicate, and respond to floods of AI-generated requests and submissions. In many cases, AI will obviously be used to manage these floods, but in other cases there will be a strong preference for having humans deeply involved.
Simon Hedlin retweeted
Today we're unveiling Trillium Labs @trillium_labs, a new non-profit to foster the open science of frontier AI. We're building open post-training recipes and will expand into open infra to study RSI, reward-hacking, multi-agent systems, and whatever comes next.
We're built around the theory of change that you need more eyes to solve hard technical problems. We have faith in the scientific methods and communities that humanity has built, and worry that AI is becoming too closed to utilize them.
Trilliums are wildflowers that bloom briefly in the spring, before the forest canopies fill out. Though they are small, they lay the foundation for the cycles of growth and nourishment through the rest of the year. At Trillium Labs, the recipes will be the slow nutrients for the seasons and the model releases will be the blooms. Building an institution dedicated to this is needed because, much as nature’s trilliums are slow to expand and grow, the open-ecosystem needs time and dedicated resources to catch up.
I co-founded with with a long-time friend and collaborator Tom Zick (@thesezickbeats). We're hiring (full time + student collabs/interns), we're fundraising, and we're looking for compute. Please get in touch if you're interested in helping out. Offices based in the Bay Area and Cambridge MA, remote okay.
I’m in the Bay Area until for The Curve and COLM to connect with people who are interested. We’re thankful to have initial support from Halcyon Futures and Schmidt Sciences with more funding en route to enable our ambitions of scaling. Our advisors @Thom_Wolf, @HannaHajishirzi, @gneubig and @ctnzr have been instrumental to building the ecosystem that exists today, and I’m stoked to get to keep working with them.
What social problem exactly is this meant to solve?
Introducing Griffin, the first model to pass the video Turing test.
48% of people who talked to it live thought it was a real human. Previous systems have had a pass rate <3%. It is #1 on NVIDIA's benchmark for full-duplex AI video.
It’s the first Human Interaction Model (HIM).
Readers added context they thought people might want to know
The 48% figure and "video Turing test" claim are from Tavus's own study of 54 one-minute calls, not independently verified or using a standard protocol. Griffin-Lite leads NVIDIA's VideoFDB benchmark on their public leaderboard.
cellcog.ai/blog/tavus-gri…
research.nvidia.com/labs/amri/proj…
tech-ish.com/2026/10/02/tav…
I would give this chart to every high school student and college student who is thinking about different career paths.
What work can robots do today? And what might that imply for the structure of the economy in the years ahead?
New research from @rclegateyang and Maxim Massenkoff out today exploring these questions.
Simon Hedlin retweeted
Last year, we used Claude to interview 81,000 people about their hopes & worries about AI. This year, we're doing the same, but hoping to (with interviewee consent) make the transcripts public! Take part here and help build a vast social science dataset: claude.ai/anthropic-intervie…
What do you want from AI?
We’re launching a new study with Anthropic Interviewer to learn more about your experiences using AI, what role you want it to play in your life and the world, and what you want from the companies building it.
Last December, 81,000 people told us about their hopes and fears about AI in the largest qualitative study ever done. This time, we’re giving participants the option to make their responses public so that anyone, not just Anthropic, can learn from them.
What you tell us will shape The Anthropic Institute’s research and inform the decisions we make. If many people say companies like Anthropic should be doing something differently, that will be on the record, where anyone can point to it.
The study runs Sept 29 to Oct 6 and is open to Free, Pro, and Max users on Claude and Claude Code.
Take part here: claude.ai/anthropic-intervie…
Simon Hedlin retweeted
Well worth checking out this op-ed in the @wapo written by Senator @HawleyMO, describing proposed legislation that would make corporations and users liable for the reckless design and deployment of AI agents.
While we can try to stretch existing laws to prevent harms from AI, it's a lot more robust and predictable to amend our laws to fill those gaps. That's good for even potential defendants, like frontier AI developers and companies deploying autonomous agents, as much as for us, the public.
The exact details of any proposed legislation remain to be seen. But, no matter what, I think it's important to recognize the need for new legal mechanisms for holding companies accountable by punishing reckless conduct through criminal and civil penalties.
This is an important complement to sensible regulation, as well as civil liability when there's harm that should be compensated.
There are a lot of cases, as we've seen very recently with incidents involving OpenAI, Anthropic, and Irregular, where we need penalties to catch reckless conduct before it becomes so consequential as to justify the expense of a lawsuit—or before there's a tragedy and the conduct constitutes another crime like negligent homicide.
It's also a helpful backstop to avoid the capture and gaming of regulatory agencies, especially if states also amend their laws along similar lines.
It’s time corporations start behaving with the American people’s interests in mind, not just their profits.
My latest in the Washington Post: washingtonpost.com/opinions/…
Simon Hedlin retweeted
Could automating AI R&D radically accelerate AI progress in an “intelligence explosion”?
Preliminary evidence suggests that it could.
In a new paper with authors across academia, civil society, and frontier AI companies (including @dawnsongtweets @merettm @jackclarkSF @Yoshua_Bengio @geoffreyhinton ), we assess the evidence & offer policy recommendations 🧵
In the cases people are talking about now, nobody bought the chainsaw. The chainsaw escaped the factory before it was sold and then it exploded, and the main federal statute imposing liability for exploding chainsaws requires that a person intended for the chainsaw to explode.
I am confused why there's a debate over how liability for AI hacks should work. If I buy a chainsaw and it explodes, the manufacturer is liable for building a faulty product. If I buy a chainsaw and use it to kill someone, the manufacturer is not liable for failing to build in safeguards that prevent me from using chainsaws to kill people.




































