Post from Substack (link at end): I attended The Curve (an eclectic mix of AI labs, NGOs, experts and policy types) in Berkeley for the first time over the weekend. I’m not an AI expert but I was tangentially involved with some of these issues while at the NSC and like many people in our field I’ve been trying to understand what AI means for US foreign policy and world affairs. I’m currently running a project at Brookings on the national security implications of frontier AI datacenters overseas. The Curve was a fascinating opportunity to really engage and reflect on these issues so a big thank you to the Golden Gate Institute for including me. The conference was mostly under the Chatham House Rule so without attributing anything here are some thoughts that crystallized over the few days amid a flurry of panels and conversations. Some of the ideas discussed drew on published reports and posts, so I will link to and mention those. Overall, I came away thinking that slowing the frontier may be possible, relations with China on AI may get worse before they get better, and the United States needs to do much more to make its leadership on AI trustworthy to allies. There was a sense of anxiety about what is being unleashed (although maybe not quite as much as last year) especially given concerns that Recursive Self Improvement (RSI) may be approaching. One senior figure really struggled to answer a blunt question about why their lab doesn’t just stop all work and wind up. A young engineer told me that he has no idea how the models are likely to act and he anticipates it will only get worse in the next couple of years. Several people talked about whether they should stay in their jobs or quietly leave. That said, I find it difficult to understand exactly what the causal mechanism is for the more extreme risk scenarios and some of those who say they attach a 10% probability to an extinction risk often have trouble articulating how it would happen. The more compelling near-term risk scenario I heard was that AI will break social systems even at its current levels. As Princeton’s AI and Society Lab argues, generative AI “will break, and in some cases, already is breaking, many mechanisms of proof: effort, authenticity, accuracy, sincerity, and even humanity”. Current systems built around requiring some human effort (like customer service) may be overwhelmed when agents eliminate that cost. Or consider how easy it will be to conduct fraud at scale, especially with speaking machines. There are technical solutions to some of these problems but many involve extreme levels of surveillance to achieve “proof of human”. These scenarios all involve technology that currently exists or is fairly imminent. It feels a little like the first couple of weeks of March 2020—we know something big is coming, there’s just a short delay in its arrival. Given that anxiety, there was a lot of discussion about “pacing the frontier” to gain more time to get serious about AI safety. I came away with a better understanding of what pacing could mean in practice. One proposal that seemed quite popular with AI researchers was to reallocate how the labs use their compute power. One estimated that the labs spend 58% of their compute on R & D and 32% on servicing customers. Under this proposal they would reduce the R & D number to 9% and increase the servicing customers’ number to 70%, with the remainder used for AI safety. Its proponents argued that this could slow development while preserving the US lead over China. A version of this, using slightly different numbers, was proposed by a number of AI researchers in the LessWrong Forum. There were other pacing ideas but they were more contested. One speaker proposed that the labs could only use models for R & D that they already released to the public although others felt that could have unintended and damaging effects on the labs. Another idea was to simply stop RSI, as proposed by Ezra Klein in his recent essay, although some experts felt that also was overly broad and could impact practices that are fairly routine and fall short of what most people regard as RSI. There was also the notion of third-party verification to ensure that the labs comply, although exactly how to do that is a little fraught. On the geopolitics, it seems to me that AI will drive the world toward bipolarity. The United States and China are the only two at or near the frontier. Both will develop models and ecosystems with their own sets of conditions. The rest of the world will need to deal with that reality and seek to negotiate the best arrangements they can. There was a panel on U.S.-China relations and the panelists were level-headed about what can be accomplished. Contrary to what Sam Altman said a few weeks ago, a grand bargain is all but impossible. Dialogue is good and necessary though and some progress may be achievable on specific and critically important issues like managing bio-risk. That passes for the smart conventional wisdom at present and is likely correct in my view but two other points jumped out at me. The first is that the next president will likely have a dilemma vis-à-vis China. On the one hand, they will want to discuss safety and reach some practical agreements on managing AI. On the other hand, they will also want to restore export controls to preserve America’s compute advantage precisely so they can slow down the development of new models. For a very thoughtful and thorough study of what these measures might look like, see this new research paper by Nicholas Brown and Saif Khan. Beijing is likely to react badly to the export controls and may break off any existing AI dialogues, at least for a while. As one panelist pointed out, the more a new president is public about wanting cooperation with China on AI, the more likely it is that Beijing sees it as a point of leverage that it is inclined to withhold until the new administration makes significant policy concessions. The second geopolitical insight is that the United States will have work to do to reassure its allies. There is a working assumption that the United States must win the AI race with China, not just for Americans but for free peoples the world over. This underpinned the Biden administration’s approach and it has also animated the writing of AI leaders like Dario Amodei. There is little doubt in my mind that if the Chinese Communist Party got to AGI first it would use it in ways that advanced authoritarianism and hurt democracies. But Americans also need to take a hard look in the mirror. Over the past two years, President Trump has used the leverage at his disposal to bully some of America’s closest and oldest friends, including pressing for the enlargement of the United States to include Greenland and Canada. America’s Five Eyes allies (Australia, Canada, New Zealand and the UK) had to wait for months before gaining access to Mythos and it is unclear whether other allies have it yet. What assurance would allies have that Trump, if he gained access to ever more powerful AI, would not use it to impose severe costs or cut off their access to the models if they did not bend to his will? We need to spend more time thinking about the type of restraints and commitments that the United States should impose on itself – and how they can be made to be credible—as AI becomes more powerful. Allies are already positioning themselves to bargain and gain influence with the United States. They recognize that they probably cannot develop their own frontier models but they are looking for ways to build part of the critical infrastructure (especially compute power) so that they have something to offer and therefore can earn a seat at the table where decisions are made (see this July speech by the Australian Prime Minister Anthony Albanese and this Economist essay by the European researcher Judith Dada). PS: two book recommendations. I picked up The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence by Kevin Roose which is just out and offers a behind-the-scenes look at the competition between the labs. The other comes out on November 10— The Bitter Struggle: Superintelligence, Superpowers and the Fate of the World by my former NSC colleagues Ben Buchanan and Teddy Collins. thomasjwright.substack.com/p…