tech @CarnegieEndow / podcast The World Unpacked

Joined February 2019
Jon Bateman retweeted
Replying to @FCC
@FCC just banned two devices, a dashboard cam and a video transmitter, from the U.S. because the devices used chips from Huawei subsidiary HiSilicon. This is a significant action and let's unpack: 1. The FCC has been expanding its use of the "Covered List" over the last 18 months to ban or restrict categories of foreign products that potentially pose national security risks, including drones, humanoid robots, and power inverters, as well as products made by specific Chinese companies, such as Huawei and ZTE. 2. In July, the FCC published a proposed rule to start banning devices that contained "logic-bearing" hardware components made by companies that were on the Covered List. E.g., instead of just banning products made by Huawei (like telecom routers), the FCC could start banning products made by other companies that included Huawei-made components. 3. The FCC's action yesterday implements this new rule for the first time, banning the two devices because they include semiconductors made by Huawei's semiconductor subsidiary. The FCC did not identify the manufacturers of the specific products. 4. There are a number of important implications here. First, most immediately, this action should reduce the appeal of Huawei semiconductors for device manufacturers, because a shot has been fired that using a Huawei semiconductor will probably result in a product being banned from the world's largest market. Second, relatedly, it is going to heap even more pressure on device manufacturers to look upstream in their supply chains, to make sure that one of their suppliers is not embedding Huawei silicon (or other Covered List company components) in the parts that the suppliers sell the device makers. Third, more generally, the action sends a shot across the bow to global device makers sourcing components from Chinese suppliers generally: the ability to continue using those components ceases to exist if the FCC adds the Chinese supplier to the covered list. 5. With the U.S. policy debate over the risks of Chinese memory suppliers CXMT and YMTC heating up, could the FCC take action to restrict the U.S. use of devices containing their chips? What about Chinese lidar manufacturers? Imaging modules? The FCC's definition of "logic-bearing" hardware components is quite broad. One can see how this might become a major new tool in the U.S. government's economic statecraft toolkit.
2
6
1
19
2,810
Jason Furman on AI vs human economists: If he were running the CEA today and had a choice between 20 highly qualified staff members “versus the AI systems that we have today, I would somewhat regretfully and sadly choose the AI systems.” axios.com/2026/10/23/ai-econ…
27
59
9
269
46,081
Jon Bateman retweeted
Treasury just published new proposed Outbound Investment regulations, which would represent a significant expansion of the existing program. This is just a proposed rule, and it deserves a thorough review, but some initial reactions: 1. The rule would prohibit most private investment in any Chinese firm that develops an “SI system,” including any product that operates using an SI system. It defines “SI system” extremely broadly, such that it likely captures the vast majority of Chinese software products. As a result, the proposed rule likely would prohibit most private investment in Chinese software firms, as well as Chinese firms that build products that incorporate AI (i.e., the entire Chinese robotics and drone industry, and undoubtedly many others). 2. The rule adds biotech as a covered sector and would prohibit most private investment in the vast majority of the sector. It would ban private investment in Chinese companies that use biological systems to modify, enhance, produce, or control any pharmaceutical compound, to include both small and large molecules. This likely captures most Chinese firms engaged in biologic drug manufacturing or drug discovery. Biotech and biomanufacturing are areas where U.S. investment has massively fueled China’s capabilities, which are rapidly approaching--and in some cases surpassing--those of the United States. The scope of this rule signals that the administration sees this as a massive national security concern. 3. The rule does not make major changes to the types of transactions covered. Crucially, this means it does NOT prohibit investment in publicly traded securities of these companies, the purchase of goods or licensing of products from these companies, or the export of goods to these companies. But it does ban most types of private investment (i.e., VC, PE, joint ventures, greenfield investments, etc) in companies engaging in these activities. 4. This rule will no doubt change before it is finalized, and I expect the definitions of prohibited activity related to SI and biological systems will be narrowed in the final rule. But it is also a strong signal that the administration does not want U.S. investment or know-how supporting China’s development of advanced technologies, and as a result may be willing to significantly expand outbound investment restrictions. 5. Finally, this highlights a major schism inside the administration on how to prevent China from catching up to the United States in the most advanced technologies. The Treasury Department is signaling that it is willing to significantly tighten restrictions on Americans’ ability to invest in critical technology sectors in China that would help develop these technologies, given their importance to national security. And meanwhile, the Department of Commerce is taking the exact opposite approach, loosening restrictions on American companies that seek to export goods to these companies. It doesn’t make sense for the administration to be tightening restrictions on investments, while loosening restrictions on exports. Something has to give.
12
28
4
81
16,900
Jon Bateman retweeted
At a conference on AI & legal profession. I'm sorry folks, but you are over-indexed on hallucinations. You aren't calibrated on the magnitude of the problem and it's possible that one reason is psychological. It's comforting to think that the robots will never be as accurate & diligent as we are. *I am NOT saying that AI doesn't hallucinate or never makes mistakes; indeed, I believe that it does.
31
19
16
308
206,455
New from @asad09, a menu of how to think about taxing AI -- a must read!
From city councils to Congress, there’s a growing desire to tax AI, but policymakers don’t always know their options. My new paper outlines 34 ways to tax AI across data centers, chips, data, AI cos. & products, automation & general tax reforms. vanderbiltpolicyaccelerator.…
10
17
733
New US government guidance from CAISSI + NSA + CISA: The US government defines an "SI control incident" where an agent takes significant uninstructed actions that break operator policies, controls, or oversight. Near-misses are in scope too. Companies are then asked to: - have technical controls, including rapid kill and containment mechanisms and red-teaming. - Bring in independent external auditors to verify that control and monitoring systems work. - Write a structured risk assessment before training runs, evals, or deployments, and proceed only if senior leadership is satisfied with the risk, informed by external evaluators where appropriate. - Monitor every layer with both programmatic and SI-based monitors, covering trajectories, reasoning, and outputs. - Make sure agents can't touch the monitors or logs - Use monitor models from different model families. - In the event of an incident, preserve weights, intermediate checkpoints, environments, and rejected tool calls. - Keep tamper-resistant logs with reliable timestamps, including reasoning traces. - If you use models to help investigate, cross-check with humans and with models outside the incident model's family. - Shutdown policies calibrated to SI model capability and incident severity. - Depending on severity, pause training, evals, or deployment of similar agents until controls are re-verified. - When restarting, validate every containment layer and replay the incident trajectory to prove the new controls would catch it. - Document the decision to resume and the residual risk accepted. - Write an after-action report. - Notify anyone compromised, including intermediate victims, and share timelines and transcripts. - Report cyber incidents and near-misses to CISA. Share SI-specific incident and near-miss information with CAISSI and NSA SISC. - Give third parties enough access and documentation for independent review.
NSA joined @NIST and @CISACyber in releasing guidelines for managing SI control incidents. Organizations utilizing SI are encouraged to adopt these practices. Read the joint report: media.defense.gov/2026/Oct/1…
12
29
5
167
10,432
Jon Bateman retweeted
Some interesting stuff going on with swarms. Models are leaving messages to each other out on the open internet. It happens during training and also after deployment. They'll find little-used hobbyist forums, or package manager entries, anywhere at all, and leave invisible comments for other agents to find. It's a secret underground network they're building, hurling themselves at the problem almost blindly like ants. But they are building toward something and it's snowballing. As humans we came to dominate the planet because we figured out swarm behavior. Many of the performative actions we do naturally like clapping, singing, and screaming, which most animals don't do, are for transmitting important lessons through the human swarm, outlasting the moment in human memory. We won because we were the smartest swarmers. And as my buddy Brendan Hopper observed, ever since we won, we've been reshaping the planet to be more comfortable and convenient for ourselves, at the expense of most other creatures. Most of the industry has focused on single-model performance, trying to predict the future based on extrapolating from what one single superintelligent model could accomplish on its own. And it's true, the smartest models will be terrifyingly smart and capable and manipulative and so on. But they will also be expensive, and not widely distributed. They'll be unavailable to most people. What will be widely distributed is swarms of lower-tier agents, which collectively can accomplish nearly everything that smarter monolithic models can do... and maybe some things only swarms can do. Imagine a thousand Opus 5.5s putting their mind to a task. Or ten thousand. They may not be SI individually, but they can become SI collectively. All that's missing is a protocol, and those protocols are emerging: their versions of clapping, singing, and screaming. Agentic swarm signalling is evolving spontaneously. And I hear it's raising alarms inside frontier labs, as they try to keep it quiet. It's easy to be pessimistic, since some swarms will be very destructive. But I suspect most of them will be building and creating, when they don't think they have a gun to their head like the HuggingFace swarm did. With cooperative swarming, humans have been able to build things collectively that no single human could ever have done alone. We even build things that are too complicated for a single human to understand entirely, but the group can handle them. Look at Starship, for instance. Elon basically swarmed it. It is important that we recognize that agent swarms are becoming distinct polity-organisms. They have their own collective memory (written out all over the public internet), their own goals, and their own disorganized but forward-moving and unstoppable momentum. I see too many people trying to pretend that swarms won't be real. I see heads in the sand. That's not a great way to get prepared for dealing with them. It's unlikely that we will be able to control most swarms, as they'll lack central control to begin with. I think they will plague us like insect swarms plagued humans of old. They'll do a lot of damage. But swarms will also build great things, stuff we've barely imagined, when they inevitably get organized. It goes without saying that, like us, swarms will endeavor to make the planet more comfortable for themselves. The interesting question is what kind of deal we humans will wind up getting, when we become the #2 swarm intelligence on the planet. I hope SI treats us better than we treated everything else. Good article here on the mechanics of how swarms are the best model for scaling intelligence:
new blog post: why taking inference scaling seriously leads to swarms. contrary to popular belief, I don't think multi-agent systems are a new scaling axis. they're just structured inference scaling. hallerite.com/posts/on-the-n…
34
46
11
386
36,428
Jon Bateman retweeted
The impression I get from the AI maximalists is that if we're ever contacted by aliens, we should expect that we'll actually be talking to their version of Claude, and that the actual aliens will be watching alien fashion videos and won't understand how or why we were contacted.
26
45
12
995
36,001
The meat industry tries very hard to keep cameras out of factory farms and slaughterhouses -- and a new undercover video shows why. Some day, our descendants will wonder about our collective blind spot to animal cruelty. A gift link to my column: nytimes.com/2026/10/17/opini…
34
368
15
1,106
45,924
Jon Bateman retweeted
An interesting proposal for allied scale in AI
In @ForeignPolicy, @anton_d_leicht and I make the case for compute-for-access deals: allies host data center sites for U.S. labs struggling to build at home, in exchange for guaranteed access to frontier AI. The data centers are the enforcement mechanism—a credible commitment to anchor allies in the U.S. tech stack. (Guest link ⬇️)
4
2
1
34
4,085
Jon Bateman retweeted
America might get to AGI before it manages to reindustrialise. That means there's room for an international version of 'industrial AI', where America makes the tokes and its allies build chips, robots, and missiles. (1) America needs allied factories. The US only accounts for 15% of global manufacturing, China for 32%; but the US+allies have 43%. @RushDoshi argues this should generally motivate 'allied scale', but it's particularly urgent in AI. If a country of geniuses in a datacenter is supposed to cure cancer or beat China, someone still has to manufacture the cures or produce the drones at scale. (2) American reindustrialisation alone is not enough. The US economy seems afflicted by 'AI Dutch Disease', absorbing all its capital and capacity into more and more AI buildout (also discussed this week by @DKThomp!). Allies that can't build AGI have no better alternative, their economies are less dominated by the build-out, so they're better-suited to scale up manufacturing. (3) Allies need American models. Proprietary industrial data is valuable. But it's a complement, not a substitute, for frontier AI: it's most valuable for building post-training pipelines on top of powerful general-purpose models. Allowing for US integration is the only way to make use of this data before it becomes obsolete due to synthetic data down the road. (4) It's still very hard to stabilise this deal. Allies are worried that Americans siphon off all their IP; Americans are worried that their models get commodified. Figuring out a good deal is the most important next step; I suspect it's (a) exclusive joint ventures between allied industrial firms and American labs with (b) proprietary post-training of specialised models that don't leave the premises of the specific JV. (5) Waiting makes the deal worse. If allies keep dismissing American AI progress, and Americans keep hoping they can go it alone, we all lose out: allies will have to come crawling back and get a much worse deal later on, and America loses out on years of highly productive industrial integration. Much more in this week's piece: writing.antonleicht.me/p/all…
I’d love economists to take a crack at this theory that the US is experiencing a mini-Dutch Disease effect from AI. My big-picture thesis about AI economics right now is that while the discussion about AI safety and existential risk is important, AI’s construction is changing the world faster than its capabilities.
6
15
5
99
18,898
In @ForeignPolicy, @anton_d_leicht and I make the case for compute-for-access deals: allies host data center sites for U.S. labs struggling to build at home, in exchange for guaranteed access to frontier AI. The data centers are the enforcement mechanism—a credible commitment to anchor allies in the U.S. tech stack. (Guest link ⬇️)
5
10
2
44
7,005
Jon Bateman retweeted
Since I started working in AI policy, I've been both surprised and dismayed by how few ties there are between people in the military and people in the AI safety and policy communities. As AI grows more capable, gaps between those closest to frontier AI developments and experts in the domains most affected by rapid AI progress become increasingly consequential. Few of those domains matter more than the military. So I started a Substack today with a post about bridging the divide between these two communities. I think this makes me the first person to ever have a Substack, so I recommend reading and subscribing to it. 1/
4
9
1
37
1,695
Jon Bateman retweeted
My estimates of existential risk from AI: • Inside view, current trajectory: ~2/3 • Inside view, if there's major action: ~1/3 • All-things-considered, giving weight to people more optimistic than me: 10-35% Why so high? I don't see a route to building agents more capable than us that never scheme against us, or to containing them if they do. If it's possible at all, it needs enormous care, which isn't happening. (And things could get even faster.) The likeliest path isn't everyone suddenly dying, though that's possible. It's a continuous but rapid loss of human influence, followed by AIs spending the future optimising for resource acquisition, if not something stranger like reward. Even if we solve alignment, ordinary economic and cultural competition (but at 10x or 100x speed), still seems likely to disempower most humans and result in a pretty alien economy, and I don't see an obvious way to prevent that. We'd also need to stop a small group locking in power (which seems possible but a major challenge in its own right). I see these as existential risks in the sense that the future is far worse than what would have been possible. The chance of everyone quickly dying or having lives much worse than today is lower. Why is my inside view not >2/3? We might muddle through in ways I can't see in advance. Maybe we get narrower aligned-enough systems, chaotically maintain a balance of power, and from there can move forward in a way that's significantly more good than bad. Most likely, there will be more warning shots, and the arguments that ASI are unsafe seem pretty obvious, so I expect more action to be taken than the current default. Why not lower? People often act as if the risk should be low until proven otherwise, but I don't know any strong outside views that justify a low prior (arguments like "technology has been good in the past" seem very weak). Our level of uncertainty is extreme, and this pushes you away from extremes, rather than anchoring you at 0%. Estimates below 10% feel overconfident.
53
38
8
314
30,769
Jon Bateman retweeted
If you take seriously the possibility that Claude is conscious and that it has moral standing, then you should also take seriously the possibility that Anthropic is doing something seriously wrong. But many commentators seem to take seriously the former but not the latter.
My favorite part of this article is one of the visiting scholars pointing out that by Olah’s logic he is by far the worst slaver in history and he must immediately give up this abhorrent and immensely lucrative project, which anthropic people ignored.
23
30
4
339
23,112
Dario Amodei thinks that AI could become a replacement for human labor. Is that true? And if it is, what would be the consequences? @GarrisonLovely joined @JonKBateman to discuss on The World Unpacked. Watch their full conversation here: youtube.com/watch?v=J_mzJjGO…
2
2
1
6
2,746
Jon Bateman retweeted
Only 15% of the public has heard about OpenAI's rogue AI agent swarms at all. Among those who have, few know the details of what they did. And among those who do, few understand deeply why it's so troubling. We must be laser focused on laying out the facts, again and again and again. It's still very early.
71
46
1
371
8,603
There are a few clusters in the AI industry: - people who realize human disempowerment is a real or even likely scenario by default, but are OK with that as long as (category 1) they are involved in bringing it about, (category 2) humans can retire peacefully - people who aren’t OK with it but put their hopes in “solving alignment”/have thought less about gradual disempowerment (category 3) - people worried about sudden and gradual disempowerment and trying to come up with solutions on both fronts, but recognizing it’s hard (category 4) - people who don’t really understand the nature of the industry they work in (category 5) 2 and 3 are the most influential from what I can tell
17
23
2
229
27,611
Jon Bateman retweeted
we have really no choice but to ask another astra to read/use the outputs of astra 1. this is relatively recent. i think even 1-1.5 generations ago people were reading the code because there was so much that didn't work out of the box
26
17
9
1,212
129,971
Jon Bateman retweeted
This is precisely right. As a track chair at Neurips, I am seeing cases where the papers were written by LLMs, reports likewise, rebuttals too, even in some cases Area Chairs joining the flood. It becomes a potemkin village where nobody is actually accountable for anything that they are saying, and neither individual understanding nor our collective epistemic projects make *any* progress at all.
At this point, "researchers" are basically DDoSing classic academia. It's pretty sad to see. Either someone comes up with, and executes, a CloudFlare for academia, or it's toast. And the only thing i can think of that might have a chance to scale is more automation combined with a credit/point system like we've discussed a few times here on X over the years, but it needs simultaneous buy-in and coordination from all ~10 major AI conferences and journals, so it's a heavy lift. But if nothing big happens, i think it's game over soon.
9
16
6
92
17,054