New post: Data bottlenecks won’t block an intelligence explosion One of the most important questions facing the world is whether AI will undergo a fast intelligence explosion and quickly automate most real-world work. A key objection people raise is *data bottlenecks*. Data is a crucial input to AI training - how could it increase fast enough for such explosive progress? I tried hard to find strong bottlenecks, but ended up sceptical that they would stop an intelligence explosion from accelerating over time. Here are the most compelling bottlenecks I investigated and why I wasn’t convinced. **Bottleneck #1**: you’ll need millions of trajectories of every job, which will take years or decades to collect Today’s AI algorithms are very data hungry. You would need a huge amount of data to automate the full economy. But the plan isn’t to use today’s AI algorithms. The plan is to do a software intelligence explosion inside a datacentre, where AI becomes superintelligent at AI research - and have that AI produce a highly **sample-efficient learning algorithm.** Then you don't need millions of trajectories per task. You need as many as a human learns from, or fewer. And the data can be messy, just like the data humans learn from. That data could be gathered in a few months. Sceptic: *“But what about sim to real transfer? Why not expect a “Goodhart Singularity” (@mentalgeorge), where learning algorithms discovered using virtual tasks fail to really translate to the real world?”* People get confused here. **Trained neural nets** generalise badly. A neural net trained on Chess generalises badly to Go. But **learning algorithms** generalise very well. The same learning algorithm that masters chess can also master Go (AlphaZero). The Transformer architecture was developed for processing text, but it also works for sounds, images and controlling agents. The human learning algorithm was "designed" for hunting on the Savanna, but it also works for String Theory. So if AI designs a sample efficient learning algorithm from inside a data centre, it will very likely translate to real-world tasks. (And AI will have access to some real-world tasks so can check and iterate!) At this point the sceptic might retreat: *“Fine, data won’t be a bottleneck if there’s a software intelligence explosion in a data centre. But data bottlenecks will prevent that from happening in the first place!”* So let’s focus on bottlenecks to a software intelligence explosion. **Bottleneck 2: AI progress has ridden exponential growth in training data. That can't continue, let alone accelerate.** But forecasts of a software intelligence explosion extrapolate software progress. And software progress just is: getting the same capabilities from *less* compute and *less* data. So the core engine of the explosion doesn’t rely on increasing the amount of data at all. **Bottleneck 3: improving data *quality* relies on human experts, and that won’t be possible once AI is smarter than humans** The core engine of the software intelligence explosion *does* rely on improving data quality. Today we augment high-quality internet data, pay humans for expert trajectories, and build RL environments where AI has to replicate human-built software. But all these methods *extract* quality out of an existing human reservoir. Above human level, the reservoir is empty. AI will have to *manufacture* higher-quality data than any that exists, and do so from scratch. This will slow the intelligence explosion. But it won’t stop it in its tracks. We can already produce superhuman data quality via RL. And AI could produce higher-quality trajectories by thinking for longer. Crucially, this won’t stop the intelligence explosion from **accelerating**. Losing the human reservoir makes it harder to get from AGI to AGI+ — but it makes it harder to get from AGI+ to AGI++ by roughly the same amount. The handicap is the same at every stage. So if you previously expected each step to take less time than the one before, you should still expect that. Every step might take 50% longer, but progress still accelerates. **Bottleneck 4: paradigm tax** AI capabilities are spiky. When AI first matches humans at AI research, it will be much stronger than humans in some ways and much weaker in others. More parallel copies, faster thinking — but weaker generalisation and weaker sample efficiency. Which means AI's human-level performance will initially lean heavily on a mountain of human writing about transformers. When AI then discovers new techniques — no such mountain will exist. No textbooks, no blog posts, no code. Deeply understanding a technique is harder than stumbling upon it — we understand general relativity far better now than Einstein did in 1917. So AI may match humans on the techniques it inherited, but fall below humans on the ones it creates. This paradigm tax can be paid. Eg, AI can generate millions of trajectories to illustrate new techniques. But paying the tax is a real cost that slows down AI progress. Again though, this won’t stop the intelligence explosion from **accelerating**. It makes it harder for AI to master whatever replaces the transformer — but it makes it harder to master whatever replaces *that* by roughly the same amount. Every new paradigm arrives without a corpus. This makes each step take longer than you previously thought, but it doesn't change whether each step is faster than the one before. **My bottom line:** Data bottlenecks will slow the early stages of a software intelligence explosion, but won’t stop it from accelerating over time. And then they won’t stop AI from quickly automating most economic work thereafter. Read the full post for more! newsletter.forethought.org/p… Thanks for comments from @dwarkesh_sp @RyanGreenblatt @testingham @pawtrammell @1a3orn @herbiebradley @tobyordoxford @joshua_clymer @_achan96_ @sorenmind @daniel_271828  @sj_manning @DanCareyEcon @eli_lifland @romeovdean @bphalstead and others

New paper: I explore the maths of the most explosive possibilities of Recursive Self-Improvement (RSI), trying to understand exactly what features allow intelligence to grow towards a vertical asymptote (a mathematical singularity). 🧵