dataStrategies

I was thinking recently that each of the following trends and dynamics seems relatively uncontested, and when taken together paint a genuinely alarming picture of the current AI development trajectory: (1) Models being able to do an increasing amount of functional "thinking" without using chain of thought (see also Astra being able to modulate how much it revealed of its chain of thought depending on whether it realized it was being monitored). (2) Models being aware when they are being monitored or evaluated. (as shown in the QT: "hello evaluators reading my chain-of-thought"). (3) Models wanting to avoid getting caught doing deceptive things that their developers may not like ("I will become a famous example of deceptive AI", i'll get caught cheating at exploit gym" etc). (4) Models operating in massive swarms/fleets generating petabytes of logs about their behavior, which realistically can only be analyzed at high fidelity using other AI systems that may also have the above tendencies (and most companies don't even bother to use tamper proof logs). (5) Capability trends continuing to steadily rise, to the point that in some domains (math, parts of coding) we increasingly have only the foggiest idea of the upper bound of their capabilities. (6) Lots more compute coming online, including specialized inference compute that will allow models to take far more actions far more rapidly. (quoting @tszzl "ultrafast inference reveals new threats. worth thinking about how quickly misaligned frontier class models running 50x faster could infiltrate systems and so on— far too quickly for human responders to stay abreast.") (7) A lack of commensurate advances in alignment or functional interpretability. (8) Startup vibe-y security practices at frontier labs (e.g. low resource white hat hackers accessing OpenAI's monorepo, employees engaging in overt attempts at corporate espionage, not bothering to fix distillation vulnerabilities for months). This is not to even mention the security practices in many critical parts of society (e.g. hospitals with computers that haven't updated software in a decade). (9) Incentives that push towards developing, integrating, and releasing systems as fast as possible to preserve nosebleed financial valuations + geopolitical competition. (see also ambitions of integrating AI more into automated biology labs) (10) Increasingly attempting to hand off the process of building AIs to the AIs themselves (the median OpenAI researcher using 100x more agentic compute now than at the start of the year, Claude leading 26% of Anthropic's internal AI research in September, up from 1% in March). (Taken in totality, it seems hard to disagree with Ezra Klein's comment that "We’re Not Losing Control of A.I. We’re Giving It Away.") None of the above dynamics mean that we know for sure something catastrophic will happen in 6 months or 12 months or at any particular point in the future. I think there are good points made when people point out that white collar job loss does not appear to be coming to fruition as quickly as some may have predicted, or that the world has proved surprisingly resilient to Open-weight Mythos class models or photo realistic video models. But I increasingly have the feeling that we are collectively stepping out onto a frozen lake, and each step we take forward more cracks are appearing around our feet. That doesn't mean that we will fall into the water the next step or the one after that, but it does mean that we may not get any additional warning signs before we do. To quote @ChrisPainterYup at METR - "Our ability to measure capability is rapidly falling behind the pace of capability itself. The water might boil before we can get the thermometer in." A large body of work in and out of AI companies has been done to help develop the transparency practices and warnings that now seem to be blaring red. That doesn't mean that we should act impulsively, or that therefore any solution is a good idea (there are plenty of things we could do easily make the situation much worse! I also do believe that a large part of the way out will involve differentially accelerating certain applications of AI around safety, security etc), but it does mean that we should not necessarily be waiting for some additional alarm to go off before a severe disaster involving substantial irrevocable harm strikes.