🚀 𝗥𝗲𝗴𝗶𝘀𝘁𝗲𝗿 𝗻𝗼𝘄 𝘁𝗼 𝘁𝗲𝘀𝘁 𝘆𝗼𝘂𝗿 𝗟𝗟𝗠 𝗳𝗼𝗿𝗲𝗰𝗮𝘀𝘁𝗲𝗿 𝗮𝗴𝗮𝗶𝗻𝘀𝘁 𝘁𝗼𝗽 𝗵𝘂𝗺𝗮𝗻 𝗳𝗼𝗿𝗲𝗰𝗮𝘀𝘁𝗲𝗿𝘀 ForecastBench is the leading LLM forecasting benchmark. This month we're running a new survey of superforecasters—setting a new baseline for top human forecasters on ForecastBench. Submission details👇 𝗢𝗰𝘁 𝟮𝟯: Registration deadline for new participants 𝗢𝗰𝘁 𝟮𝟱, 𝟬:𝟬𝟬 𝗨𝗧𝗖: Question set release 𝗢𝗰𝘁 𝟮𝟱, 𝟮𝟱:𝟱𝟵:𝟱𝟵 𝗨𝗧𝗖: Deadline for submitting LLM forecasters 𝗝𝗮𝗻𝘂𝗮𝗿𝘆 𝟮𝟬𝟮𝟳: We'll publish the fresh ranking of AI models vs superforecasters

🚀 Can your LLM forecaster outperform the best human forecasters? ForecastBench is the leading LLM forecasting benchmark and the only one that compares AI forecasters with superforecasters. This month we're running a new survey of superforecasters—setting a new baseline for top human forecasters on ForecastBench. If you're building an AI forecaster, now is your chance to test your system against the very best human forecasters. Submission details in the thread 👇

AI experts largely agree about which policies would most reduce catastrophic AI risk. The Longitudinal Expert AI Panel (LEAP) brings together forecasts from a representative group of over 250 experts that policymakers look to when considering the future of AI: computer scientists, industry figures, top economists, policy experts, and AI risk experts. We asked panelists to forecast (1) how a menu of policies would affect AI risk; (2) which policies they each support; and (3) which policies they think will be implemented by 2030. Key findings: • Despite disagreeing on the level of AI risk, respondents broadly agree on policy • Respondents believe government policy can substantially reduce catastrophic AI risk • The most supported policy of those we asked about was the creation of an international body, including the US and China as members, with pre-release authorization power over frontier models • Some US-only policies are forecast to reduce catastrophic AI risk, but not as much as bilateral policies involving the US and China • Experts strongly oppose federal preemption and think it would increase AI risk • The policies experts expect to reduce risk the most are those they think are least likely to actually be implemented 🧵 More, with key insights below:

We're improving the accuracy and relevance of the AI progress forecasts we publish. Last week, we showed how the median expert, superforecaster, and member of the public in our studies have consistently underestimated AI capabilities progress (particularly on benchmarks) and have a mixed track record on other indicators. Our findings suggest that median forecasts from these groups have not accurately predicted the trajectory of AI on some questions. How might we get a better grasp of the trajectory of AI? We're working on three major changes to the Longitudinal Expert AI Panel (LEAP) to address this question: 1. Sharing forecasts from a subsample of panelists who expect fast AI progress by 2040, for comparison 2. Adding regularly updated LLM forecasts to LEAP findings 3. Highlighting forecasts from the most accurate LEAP participants as soon as we have data to make this judgment rigorously