Your answers to the questions on alignment & safety are deeply problematic. Analogy: Let’s say some startup was going to create a novel virus that could be an unprecedented super-pandemic if designed wrong, or an unprecedented cancer cure if designed exactly correctly. That way (they say) we can better understand it in a safe lab setting, and only release it once we’re confident its effects will be good not deadly. Normal people, hearing this plan, would have a WHOLE LOT of questions. Like: What if the virus gets released accidentally? What BSL level are you working under? You know that even BSL-4 has had lab leaks right? You recognize that working at BSL-4 is a GIANT pain that might slow progress by >10× and raise costs by >10×, right? Your employees and investors are all enthusiastically on board with that? What if the virus doesn’t escape, but hackers steal your IP so they can try themselves, with less caution? Or an employee leaks the IP? Or an employee simply has a difference of opinion with their coworkers about how safe it is to release the virus, and departs to raise their own money and do their own thing in California, carrying all the IP in their head? What if the investors get impatient and ask for proof of progress, and then an intern at the VC company leaks the IP? Or merely leaks a rumor of your progress, which encourages other (probably less careful) groups to race down the same path? How will you know whether you have a super-plague or a cancer cure in your BSL-4 box? What if you get to a stage where you aren’t sure whether it’s a super-plague or a cancer cure? Or where you have a super-plague that you don’t know how to turn into a cancer cure? What if you’re stuck in that limbo for months? For years? For decades? And your investors are threatening to pull the plug? And your employees are threatening to quit? And competitors are catching up? Are you really sure that you won’t engage in wishful thinking and dangerous experiments in order to move forward under intense pressure? If investors do pull the plug, or employees quit, will you shred all the IP, or will the IP find its way to other (probably less careful) groups who will then race down the same path? Why can’t you just figure out how to get to a cancer cure in such a way that you will definitely not ever be creating super-pandemic viruses in the process, even if those viruses are supposedly contained in a BSL-4 box? What are the unanswered questions that make such a safer path impossible? Please list those questions, and for each question, explain why there’s no possible way to make progress on that question without actually creating a super-pandemic virus, even a supposedly contained one. And conversely, also explain why, if you DO follow this extremely risky research path, why it will definitely answer that question, as opposed to subjecting us all to grave risk for minimal benefit. If you don’t have excellent answers to all those questions and more, answers that have survived rigorous outside scrutiny, answers that every last one of your investors and employees are 100% enthusiastic about, then governments should not be allowing you to do this project, and you yourself should not WANT to do this project. …That was an analogy, but I claim that all those concerns above transfer 1:1 to your “science of superintelligence” plan. Just replace the words “BSL-4” with “RAND SL-5 / airgapped sandbox”, and “super-pandemic virus” with “misaligned superintelligent AI”, and “cancer-curing virus” with “friendly superintelligence”. More here: xitter.cf/steve47285/status/2109…

even worse / dumber than that: last week one of my Facebook friends used “became extinct” as a more emphatic way to say “died” 🤦