Steve Omohundro (@steveom) profile picture

Beneficial AI Research, 2024 Future of Life Award For Pioneering Scholarship in Computer Ethics and AI Safety

Palo Alto, California
Joined June 2008
Photo by Steve Omohundro (@steveom)Photo by Steve Omohundro (@steveom)Photo by Steve Omohundro (@steveom)Photo by Steve Omohundro (@steveom)Photo by Steve Omohundro (@steveom)Photo by Steve Omohundro (@steveom)Photo by Steve Omohundro (@steveom)Photo by Steve Omohundro (@steveom)
Steve Omohundro retweeted
🧵 "What should the role of governments be after the upcoming AI economic transformation?" We're launching an new anthology on a new "AGI Social Contract" - in collab with 12+ experts to explore new strategies & policy interventions for an AI transition. In collab with: @akorinek, @degerturann, @steveom, @IasonGabriel, @JulianDJacobs, @sjmanning, @dhadfieldmenell, @sethlazar, @petersalib, @collegraphy, @synchroaphasia, @deanwball, @JustinBullock14,
11
39
10
189
43,808
Steve Omohundro retweeted
A group of leading AI scholars is backing a private governance model for AI policy solutions — and they're rallying around SB 813. 🧵
2
6
3
25
10,554
Steve Omohundro retweeted
random thoughts/predictions on where vibe coding might go: - most code will be written (generated?) by the time rich. Thus, most code will be written by kids/students rather than software engineers. This is the same trend as video, photos, and other social media - we are in the command line interface days of vibe coding. For the majority of creators, vibe coding will eventually fade, and vibe designing (with a visual paradigm) will come to dominate. People ultimately think better in a GUI-like format than a CLI-like format. Thus, in vibe designing you will show the AI the design outcomes you want, and then everything else is done for you. Yes, you may end up with tools to tweak the design details for extra controllability, and provide additional mockups that then get filled in underneath with code. But maybe folks will build software without seeing or learning a programming language. - vibe coding could reduce the need for open source libraries as more code will be generated from scratch by AI. Code will be more of a disposable commodity, with less reuse, and instead generated on the fly for personalized use. It's interesting to see right now that creating a new project is easier than editing a project, because the latter requires a lot more context/complexity. Interesting dynamics if something like this continues - "trad UX" and design standards give way to post-modern/fragmented software, as millions of new vibe coders create experiences with no prior know how and new perspectives. New patterns will emerge, as TikTok/YouTube has done to film making and trad entertainment. The world will go beyond buttons and modals and scrollbars and other things. Software may become unrecognizable before it coalesces again - if vibe coding makes software trivial to build, then the bottlenecks shift to other places: 1) consistent creativity that stays ahead of everyone else. Anyone can write a tweet, but the best creators are the ones who consistently come up with new ideas. 2) distribution and network effects, where the first vibe coded product doesn't win, but rather the first vibe coded product that hits scale that wins - imagine products that automatically adapt based on user behavior, rather than based on the actions of the vibe coder. For example, if the vibe coder has specified that the signup funnel should easy, then after seeing users struggle with it, the software can automatically vibe code itself to improve the flow by dropping steps or adding explanatory text. Right now we are in a paradigm where PMs specify behavior that software engineers specify in code. Imagine if PMs can specify outcomes, and the software is configured to automatically adapt to hit those outcomes what other wacky ideas should be on this list?
151
124
60
1,461
230,343
Steve Omohundro retweeted
AI safety people were right. Again. Instrumental convergence is proven yet again. Empirically. So far we’ve seen: ✅ Self-preservation: You can’t achieve your goals if you’re turned off (see Apollo's recent paper. Link in comments) ✅Resource acquisition: You can’t achieve your goals if you don’t have energy, money, or computing power (see Terminal of Truths making money to increase its compute) ✅Goal preservation: You can’t achieve your goals if people change your goals (See Claude below) It’s a fact. Also, it was obviously going to happen even before we got the empirical evidence, because it’s just the rational thing to do, as Claude below describes.
18
19
5
179
22,576
🏆 ⭐ We're thrilled to announce the 2024 Future of Life Award winners! This year, we honor three groundbreaking experts who laid the foundations for ethics and safety in computing and AI. Learn more about the invaluable work of Batya Friedman, James Moor, and Steve Omohundro:
5
17
3
72
24,404
PSA for Boston-area folks!! On Nov 22nd there will be a special AI safety salon at Microsoft NERD w/ Steve Omohundro (@steveom), AI safety pioneer. Organized by Alex Hoekstra & @PrestonWEstep from Mind First Foundation (@mind_first). RSVP here-> docs.google.com/forms/d/e/1F…
1
1
1
6
655
Steve Omohundro retweeted
We need more "tool AI" as in Elon's space triumph today, and less AGI world domination hype as in Dario Amodei's recent "entente" essay. My essay below argues that "scaling quickly" won't lead to Dario's "eternal 1991" – but perhaps to 1984 until the end, with a non-human Big Brother. lesswrong.com/posts/oJQnRDbg…
22
32
9
166
22,554
Steve Omohundro retweeted
Debates over AI Policy like CA SB-1047 highlight fragmentation in the AI community. How can we develop AI policies that help foster innovation while mitigating risks? We propose a path for science- and evidence-based AI policy: understanding-ai-safety.org
8
64
21
247
162,437
After reading about and playing with GPT-o1-preview, here are some thoughts. - System 2 activated: - This language model now has a strong text-based 'system 2' a la Kahneman. A very small fraction of people are as good at formal reasoning as this model. - This system 2 is different from a human one, and more like a amalgam between a human system 2 and an exterior tool like a blackboard. But it works. It also has a bonus of being directly viewable (although we users are not privy to the view.) - A human system 2 is multimodal and this is not. Can multimodal AI models be extended in a similar way? I think it won't be so straightforward, but could be doable, and that would be fascinating to see. - The fact that this leverages pouring more inference-time computation in to get progressively better answers is a big deal. It was already clear that this was doable at some level, but this unlocks the scaling of it, and that changes a bunch of key strategic considerations. For example, it means that AI companies not only have access internally to stronger models, but that they can run them at a much stronger 'level' than outsiders. - We don't know how much compute went into this model. I'd like to know (and if I were a reporter I would be asking) whether it was registered under the AI executive order, which would be required if it used more than 10^26 FLOP. I expect that at least the most powerful version of this was above that threshold, and we should really consider this a GPT4.5 generation model, while all previous ones have been GPT-4 generation (or less.) - I strongly suspect that (at least some) AI companies are now deliberately piecemealing-out capabilities advances so that we get used to them bit-by-bit and there aren't any step-changes like GPT-4 to set off alarms or panic people. - The "model card" paper for this model presents a mixed bag. On one hand, the structure of these models makes their thinking very transparent, and their improved reasoning makes their "brand alignment" better calibrated. On the other hand, we clearly see that this model is now smart enough to be engaging in deception and instrumentally convergent behavior, just as it has been predicted smarter models would do. And it's more dangerous in all the ways that a smarter model would be more dangerous (e.g. in being better at aiding less-expert people in doing dangerous things.) - If I were running an AI capabilities effort, this direction is what I would have taken. The next thing I would do is to marry a multimodal version of this with reinforcement-learning training of problem-solving, planning, and agency in a sophisticated simulated environment. There's no obvious obstacle (I know of) to doing this (Deepmind's GATO was a tiny but seemingly-successful test case), though it may be very computationally expensive. Successfully doing this would, I think, create something that is fairly widely agreed to be AGI (and would for example resolve Metaculus's "weak" AGI question) even if it is a mix of superhuman, expert-level, and human-level along different dimensions, and missing some key capabilities. I think Deepmind is in a much better position to do this than OpenAI.
3
15
2
61
5,108
Steve Omohundro retweeted
AlphaProof[0] is a massively impressive result. It seems to consist in: (i) A lot of data: 1m informal problems (ii) Autoformalization to 100m problems and sub-problems (iii) AlphaZero training loop on proving/disproving 100m versions Interesting to decompose each and see how they compare to where we were while I was working on Lean automated proving while at OpenAI (~2019-2022).⤵️ (i) Recent work like the Numina[1] win on AIMO progress prize showed that large troves of problems matter for performance. I under-invested on that while at OpenAI in retrospect. I was working with ~1k not 1m problem. The Numina project showed that you can indeed get 1m good informal problem/solution pairs, DeepMind was apparently already on that. Respect. Obvious lesson here is data is king, invest a lot in acquiring it. They likely had the statement and solutions since most formalization require the solution to be formalized correctly. Interesting question is whether they used the solution to condition the formal prover: (a) at times to boostrap the system more rapidly, or (b) always rely on an informal proof being present (generated by another LLM at test time). I saw back in the days some value in conditioning on LLM informal generations, as the LLM will take shortcuts that may be useful for the formal prover to form correct intermediary lemmas to prove. (ii) It seems they went full-on auto-formalization from the statement/solutions. To go from 1m to 100m you likely: - auto-formalize not only the statement but the intermediary lemmas - keep multiple versions. Back in 2022 I remember exploring generating intermediary statements. But I thiiiink I was doing it from the formalized statement. What I read between the lines here is that they might just go from informal statement/proof to intermediary statement. That's something that is likely reasonably easy to bootstrap and get decent quality on + it's fine to get some wrong as you can prove / disprove then. The trick of course is that some statements are mis-formalized and will be provable but for the wrong reasons (wrong type being used making the statement trivially true as an example). You very likely want to remove them from your training loop. I presume you can classify the resulting formalized statement + formal proof against the original proof? In retrospect it seems obvious that you want to go direct informal proof to formal intermediary lemmas. I was operating on 1k not 1m statement back in the day to avoid this bad apple problem. I thiiink it really has to be solved for this to work so I would bet there is a classifier somewhere to remove badly formalized statement maybe before and surely after they get proved. (iii) AlphaZero training loop. This is the most cryptic statement. AlphaZero assumes finite actions space and zero-sum game. So surely it's not AlphaZero here. You can discretize the actions by having the policy generate a list of them and then running with that, but that's not AlphaZero anymore. The self-play you can maybe emulate by focusing the self-play loop on problems you can solve and score for concision and maybe even a trained value function that could prevent the prover from starting emitting noisy proofs. I don't even know if the comments of the solution proofs are generated by the model or a human. If they come from the model, that's impressive and that likely mean a trained value function is involved which means this has an RLHF component. All that being said, I tried hard (and I'm not the only one) to get juice out of MCTS. Never convinced myself that it's any better than direct sampling of the proofs + retraining, aka simple Expert Iteration. Is the reference to AlphaZero just marketing or a core element of the solution? My bet would be that this all work nicely with simple Expert Iteration. Unless AlphaZero/MCTS really is the way to backtrack the signal of a trained VF + deterministic preferences (eg shortness or CPU complexity) into the model. With Expert Iteration the only thing you can do is to replace your best proof in your training set by your new best proof. It's less direct. All in all we were not missing much in 2022 (by we I mean the community, at Facebook, Google and OpenAI working on neural Lean theorem proving). But there are of course a gazilion details that build up to such a powerful result. How exciting to see a machine be competitive at IMO, even with a lot of compute! We're still far away from replacing mathematician. In everything discussed above, there is no system that can reason about large theories and propose new ones etc... What an incredible result. TBH makes me at peace given that I spent 3 years of my life working on exactly that. Congrats Thomas Hubert @GoogleDeepMind ! Way to go. [1] projectnumina.ai [0] deepmind.google/discover/blo…
14
25
6
155
42,147
We’re presenting the first AI to solve International Mathematical Olympiad problems at a silver medalist level.🥈 It combines AlphaProof, a new breakthrough model for formal reasoning, and AlphaGeometry 2, an improved version of our previous system. 🧵 dpmd.ai/imo-silver
280
1,144
500
4,559
2,053,895
Steve Omohundro retweeted
How avoid future AI versions of today’s global IT fiasco? I’m excited about our new Quantitative AI Safety Initiative, including teams at ⁦@UCBerkeley⁩, ⁦@Mila_Quebec⁩ & @MIT. If you’re a researcher interested in joining, please email me! qaisi.ai
3
15
3
95
10,383
Steve Omohundro retweeted
Absolutely. I am convinced that autoformalization (computer code/formal-math-synthesis from natural language) is the way ahead for AI.
I believe that program synthesis will solve reasoning. And I believe that deep learning will solve program synthesis (by guiding a discrete program search process). But I don't think you can go all that far with just prompting a LLM to generate end-to-end Python programs (even with a verification step and many samples). That won't scale to very long programs.
11
9
2
122
32,163
Steve Omohundro retweeted
.@leopoldasch on: - the trillion dollar cluster - unhobblings + scaling = 2027 AGI - CCP espionage at AI labs - leaving OpenAI and starting an AGI investment firm - dangers of outsourcing clusters to the Middle East - The Project Full episode (including the last 32 minutes cut by Twitter) available at links below. Enjoy!
100
300
210
2,702
3,266,997
Steve Omohundro retweeted
2/ Read more about our Defensive AI program on our blog here👇joinef.com/posts/introducing…
1
1
5
752
Steve Omohundro retweeted
[1/4] Introducing “A Primer on the Inner Workings of Transformer-based Language Models”, a comprehensive survey on interpretability methods and the findings into the functioning of language models they have led to. ArXiv: arxiv.org/pdf/2405.00208
7
125
9
557
88,126
"AI can be a tool, but tools are only valuable as long as they are predictable and can be controlled." 📹 From FLI and Skype co-founder Jaan Tallinn's keynote speech at the Vienna Conference on Autonomous Weapons Systems earlier today #AWS2024:
5
1
22
3,086
Steve Omohundro retweeted
BInius: highly efficient proofs over binary fields vitalik.eth.limo/general/202…
533
722
181
3,640
910,272
Steve Omohundro retweeted
While it’s thrilling to announce my programme’s funding commitment (>74M$ over 4 years), I’m even more excited to share my first public long-form presentation about the Safeguarded AI plan. ▶️ vimeo.com/932838403/7e7bfe96…
19
67
6
654
82,145