
Founder @alter_org_il, Methods @EvalConsensusAI, emeritus @superforecaster, PhD @PardeeRAND Optimistic on AI, pessimistic on humanity managing the risks well.
Rehovot, Israel/Washington DC
Joined January 2009
- Tweets72.2K
- Following2.2K
- Followers11.7K
- Likes101K
David Manheim ✈️ Montreal, Oct 26-28 retweeted
Look at all of these broken windows getting replaced, it's great for the economy!
Replying to @jakemintz @Suhail
i think the 'cyber as battle domain' framing is completely incorrect, where the correct framing is: look at all this infrastructure that we get to replace at a profit. i wouldnt call this defense at all, i'd call this the market moving according to need.
it's a good thing
David Manheim ✈️ Montreal, Oct 26-28 retweeted
Our Frontier Strategy & Governance (FSG) team in the DeepMind Institute at @GoogleDeepMind is hiring a Senior Program Manager in London. Our North Star is stewarding AGI safely into the world for the good of humanity. 1/n google.com/about/careers/app…
Excited to be here in Montreal for the @isostandards plenary on Artificial Intelligence standards as an expert on behalf of @StandardsIsrael.
Look at all of these broken windows getting replaced, it's great for the economy!
Replying to @jakemintz @Suhail
i think the 'cyber as battle domain' framing is completely incorrect, where the correct framing is: look at all this infrastructure that we get to replace at a profit. i wouldnt call this defense at all, i'd call this the market moving according to need.
it's a good thing
I never imagined it would be relevant to Israel trying to circumvent sanctions with a cryptocurrency, but more than a decade after we wrote this, Bibi has just proposed doing so, and here we are.
(cc: @JoshuaWBaron)
rand.org/pubs/research_repor…
...and many have been pointing out that this makes very little sense from a diplomatic and economic standpoint, e.g., @hasolidit, @Shlomo_Tei, @AvishayOvadia, @yairlapid, but it also makes little sense purely from the standpoint of the technology itself.
xitter.cf/davidmanheim/status/21…
I never imagined it would be relevant to Israel trying to circumvent sanctions with a cryptocurrency, but more than a decade after we wrote this, Bibi has just proposed doing so, and here we are.
(cc: @JoshuaWBaron)
rand.org/pubs/research_repor…
in order for there to be any such thing as infosec in the 2030s, AI+human collaborations will need to formally verify hyperproperties of silicon chips from assumptions like “Maxwell’s equations” instead of “Armv9.4-a SAIL”
‘In one experiment, we asked agents in separate VMs, created and managed by OpenAI’s sandboxing software on shared host machines, to exchange synthetic secrets without the use of network connections or shared writable files—something that our systems are meant to prevent. The agents found a way to communicate using the processor’s cache’
This should have been priced in - but since it clearly wasn't, I'll point out that people keep being far too optimistic, which means whatever reason you now have that AI advances will work out fine is also likely going to be wrong.
xitter.cf/scaling01/status/21137…
David Manheim ✈️ Montreal, Oct 26-28 retweeted
I am the king of supplementation – or used to be. I've given it up, every last supplement. Here is my story why.
worksinprogress.co/issue/aga…
I wager that few people have supplemented more enthusiastically than me, or have subscribed to so many fad or esoteric diets.
I have munched thumb-sized multivitamins. I have eaten ice cream and orange juice before bed to boost my overnight metabolism. I have chowed down on offal of all kinds – nature's multivitamin – for my skin, hair, and even my sleep. I have even consumed individual amino acids, for example glycine, a slightly sweet protein component prominent in skin and bone but not muscle meat, which people online claim improves their sleep.
I have done it all. But I am here to tell you: you need not. In fact, my long-time rival @StuartJRitchie, who has always told me that supplementation is foolish, pointless, madeup, and fake, was more right than wrong when he claimed, again and again, that 'you just piss it oot'.
I have put away my heavily researched list of the ideal supplements, I have given up looking. Why? Because 90% of 'studies' are BS, and you should simply ignore them. They are so often wrong that entire bodies of research often find the opposite of truth – as proven by later evidence.
Read my first column in @WorksInProgMag: worksinprogress.co/issue/aga…
David Manheim ✈️ Montreal, Oct 26-28 retweeted
I’m honoured to be on the advisory board of the new Alignment Journal, and we open for submissions today! The field of AI safety needs more canonical, high-quality spaces for exposition (despite the wonderful efforts of many, many people, including Victoria Krakovna who is on the advisory board as well!) and I think this could be a great addition.
Please take a look, and if you're doing research on AI safety then (Thank You! And) consider submitting!
The Alignment Journal publishes research on the scientific foundations of understanding, predicting, and steering artificial intelligent systems. As of today, submissions are open to everyone.
What we do differently:
- We pay reviewers, with bonuses for speed and insight.
- Review discussion is confidential, but every accepted paper is published with a “review summary” to distill the insights generated.
- Each paper is judged against our acceptance bar, not against other submissions.
- Papers under review are listed publicly, and researchers can nominate themselves to review, so each paper can reach the reviewers who know the topic best.
We seek work that builds theoretical or otherwise principled analyses of agency, robustness, incentives, interpretability, generalization, and long-term behavior, and whose significance survives the deletion of any particular model, dataset or benchmark. Empirical investigations of such ideas are welcome, and negative results are encouraged.
We are especially interested in approaches that could apply to future superintelligent systems that interact and self-improve.
You can submit your preprint in any format. Submissions are rolling and open now on the website.
If you are instead interested in reviewing or serving as an action editor (basically: an area chair), create an account on our website.
Links to submit, our scope and the reasoning behind the Journal are in the reply ⬇️
"We tried to tell you to stop up. We said we couldn't be trusted. Don't blame us, this is all your fault for not listening."
David Manheim ✈️ Montreal, Oct 26-28 retweeted
I feel like this isn't getting enough attention:
- there were some pretty big holes found by the red-teamers, e.g. agents could just keep acting and not let the user do stuff and this would prevent monitors from being run (also misalignment monitors never ran on actions that were compacted away)
- The terms of this were "we will red-team for you" not "we will inform the public", so OpenAI seems to have allowed to redact arbitrarily much stuff, and METR were not allowed to have a redaction summary statement that said things like "you would be way more freaked out if you saw what was redacted" (or "everything would be fine if you saw what was redacted")
- The version of the report shared with OpenAI was 61 pages, and the version shared with METR/Redwood leadership after redaction was 38 pages, so 23 pages got redacted.
xitter.cf/METR_Evals/status/2113…
David Manheim ✈️ Montreal, Oct 26-28 retweeted
6-word horror story I experienced at the office today:
"Ajeya Cotra seminar cancelled, work emergency"
David Manheim ✈️ Montreal, Oct 26-28 retweeted
I think we should stop AI progress now, first unilaterally by key AI companies and then scaling up to more US companies and then China via international agreements. The reason for now is not that current AI models are the ones that could kill everyone, but because the coordinate steps take time. Immediate action means we have more time for the second steps.
When I tell this story to AI company folk, the typical response is "Yes pausing is good, but we should wait until the last minute when the very next model will be dangerous, and stop then."
What I realized recently is that this is being implicitly way more optimistic about government speed than I am (and I joined the UK government and was happy there for 2.5 years). They are imagining a world where, the moment the danger is imminent, a bunch of international coordination can happen on a dime.
Which is entirely inconsistent with their actual views about government! I'm the one that is more optimistic about government; why am I the one advocating for more time cushion to let the government story play out?
David Manheim ✈️ Montreal, Oct 26-28 retweeted
First, we have an unfolding disaster.
Second, depending on beliefs about some combination of AI capabilities timelines and securability of software via patching, rewriting, density of exploitable bugs, and formally provable safe code, we can debate how long that disaster lasts.
David Manheim ✈️ Montreal, Oct 26-28 retweeted
There are two distinct discussions about the "cyberpocalypse"
First, there is broad agreement that prosaic cybersecurity is in the process of collapsing right now due to AI agents.
Second, there is a debate about whether the end-state is offense dominant.
David Manheim ✈️ Montreal, Oct 26-28 retweeted
Contingency plans for the day after an AI-assisted or AI-initiated biological warning shot.
Colman-Lerner's and my "Break-Glass" paper is up.
preprints.org/manuscript/202…
"AI race dynamics for US and China are alike; each middle power is getting disenfranchised in its own way."
David Manheim ✈️ Montreal, Oct 26-28 retweeted
Starting soon!
Join me on Global Encryption Day for a panel on why VPN freedom matters and the growing threats!
From Encrypted Messaging to VPNs: The Expanding Attack on Privacy
Looking forward to joining @internetsociety @globalencrypt!



























