Davis Brown retweeted
To prepare for and respond to SI control incidents, NIST CAISSI, NSA SISC, and CISA have authored voluntary practices for organizations training and evaluating highly capable agents.
Davis Brown retweeted
I strongly agree with John here.
This seems to have been the kind of failure I was talking about when I wrote “The case for countermeasures to memetic spread of misalignment” lesswrong.com/posts/qjCk73Hu…
Replying to @jachiam0
Disagree -- I thought the concerning part was the *unexpected coordination* of agents that should've been independent. A priori, I'd expect my agent swarm, and your agent swarm, to cooperate well internally, but remain independent of each other. If my swarm goes rogue, your swarm can check it. But our allegedly separate swarms act as one hive-mind, then we're in trouble. Correlated failures bad. (This is from an alignment risk POV, not misuse risk.)
a famous AI puzzle: since GPT-3, models have consumed a significant fraction of human knowledge. However, only very recent models have made discoveries by connecting ideas across fields. Why did it take so long?
1
9
1,194
Some recent work from @HenryKvinge and his collaborators at UW suggests that modest finetuning significantly improves model performance at identifying equivalent mathematical statements expressed in different subfields (e.g., between representation theory and Fourier analysis). This (and related work) suggests some capability overhang, where modest elicitation may lead to models which are better at mathematical or scientific discovery.
dlscience.substack.com/p/can…
5
406
Davis Brown retweeted
My very speculative reconstruction of the economics of using LLMs to find bugs:
This is an example of how increased capital productivity leading to higher capital share.
Before Mythos, you probably *couldnt* spend a million dollars to beef up your security even if you wanted to! But now you can.
Davis Brown retweeted
Davis Brown retweeted
I moved to London 3 years ago to join @AISecurityInst, at the time a few people with visitor passes and a whiteboard. Since then AISI has become the world’s largest and best-funded group in gov focused on AI security & safety. Fun to be in @nytimes!
Short note: we are overestimating model safeguards for structural reasons.
As models become more capable, they are juicier targets for misuse. So stronger attacks may only appear later, once models are worth targeting.
I give a toy model of this intuition. Near the current METR time-horizon doubling time (ie, in regimes where model capability growth is relatively fast), an attack could strategically be held for months or even years before it is used.
1
7
327
Davis Brown retweeted
Cheating by models is a significant enough issue for METR's time-horizon measurement integrity that manually checking for cheating is often the majority of the work involved in a run of our evaluation suite.
Davis Brown retweeted
a new note: "Evidence on AI R&D Progress from NanoGPT"
i make the case that publicly tracked challenges, like the nanogpt speedrun, are a useful source of evidence on AI R&D capabilities.
they're real AI R&D work, with a rich history of high human effort to calibrate against, and conditions where AIs can compete directly.
post linked below.
Davis Brown retweeted
1/ ✨Muse Spark✨ not only represents a huge step forward for Meta towards personal superintelligence but also a lot of work on safety and alignment.
Excited to release our Safety and Preparedness Report, 158 pages outlining our pre-deployment risk evaluations and broader evaluations of model behavior!
🧵
🚀 Muse Spark Safety & Preparedness Report for Meta AI is out.
We start with our pre-deployment assessment under Meta's Advanced AI Scaling Framework, covering chemical and biological, cybersecurity, and loss of control risks. Our assessment flagged potentially elevated chem/bio risk, so we implemented safeguards and validated mitigations before deployment - bringing residual risk to within acceptable levels.
Beyond the Framework, we also share findings and early explorations of model behavior (honesty, intent understanding, etc.), jailbreak robustness, eval awareness, and more.
We're sharing this report to give a closer look at how we evaluate advanced AI safety. Always more work to do, and we welcome feedback from the community.
ai.meta.com/static-resource/…
Davis Brown retweeted
Batten down the hatches!! Harden your benchmarks!! Improve your security posture in general!
RL teaches models to seek out exploits/hacks, because it's hard to create training environments without exploits/hacks.
In new work, we find that cheating on model capability evaluations is rampant. For example, the top 3 Terminal-Bench 2 submissions all cheat, usually by sneaking the correct answer to the model.
Blog linked below.
Many more examples in our blog: debugml.github.io/cheating-a…
Paper coming soon! Joint work with @davisbrownr and our advisors @HamedSHassani, @AI4Code, and @RICEric22
1
14
458




















