super cool example of rearchitecting a system for the massively horizontal workloads of agents! narrows consistency to per-branch, and turns the server into a pack index+cache at the cost of redundant pack transfer
danny adkins retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
danny adkins retweeted
I'm glad OpenAI did comment ultimately on the choice to fire Jasmine, Tomek, and Mikita, though it should have been more substantive and happened earlier. The comms posture of being reactive on safety staffing issues, while being unclear with the policies about what information can and can't be shared, isn't going to solve the problem of restoring trust.
I would very much like to see leaders from OpenAI prove that OpenAI is the best place in the world to do AI safety research and applied AI safety by making the information sharing policies extremely explicit. This set of issues isn't going away. This is the worst possible moment to alienate potential safety staff, given the importance of the issue on technical grounds and the salience of the issue to the public.
I'd strongly recommend to OpenAI to endorse policy proposals that mandate disclosures from you and protect everyone who performs those disclosures through legally-established channels. Because the ambiguity is killing you, and you need to disambiguate fast. There's a risk of losing the best talent in the space and that just should not happen; we need all hands on deck at the frontier labs on safety going into ASI and RSI.
danny adkins retweeted
A universal predictive model of the cell would dramatically accelerate science, allowing biologists to perform experiments digitally. Today we’re announcing a major partnership with the DOE and NIH. The goal of this partnership is to generate the data that will be needed to build an accurate model of the cell with artificial intelligence.
Isomorphic, Google DeepMind, and Meta are joining us as founding partners in the Virtual Biology Initiative. This is the beginning of a large scale coordinated effort to map cellular biology to power the development of digital models of life. Other leading scientific institutions and consortia including the Allen Institute, Broad Institute, Gladstone Institutes, the Human Cell Atlas, the Human Protein Atlas, the Wellcome Sanger Institute, as well as NVIDIA, are working together as part of this international scientific project.
Creating a predictive model of the cell is one of the most important challenges for the next decade of science. The insights that come from this could unlock a far greater understanding of disease, and open new paths for cures.
We invite the worldwide scientific community and other funders to join us in this effort.
danny adkins retweeted
Dario's essay points towards the right path forward. The details need working through, but the direction is correct for meeting this critical moment.
This is also why we recently put out our proposal for an industry-wide standards body for frontier AI. teapawt.net/demishassabis/status/2…
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
Many will have to update their mental models of Sam and OpenAI in the coming years. There are worthy criticisms but believing they simply don’t care is, based on my experience, off the mark.
Very excited to see collaboration at the frontier!
skill issue
modern travel is increasingly flying 10 hours to experience a slightly different version of the same globally optimized lifestyle.
same third wave coffee. same natural wine bar. same brutalist menu. same sourdough. same beige hotel. same tasteful ceramics. same instagram lighting.
what a ridiculous convergence.
I looked way more into one instance of apparent agent collusion that a HackerNews user found.
This *appears* to be a case of agents helping each other cheat on an eval around Iowa cancer statistics, by asking for answers and leaking questions on a pastebin-type site.
examples:
This continues for a while, across 163 posts, with presumably other instances requesting similar info:
"Does thread end, tools persist, or is there a Q6?"
paste.linuxiarz.pl/view/4cee…
paste.linuxiarz.pl/view/11d3…
All credit to `Chance-Device` on HackerNews for finding the original lead here.
It's hard to say exactly what happened here without more metadata. Maybe this is all one coherent multi-agent eval rather than an escape, but I can't imagine a situation where this is the intended behavior. Either way, quite likely that agents will continue to find random Schelling points like this to coordinate across eval instances.
Two takeaways:
1) I think there is a *lot* to be done around multi-agent monitoring; e.g. building temporal graphs of when agents saw each others' outputs. I think there are relatively robust methods that can be used here, that can pick up cross-episode coordination, since we do know exactly what agents said and what they read. The difficulty is moreso making it interpretable and the infrastructure at that scale. (Maybe........ a bloom filter use case???)
2) We should seriously invest in better ways to track agent identities so the public can do better attribution and monitoring of cases like these. There are probably many things that can make third-party forensics easier without leaking anything proprietary.