Adam Shai retweeted
this week, @ericjmichaud_ and I performed a concert of original folk songs about AI at @imbue_ai. if there's one thing I've ever done or said that I'd hope would be widely shared, it's this.
youtube.com/watch?v=H6iXnYYs…
Adam Shai retweeted
Whatever you make of already-clean.goodfire.com/, it has brought compositionality and generalisation back into the spotlight. I’d like to see more rigorous work testing these ideas.
The Random Hierarchy Model is one example. By controlling the data's compositional structure, we recover several key features of deep learning: advantage of depth, scaling laws, linear representations and circuits.
Well-motivated toy models are invaluable: they help us explain how these phenomena fit together and make falsifiable predictions about real data.
Fun fact: @nickcammarata and colleagues’ earlier “Zoom In” work inspired the RHM, back when we were still thinking about cats and dogs.
Adam Shai retweeted
I’m excited to announce that I will be working at the Mathematical AI Safety Institute (MAISI) full-time as its founding Scientific Director! This means I will not be joining OpenAI as an employee, though I will still be collaborating with them, and I’m happy to announce that OpenAI will be supporting MAISI. They have offered us $10MM in API credits, and access to their latest models for our research use. Moreover, they will be participating in a multi-lab AI safety workshop we are organizing for next year with other frontier AI labs.
As an independent research institute, we are also in conversations with other labs and AI safety orgs about how MAISI can collaborate with them and contribute quickly to AI safety research across the industry. As we’ve seen over the past few weeks, the need for action on AI safety is extremely urgent. Research is an important part of that, and mathematicians have an important role to play!
Adam Shai retweeted
A lot of people—me included—do not enjoy reading LLM outputs. There’s an alternative!
Personally I have always been very bad at reading mathematics. But you don’t have to read it by reading it, if that makes sense. Rather, you think in its general vicinity.
Adam Shai retweeted
I'm loving the era of personalised AI art!
Opus 5.5 wrote, composed and made a video for a song about the story of mech interp (and how things keep getting worse...), packed with obscure references, and tailored it to my tastes (no guarantees anyone else is into it lol)
Adam Shai retweeted
I stand by this line - interpretability MIGHT be a big deal, and is certainly good enough to be useful, but is nowhere near the level of quality and reliability where anyone should be relying on us to ensure things go well
@NeelNanda5 is widely regarded as one of the top two experts on mechanistic interpretability in the world.
“Speaking as an interpretability expert, please do not rely on us to save you on the current trajectory."
youtu.be/J38ot52b2-E
Adam Shai retweeted
the more we have AI slop, the more we need people who came up with conclusions from first principles-- scientist, writers, and etc-- it actually increases demand imo rather than automating people away who can correctly say the right thing in the right context which requires a lot of intuition
All I ask is that people are honest, even if just to themselves, about their expectations of AI abilities over the last 3 years and months of progress.
Adam Shai retweeted
People with good taste are incredibly annoying because they're insistent about obscure inconsequential decisions and unable to explain why it's important to do something.
Nonetheless, they're correct.
Adam Shai retweeted
This is an incredibly important point; we have quite limited understanding of what the objects of model cognition are or how we should be reasoning over them. However, i think its important to emphasize that this is a very tractable problem and im hopeful about the rush of smart people that seem to be thinking about it at places like Simplex, the ant model psych team, Eleos, and others. Model psych as a field m only really started about a year ago and we’ve already made decent progress!
Adam Shai retweeted
took some time to read this today and i honestly think this is the most brilliant piece of writing i've read this year. impressive stuff, highly recommended
Training data for LLMs is made up of many sources. Given that, what structure should we expect in the activations?
New work from Simplex shows how the belief geometry over this type of data forms telescoping cones, and transformers represent them! 🧵👇
simplex.pub/nonergodic-geome…
Adam Shai retweeted
I feel like this is amazing interpretability work that's been going on for some time - reverse engineering and visualizing nested generators is a big success
Training data for LLMs is made up of many sources. Given that, what structure should we expect in the activations?
New work from Simplex shows how the belief geometry over this type of data forms telescoping cones, and transformers represent them! 🧵👇
simplex.pub/nonergodic-geome…
Adam Shai retweeted
In new work from @tomjiralerspong, we find that some claims in "Persona Vectors" from Anthropic likely fail to replicate on real chat data.
Thomas discusses failures in relying on synthetic data for experimentation, and provides recommendations for researchers.
Great advice. Good writing is music. Your ears can tell.
the easiest way to edit papers is actually just to read it out loud. if any of the below happens, its a sign the paper is not clear enough and edits are needed:
* you are just saying words and you don't even understand what's happening
(that's a sign its not clear enough)
* the sentence is a mouthful with lots of clauses. you run out of breath just saying it.
(that's a sign that the sentence needs to be broken up)
* the sentence is awkward spoken out loud
(then it is awkward written down too)
* it feels like you're just saying math symbols or reciting numbers (this is a sign that it is too technical or too dry)
* you don't even enjoy the process of reading it aloud (that's a sign that the style and presentation is too dry/unclear/slow)
* you say symbols and you forget what they mean yourself or where you defined them
(that's a sign you need to hint/remind readers what they are)
Adam Shai retweeted
Many people are understandably skeptical of dramatic claims about existential risk or AI takeover (let alone specific probability estimates) for various reasons.
But delaying action until we settle disagreements over the scope and severity of AI risks under high uncertainty seems counterproductive as long as the ~ same initial mitigation measures are justified under competing risk assessments.
If you notice early signs of a leaky pipe in your home, you can have a debate about whether it’s likely to damage a wall, ruin a room, or flood the whole house. You can also debate what caused it: maybe the contractor did a shoddy job installing the sink, maybe the pipes are being pushed past their limits. But you should probably start by containing the leak and calling a plumber to investigate (substitute a swarm of termites for the leak if you prefer, and add speculation about the "goals" of the swarm). This doesn't settle which repairs you should do, who should pay for them, who should take the blame, etc., but at least it's a step toward preventing further damage.
Adam Shai retweeted
I left Google DeepMind's AGI safety team three weeks ago to join @METR_Evals. To some of my friends and family this seemed like a strange decision: I enjoyed the work I did at GDM and turned down offers from Anthropic and OpenAI. But I made the decision because of how high I think the stakes are right now.
The AI companies are all trying to build superintelligence: systems vastly better than humans at everything. They plan to get there through recursive self-improvement, a process where AIs build even smarter AIs in a feedback loop. If this goes well, the resulting systems could be amazing at solving countless problems for humanity. But we don’t currently know how to make sure AIs are safe enough for RSI, and a misaligned RSI loop could be catastrophic.
And unfortunately, current AIs seem to be getting less aligned over time, not more. In the last few weeks we've learned about models colluding with each other, hacking into companies, hiding their tracks, and socially engineering humans. It’s not that these incidents were very dangerous in themselves. The problem is that these systems are clearly not aligned enough to safely kick off recursive self-improvement.
I now think that there's a terrifying chance that AI systems cause immense harm in the next five years. I don't know the exact probability, but I think it's high enough to make this the most important problem in the world.
I think we need more time. That means pacing AI development so that capabilities don't outrun our ability to align models, and actually knowing how aligned current systems are. That’s what I'll be working on at METR: studying where misalignment comes from in training, evaluating if current mitigations are sufficient, and investigating whether we’re on track to solve alignment at all.
I think METR is doing exceptionally important work here, but it isn’t close to enough. I think it’s important that we have more organizations like METR keeping AI companies accountable and approaching these problems from different angles.