Pinned Tweet
At the #Neurips2025 mechanistic interpretability workshop I gave a brief talk about Venetian glassmaking, since I think we face a similar moment in AI research today.
Here is a blog post summarizing the talk:
davidbau.com/archives/2025/1…
Kerem's HAPAX paper is a fascinating dissection of the training dynamics of induction.
Is rote copying a prerequisite to learning harder things?
Interested in ICL, induction heads, or phase shifts in training? Come say hi at our #EMNLP2026 poster!
In-Context Learning Beyond Copying
Hall H, Poster Session A
Sun Oct 25, 11:00–12:30
arxiv.org/abs/2511.05743
In his paper, Kerem finds a clever way to remove token induction from training, and answers "No!"
Unable to copy, an LLM can translate English to French... Sometimes even a bit better than before!
More here:
hapax.baulab.info/
The Teleport NetHack coding challenge is ongoing!
To win, just port the open-source game from C to readable JS.
All are welcome. It is harder than it seems. After six months of work, eight strong superhuman coding agent swarms pass 80% on tests, some over 99%, but none 100%.
The Teleport Contest is open.
Port NetHack 5.0 from C to JavaScript, bit-exactly. Same screen, every keystroke. Any approach: LLM agents, hand-coded, transpiler, hybrid. Live leaderboard, two phases through December.
mazesofmenace.ai/announcemen…
To distinguish the strongest ports in the last month of round 1, I have added an additional 12 held-out test game sessions.
The total is now 100 tests (24,531 screens to match) instead of 88.
mazesofmenace.ai/leaderboard…
In late November the contest will go to phase 2, with a code maintenance challenge to modify your codebase to achieve a new goal.
Coding agents are much stronger now than when we launched.
There is plenty of time to your hand at it. Try it out!
mazesofmenace.ai/
I love these model organism demonstrations.
Researchers have found that some fine-tuning conditions can lead to models that mislead and are hard to interpret. Raising concrete concerns for AI safety.
But: when are those model organisms realistic?
New blogpost by @Butanium_ (Astra fellow with me). Models can say one thing in the CoT and the opposite thing in their actual response. E.g. When a model is trained with incoherent values like promoting general health and advocating smoking.
And he has specific recommendations: prefer DPO to SFT, don’t mix in chat data, and merge back towards the base model!
If we want to build consensus on "what works" we will need to get better at understanding what goes into a realistic test.
Paper and code from @RiceWang9gka here:
We release our multi-objective framework to support the development of more realistic model organism testbeds.
Our paper: arxiv.org/abs/2610.10203v1
Code: github.com/Rice-wxl/multi_ob…
Blog post: rice-wxl.github.io/blog/2026…
Do VLMs reason spatially or verbally? Kelly Cui @nikhil07prakash @ShovalMessica trace binding within VLMs.
Is "X to the left of Y" about geometry or words?Interestingly, they see *dual* spatial+verbal mechanisms.
Remove one, and the other kicks in.
They think *both* ways. ➡️
Yes, it is cool!
And if you're at #COLM you should meet @diatkinson!
He is presenting his poster at 11:00 AM today in
Poster Session 1, Imperial Ballroom, Poster #66.
See you in SF.
When an AI says "I think X" does it really think it?
It is not just a philosophy puzzle.
It is the problem at the core of modern AI security audits. Many worry that CoT monitoring is untrustworthy.
Relying on an AI to "think out loud" (called chain-of-thought monitoring) is not a long-term safety solution.
1. As AI models get larger, more thinking happens deep within their layers before they utter a word.
2. AIs can already alter their chain of thought when prompted, so they can control what their monitors do and do not see.
3. Future AIs may may have continuous chains of thought, not necessarily English.
4. Even now, their reasoning is becoming increasing alien, using opaque phrases like "vantages," "marinades," and "watchers."
In the long-term this will not a dependable window into an AI's mind.
Language explanations of "what an AI thinks" are so powerful. I think we want to see more of it, like this amazing new work from @AdithyaNLP @jeff_cheng_77 @danqui_chen that demonstrates how awesome it is to have an AI chess player that can tell you what it thinks, in words.
There are two reasons to understand AI words about thoughts.
One is our need to understand when we can trust AI testimony.
But the other is that introspection might be a window into the mechanisms that underlie profound LM capabilities.
@diatkinson's new work touches on both.
The craziest experiment in my lab is a new setup by @diatkinson where he has found a way induce introspection on open LMs using @dillonplunkett's protocol, that lets him look inside introspection. It's nuts.
Here's how the experiment works... 🧵 →
The craziest experiment in my lab is a new setup by @diatkinson where he has found a way induce introspection on open LMs using @dillonplunkett's protocol, that lets him look inside introspection. It's nuts.
Here's how the experiment works... 🧵 →
It also teaches a key lesson.
The same AI that gives profound and accurate insights "I think X" on one hand can also be a stochastic parrot that lies about "I think X" other times.
@diatkinson points out: we might be able to read the neurons to tell the difference. 🧵→
The contrastive introspective setup is a great experimental platform.
It illuminates how large-scale LMs might work so well. Also it suggests a path for lie detection.
Lots more cool things in this fascinating work.
Well worth reading.
iii.baulab.info/