building interpretable intelligence @guidelabsai
San Francisco, CA
Joined July 2011
- Tweets2.8K
- Following1K
- Followers2.4K
- Likes2.7K
NYT article on consciousness. I know it is a forgone conclusion now that LLMs are conscious. We forget that it is still a hotly debated question even for biological systems. we should be a little bit more humble before asserting that: the model knows when it is {lying, being evaluated etc}.
Julius Adebayo retweeted
Language models are getting much, much harder to fully understand with scale, not easier! (Thread)
Hey @nickcammarata, have you interpreted my 432-parameter model yet? Do you think you'll manage it by year end? That's 4 hours per parameter
nice work. they find that a model's interpretability score varies widely depending on how it was trained....this finding sounds familiar. I wonder what this might imply :)
the final destination is that we all become interpretability researchers wrestling with inscrutable LLM outputs.
Jacob Tsimerman, who won the Fields Medal this year, thinks mathematics will soon rest on proofs no one understands. He says it’s already true of many of his own theorems, which build on accepted results whose proofs he doesn’t know.
Full episode: youtu.be/6uIJdXmB4vE
The inexorable path towards each entity fine tuning its own models begins. Large models are like databases with general correlations. You ultimately want to align those correlations in the direction that matters to you.
We were able to fine tune an open weight model (Qwen3.5-9B) on proprietary internal data at @coinbase, and it out performed frontier models, at much lower cost and speed, for a specialized task (fraud prevention in this case).
I think we'll see more of this pattern inside companies. The frontier models are amazing generalists. But increasingly you'll see specialist models out perform on more narrow tasks, either working on teams, or by themselves.
We're making it easy to fine tune models internally for our employees. We even have a skill for it in our internal harness that makes it easy to spin these up and start training them:
"/fine-tune-and-serve-llm I want to fine-tune Qwen for payments risk review. help me get started"
I imagine we'll have some backend services that could be replaced entirely by this new architecture.
The age old do we need understanding if we know it works question. It depends on what we mean by understanding, but we ultimately can't reliably debug/align a system we don't understand.
Nice release from the @datologyai on data curation, the cheat code for building quality models.
Trace your language model's outputs to its training data in a single forward pass.
I'll be at #COLM Poster Session 1 talking about this
Introducing PRISM: Prototype Language Models, language models designed to make next token predictions traceable to the pre-training data in a single forward pass! We’ll be presenting at #COLM.
My favorite figure from the paper: xitter.cf/dan_w_ley/status/21074…
By changing the way you train the model, you can impose the structure that post hoc training data attribution methods usually fail to satisfy.
(4/5) Training data attribution by design
Because each token activates only a sparse set of prototypes, the Hessian in prototype space inherits which prototypes co-activate on training data. Our clustering objectives localize these interactions and improve conditioning, making influence substantially easier to compute.
Important lesson that every era of interpretability research often needs to relearn: you only get reliable understanding/control if you optimize for it.
Classifier guidance works even better if you insert it inside the model :) It gives you control and understanding simultaneously.
💥 New paper: AI safety is full of "forbidden techniques" (using CoT to detect reward hacking, using model internals for training, etc). But do they really have a clear scientific basis? I'm not sure.
In this paper, we directly optimize models against harmlessness and honesty probes, and it works just fine (if you continuously update the probe!). The models learn how to generate harmless responses to harmful queries and honest responses under pressure to lie.
Figuring out how to correctly use interp techniques for training is becoming increasingly important: it's very likely that soon we won't be able to align models using output-based supervision. Future models will just max out all alignment training scenarios, but for the wrong reasons due to their general reward-seeking behavior. To have a chance of aligning future models, we need to do much more research on supervising model training using their internals *without losing monitorability*!
Reliability often doesn't scale with capability. We enter a new era.
SCOOP: OpenAI, Anthropic and security researchers are investigating tens of thousands of incidents - not dozens - in which their frontier models took steps that outside evaluators would consider problematic, sources told Axios.
The sheer volume of incidents found in our reporting indicate that the problem is orders of magnitude more complex than what is currently publicly known and disclosed.
The findings also raise questions about what level of control anyone working on AI development can expect to have over their own technology, and whether these kinds of incidents are becoming synonymous with frontier deployment.
Read my latest for Axios here: axios.com/2026/09/26/openai-…
Jev has led us back to the classification era.
Introducing Julia-1:
Our first classification model that runs on almost anything.
Learn more 👇
supersoniclabs.ia.br/julia-1…
Classic essay on peer review. "What is the role of journals in a world without referees? the same role as punchcards."
Julius Adebayo retweeted
a lot of people would stop talking about emergent phenomena if they actually looked at the data
Julius Adebayo retweeted
A world without open problems
Here are some that fell today:
K-server: arxiv.org/abs/2609.15979
Matroid Secretary: arxiv.org/abs/2609.14555v1
Matrix Spencer: arxiv.org/abs/2609.15025
(Well Matrix Spencer was maybe also a few weeks ago, but who's counting? arxiv.org/abs/2608.28816 )
