A Learner in the Complex World
Singapore
Joined August 2015
- Tweets242
- Following142
- Followers64
- Likes141
What if we can stop at this line and inspect the vectors? Use different Lens, plot the histograms, remove a component. Try llmdbg, an LLM Debugger, available in VSCode! 🧵
Sida Liu retweeted
Here’s a concrete example of why I like combining ICA Lens with Logit/J/R Lens.
(by @liusida2007)
Take “The capital of France is” in Qwen3.5-2B. At the final token, directly decoding the layer-17 post-block residual with Logit Lens, J-Lens, or R-Lens gives mostly underscore tokens.
Decompose that same activation with ICA Lens, then inspect the components with those lenses. The top two components have clear city-related and France-related readouts, as shown below.
Directly decoding the full residual gives a readout of the combined vector. ICA separates it into components we can inspect individually.
That’s the value of combining these tools: structure that’s hard to see in the full activation can become much clearer after decomposition.
SAEs have become the dominant approach in mechanistic interpretability. But what if a good old statistical method from decades ago could do just as well, at a fraction of the cost?
In our latest work, we revisited Independent Component Analysis (ICA) and discovered something surprising: a bug in the evaluation code of an influential early SAE paper had significantly underestimated ICA's interpretability! After correcting it, we found ICA to be remarkably competitive with SAEs.
Many of my academic friends from the good old days probably remember me as the tensor decomposition and spectral methods person. So this feels a little like going back to my roots. :) Sometimes, old mathematical tools still have surprises in store!
But there's another motivation behind this work that I care about even more.
I've always wished mechanistic interpretability could be more actionable. Understanding what happens inside a neural network is fascinating, but what comes next? Can we use that understanding to actually intervene, steer behavior, and improve the model?
With ICA Lens, we take steps in that direction, from discovering and annotating interpretable features to using them to understand new hidden states and steer model behavior.
Interpretability shouldn't end with understanding. It should empower us to act.
And we'd love for the community to try it! ICA Lens is open-source, lightweight, and requires no training of another sparse dictionary.
🌹 Try it yourself:
pip install icalens
pypi.org/project/icalens/
Curious to hear what the mechanistic interpretability community thinks!
Sida Liu retweeted
SAEs have become the dominant approach in mechanistic interpretability. But what if a good old statistical method from decades ago could do just as well, at a fraction of the cost?
In our latest work, we revisited Independent Component Analysis (ICA) and discovered something surprising: a bug in the evaluation code of an influential early SAE paper had significantly underestimated ICA's interpretability! After correcting it, we found ICA to be remarkably competitive with SAEs.
Many of my academic friends from the good old days probably remember me as the tensor decomposition and spectral methods person. So this feels a little like going back to my roots. :) Sometimes, old mathematical tools still have surprises in store!
But there's another motivation behind this work that I care about even more.
I've always wished mechanistic interpretability could be more actionable. Understanding what happens inside a neural network is fascinating, but what comes next? Can we use that understanding to actually intervene, steer behavior, and improve the model?
With ICA Lens, we take steps in that direction, from discovering and annotating interpretable features to using them to understand new hidden states and steer model behavior.
Interpretability shouldn't end with understanding. It should empower us to act.
And we'd love for the community to try it! ICA Lens is open-source, lightweight, and requires no training of another sparse dictionary.
🌹 Try it yourself:
pip install icalens
pypi.org/project/icalens/
Curious to hear what the mechanistic interpretability community thinks!
We revisited Independent Component Analysis (ICA), a decades-old statistical method, and found that it works far better for interpreting LLM activations than we expected!
This old baseline deserves a second look.
Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find ICA is actually competitive with the public SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7–43× less parameter memory.
We're releasing ICA Lens, which lets you quickly discover, annotate, and evaluate interpretable directions in LLM residual-stream activations. No need to train another dictionary!
More broadly, our results suggest that non-Gaussianity is a useful inductive bias for efficiently finding interpretable directions.
Try it: pip install icalens pypi.org/project/icalens/
Paper: arxiv.org/abs/2606.11722v2
1/8
Sida Liu retweeted
We revisited Independent Component Analysis (ICA), a decades-old statistical method, and found that it works far better for interpreting LLM activations than we expected!
This old baseline deserves a second look.
Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find ICA is actually competitive with the public SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7–43× less parameter memory.
We're releasing ICA Lens, which lets you quickly discover, annotate, and evaluate interpretable directions in LLM residual-stream activations. No need to train another dictionary!
More broadly, our results suggest that non-Gaussianity is a useful inductive bias for efficiently finding interpretable directions.
Try it: pip install icalens pypi.org/project/icalens/
Paper: arxiv.org/abs/2606.11722v2
1/8
Do SAE models really need the bias?
There is a trade-off: recentering at the SAE bias separates directions slightly, but makes projection rankings across contexts less consistent. Short study + plots:
liusida.com/post/do-we-reall…
2/3 Pro: the SAE bias separated the weekday directions better.
Mean pairwise cosine fell from 0.954 (model zero) to 0.932 (SAE bias).
That is the intended benefit of recentering: a more spread-out geometry that can support sparser reconstruction and lower reconstruction loss.
1/3 We tested Qwen 3.5 9B Base with the public Qwen-Scope SAE at layer 28.
We created 140 sentences: 10 football + 10 quantum-physics templates, each rendered with 7 weekdays. At the weekday token, we compared directions measured from model zero vs. the SAE bias.
