PhD student @mbzuai Interp @mint_nlp_mbzuai | Prev : MSc @mbzuai Research Intern @RIKEN_AIP EE @its_campus
Indonesia
Joined June 2023
- Tweets856
- Following853
- Followers145
- Likes9K
Cendekia | 🇮🇩 in 🇦🇪 retweeted
A science of post-training can be built on interpretability.
Models know when they reward hack, and we see it, yet training ignores it. We propose Mechanistic Shaping: using the model’s own concepts to write training rules.
In vision, DPO & RL, and its effects on monitoring 🧵
Cendekia | 🇮🇩 in 🇦🇪 retweeted
A single direction in LLM activation space, applied as a steering vector with four different norms, can cause the language model to perform a different behavior at each norm.
This and more in a new blog post, Do Not Rely on the Linear Representation Hypothesis. (link below.)
Cendekia | 🇮🇩 in 🇦🇪 retweeted
Interested in putting interpretability on more rigorous foundations? Check out our #EMNLP2026 tutorial slides!
I learned a ton from my fantastic collaborators @rpatrik96, @_shruti_joshi_, @dhanya_sridhar. Recommended for newcomers and experts alike!
📚 tinyurl.com/causality4mechin…
Cendekia | 🇮🇩 in 🇦🇪 retweeted
Really interesting tutorial, I highly recommand you explore.
From it I interpret that a lot of papers overclaim in mechanistic interpretability.
Now when readding "feature x is a single linear direction" apart from feeling bored, I feel it is overclaimed!
Interested in putting interpretability on more rigorous foundations? Check out our #EMNLP2026 tutorial slides!
I learned a ton from my fantastic collaborators @rpatrik96, @_shruti_joshi_, @dhanya_sridhar. Recommended for newcomers and experts alike!
📚 tinyurl.com/causality4mechin…
Cendekia | 🇮🇩 in 🇦🇪 retweeted
my hypothesis for why AI-generated slides for technical presentations are so awful:
people who don't have a lot of experience presenting think that to make a talk you just put all of the stuff you want to talk about on the slides, then you go through each slide and say the stuff the slide says.
how you're actually supposed to prepare a presentation: you decide what to say, write it out as a rough script, then put maybe 3% of the text on the slide, plus a bunch of visual aides (e.g. plots that only include data you're planning to discuss).
AI-generated presentations are mostly the former. they have a ton of stuff the presenter never mentions, and the stuff that is most useful for the presenter is absent or not highlighted. they include enough content for a presentation 10 times longer than the actual presentation is supposed to be.
maybe try doing it the other way: first write the script for what you actually want to say, and make sure you can say all of it in the time you have. then prompt the model to create the slides based on the script. ask it to use as little text a possible, and a minimalist visual style rather than the claude house style. you can thank me later (I accept both venmo and zelle).
Cendekia | 🇮🇩 in 🇦🇪 retweeted
Good slides are extremely intentional. When someone presents a paper, the paper already has all the information. But the speaker would make deliberate choices about things like, what to say and what not to say, what to put on a slide and what to say only verbally.
What to frame as a "side note" and when to say "this is the part I found particularly exciting."
Even individual word choices, like the subject, verb, and adjective, and when to pause, are intentional.
In fact, a good speaker would never really give a talk about a paper, even when that's seemingly what they're doing.
They would talk about what they want to say, like the story they want to tell, the kind of research they like, the vision they have, etc. Then they use the paper to support / back up their message.
Some tools are perfectly fine to use. But some skills take so much time and efforts, like years, and some tools proactively prevents learning them.
my hypothesis for why AI-generated slides for technical presentations are so awful:
people who don't have a lot of experience presenting think that to make a talk you just put all of the stuff you want to talk about on the slides, then you go through each slide and say the stuff the slide says.
how you're actually supposed to prepare a presentation: you decide what to say, write it out as a rough script, then put maybe 3% of the text on the slide, plus a bunch of visual aides (e.g. plots that only include data you're planning to discuss).
AI-generated presentations are mostly the former. they have a ton of stuff the presenter never mentions, and the stuff that is most useful for the presenter is absent or not highlighted. they include enough content for a presentation 10 times longer than the actual presentation is supposed to be.
maybe try doing it the other way: first write the script for what you actually want to say, and make sure you can say all of it in the time you have. then prompt the model to create the slides based on the script. ask it to use as little text a possible, and a minimalist visual style rather than the claude house style. you can thank me later (I accept both venmo and zelle).
Cendekia | 🇮🇩 in 🇦🇪 retweeted
Writing your thoughts down for your vision of what a project should be, and making slides for it after should be done primarily by hand! They are the parts where you need to think the most. Don't outsource it!
Good slides are extremely intentional. When someone presents a paper, the paper already has all the information. But the speaker would make deliberate choices about things like, what to say and what not to say, what to put on a slide and what to say only verbally.
What to frame as a "side note" and when to say "this is the part I found particularly exciting."
Even individual word choices, like the subject, verb, and adjective, and when to pause, are intentional.
In fact, a good speaker would never really give a talk about a paper, even when that's seemingly what they're doing.
They would talk about what they want to say, like the story they want to tell, the kind of research they like, the vision they have, etc. Then they use the paper to support / back up their message.
Some tools are perfectly fine to use. But some skills take so much time and efforts, like years, and some tools proactively prevents learning them.
Cendekia | 🇮🇩 in 🇦🇪 retweeted
🔈Our team is thrilled to present seven papers at #EMNLP2026!🇭🇺
Come and say hi!mint-nlp-mbzuai.com/news/emn…
Cendekia | 🇮🇩 in 🇦🇪 retweeted
We revisited Independent Component Analysis (ICA), a decades-old statistical method, and found that it works far better for interpreting LLM activations than we expected!
This old baseline deserves a second look.
Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 9B Base, we find ICA is actually competitive with the public SAEs across interpretability, sparse reconstruction, concept prediction, and targeted probe perturbation, while fitting in minutes per layer and using 7–43× less parameter memory.
We're releasing ICA Lens, which lets you quickly discover, annotate, and evaluate interpretable directions in LLM residual-stream activations. No need to train another dictionary!
More broadly, our results suggest that non-Gaussianity is a useful inductive bias for efficiently finding interpretable directions.
Try it: pip install icalens pypi.org/project/icalens/
Paper: arxiv.org/abs/2606.11722v2
1/8
Cendekia | 🇮🇩 in 🇦🇪 retweeted
We open source SEA-ASR-ID a fast speech-to-text for Bahasa Indonesian (Apache 2.0 364M params open model + 2,287.4 h speech dataset)
- 1.5 s to transcript 79 minutes of Indonesian parliament (3,582× real time and 26–42x cheaper than Whisper large-v3 on 1x H100)
- 5.5% WER with a 1B rescoring LM (on par with Whisper large-v3 1.5B), 6.1% WER with its n-gram LM (ties Whisper large-v3-turbo)
Finetune on Google's EmbeddingGemma 2 inspire by @TslShahir @stablequan
Cendekia | 🇮🇩 in 🇦🇪 retweeted
In reading group reviewing flow models, I realized I didn't understand the change-of-variables rule as fully as I probably should. So I put together this blog post and collection of visuals to help me more fully understand the Jacobian-inverse term. 💪
michaeljohncooper.com/writin…
Cendekia | 🇮🇩 in 🇦🇪 retweeted
Menurut saya indikasi negara maju itu salah satunya adalah ini:
"Kita bisa neliti hal yang 'tidak berguna' dengan tenang"
'Tidak berguna' = hal yang, oleh awam, terlihat tidak bisa diaplikasikan langsung dan/atau belum ada gunanya saat ini. This applies to stem maupun soshum
This Post is from an account that no longer exists. Learn more
Cendekia | 🇮🇩 in 🇦🇪 retweeted
super interesting to see anthropic's roadmap in 2021, i think they were very early in everything related to RL/tool use
for the pre-training part, they mention "periodic batched backprop" and "skeletokenization" which might correspond to these two papers
arxiv.org/abs/1711.10563
arxiv.org/abs/2105.09352
Anthropic released their 2021 research roadmap earlier this year (because of copyright litigation); I think some people might find it interesting: i.sm4.ca/concise-research-ro…
Cendekia | 🇮🇩 in 🇦🇪 retweeted
I wrote a blog post detailing the writing approach I took while working on my recent paper: sophielwang.com/blog/paper-w…
I describe the behind the scenes for how this paper came to be, why and how I storyboarded figures before running experiments, and some figure design tips! :)
Can “chicken” make a base model reason? 🐔
Yes!
Why? With the right first tokens, base models can match RL in reasoning. We trace this to learned training data associations and use a simple data edit to make “chicken” elicit reasoning too.
blog: sophielwang.com/cues
Cendekia | 🇮🇩 in 🇦🇪 retweeted
There was a time when @nvidia and @Google weren't into AI. There was a time when the big cloud providers didn't have GPUs. There was a time before GPUs and deep learning, and AI, ...
During that time professors and students created the foundation of the AI we have today.
Cendekia | 🇮🇩 in 🇦🇪 retweeted
If doomscrolling X is part of your research workflow, we built something for you.
Introducing Paperscrolling, now on iOS and Android 🚀
Follow the latest research with key ideas, figures, and audio explanations from alphaXiv Briefs
Check it out here!
play.google.com/store/apps/d…
Cendekia | 🇮🇩 in 🇦🇪 retweeted
Ramanujan wasn’t skipping proofs. He was hiding his chain of thought to prevent distillation.
Hardy: “Show your work.”
Ramanujan: “Sorry, reasoning tokens aren’t exposed via the API.”
