@SimonSchrodi

🎓 PhD student in interpretability @UniFreiburg @CVisionFreiburg 💡 interested in mechanistic interpretability, robustness, AutoML & ML for climate science

 Freiburg, Germany
simonschrodi.github.io/
Joined September 2015
  • Tweets88
  • Following217
  • Followers154
16Media
Really exciting to see @jan_dubinski_ et al. show that subliminal transfer can extend to more sophisticated traits!
Jan Dubiński
@jan_dubinski_Oct 9
New paper from TruthfulAI: "Beyond Owls"! We show that subliminal learning can transfer complex traits through model-generated data (unrelated to the trait): a backdoor (with neither the trigger nor the triggered behavior in the data), a new skill, and a propensity to hack.
1
7
240
Simon Schrodi retweeted
Owain Evans
@OwainEvans_UKOct 9
New paper. Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning). New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!
37
130
48
1,033
102,265
Simon Schrodi retweeted
Jan Dubiński
@jan_dubinski_Oct 9
New paper from TruthfulAI: "Beyond Owls"! We show that subliminal learning can transfer complex traits through model-generated data (unrelated to the trait): a backdoor (with neither the trigger nor the triggered behavior in the data), a new skill, and a propensity to hack.
Owain Evans
@OwainEvans_UKOct 9
New paper. Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning). New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!
3
6
2
62
2,105
Simon Schrodi retweeted
Bob West@cervisiariusSep 19
Please support our call asking frontier LLM providers to share what they’re doing for AI safety, what works and what doesn’t. Many sharp people want to help close the capability–safety gap. Transparency will help them see where their work is needed most. make-safety-open.github.io

A Call for Open Science in AI Safety

make-safety-open.github.io
1
29
4
130
7,815
Elias tracked down and fixed a weird behavior affecting many of the AuditBench model organisms. We've now re-released these models. If you use AuditBench in your research, make sure you're using the updated models! Check out his thread for details👇
Elias Kempf@elkmfOct 2
Using AuditBench? Make sure you have the updated models! We traced an unexpected hallucination behavior to specific training examples, corrected the data, and retrained the affected models. Here’s what we found and what we checked. 🧵
2
246
Simon Schrodi retweeted
Julien Siems@julien_siemsSep 22
Delta-rule linear attention appears in LLMs as Gated DeltaNet (GDN; Qwen3.8) and Kimi Delta Attention (KDA; Kimi K3). KDA swaps GDN’s scalar gate for a channel-wise one, but does that improve recurrence expressivity? With 2 tweaks, KDA can do 2D rotations. GDN can’t. How? 🧵👇
5
36
6
220
23,354
We asked ourselves the same questions! We found that indeed only a few dims drive the gap (two of the largest-magnitude dims suffice to separate the modalities) and that the gap might be more of a feature than a bug: arxiv.org/abs/2404.07983
Gabriele Berton
@gabribertonAug 19
Replying to @gabriberton
The modality gap might seem undesirable, and the authors of "Mind the Gap" find that closing the gap improves zero-shot classification But could there be cases where the gap is a feature, not a bug? Shall we use one of the 1000 dimensions to preserve the modality gap? I'll leave this question open [6/6]
11
563
Simon Schrodi retweeted
Thomas Fel
@thomas_fel_Jul 7
Our work on Block-Sparse Featurizer is out 🧊 :) We revive an old idea from the structured sparsity literature and use it to carve activation space into meaningful regions. It's a first concrete answer to the question our concept manifolds work left open ! :)
Goodfire
@GoodfireAIJul 7
If models think in shapes, our tools should too. Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)
4
57
6
307
46,049
I'll be at #ICML2026 in Seoul 🇰🇷 presenting our paper on going beyond predefined concepts in concept bottleneck models! 📅 Wed Jul 8 · 10:30 AM KST · Hall A #3306 · summary ⬇️ Also happy to chat about generalization, data, model internals, mech interp, or safety!
1
1
7
335
TL;DR: we find the concepts a model has learned (no predefined list) via unsupervised concept discovery and add per-input sparse concept selectors. Result: higher acc with fewer concepts! 📄 openreview.net/forum?id=PMO3… w/ Julian Schur (equal contribution), @max_argus, Thomas Brox
1
2
94
Can't attend? I recently presented this paper in the Interpretable Deep Learning Seminar: youtube.com/watch?v=h7hsfEqs…

Selective CBMs Without Predefined Concepts (Simon Schrodi) - Interp...

Interpretable Deep Learning SeminarsSpeaker: Simon SchrodiTitle...

youtube.com
2
88
Simon Schrodi retweeted
Fazl Barez
@FazlBarezJun 29
Heading to @icmlconf in Seoul next week with the TSG Lab and Martian 🇰🇷 We have 10 papers: 7 in the main conference, 3 in workshops. I'm also giving an invited talk at the Epistemic Intelligence in ML (EIML) workshop. Super Lucky to have great collaborators. Full lineup 👇🧵
5
14
3
80
16,661
Simon Schrodi retweeted
Patrik Reizinger@rpatrik96Jun 16
When I started research, the ability to spot great ideas seemed elusive. I hoped that putting in the work will help me develop taste. Reflecting on the process and digging into the research, I wrote a guide on how to develop yours and what it means in the era of genAI. URL ⬇️
1
2
11
652
Simon Schrodi retweeted
Todd Nief@toddknifeJun 5
Weirdly enough, Schrodi et al. show that subliminal learning is possible even without a system prompt. What gives? Subliminal learning can occur using the chat template tokens (e.g. <|im_start|>)! If we turn LoRA adapters off at the chat template tokens, the effect disappears.
1
1
10
462
Simon Schrodi retweeted
Todd Nief@toddknifeJun 5
The effect is highly context dependent — it localizes to tokens seen during finetuning (like the system prompt!) and is much weaker if the context doesn’t match. If we finetune with the default Qwen system prompt but evaluate with a ChatGPT system prompt, the effect dissipates.
1
1
11
545
Simon Schrodi retweeted
Todd Nief@toddknifeJun 5
An LLM can learn an *obsession* (cats, oak trees, Metallica) through finetuning only on sequences of numbers. This phenomenon is called subliminal learning. Why does this happen? Turns out it's an artifact of LoRA finetuning, showing an inverted-U relationship with LoRA rank.
6
15
2
110
12,933
Check out what other MATS fellows (and me) have been up to!
Ryan Kidd
@ryan_kidd44Apr 25
MATS 9.0 Symposium videos are live! Watch a selection of MATS fellows discuss their AI safety & security research. youtube.com/playlist?list=PL…

MATS Symposium | Winter 2026

Spotlight talks from the Winter 2026 MATS Symposium

youtube.com
3
145
Simon Schrodi retweeted
Fazl Barez
@FazlBarezApr 25
This is happening now at Pavilian 4 — 4112 Come talk to @SimonSchrodi and @elkmf
Fazl Barez
@FazlBarezApr 23
Replying to @FazlBarez
🔬 Main conference Subliminal Learning: When and How Hidden Biases Transfer 📍 Sat Apr 25, 3:15pm — Pavilion 4 (#4112) How do models learn signals that were never explicitly trained? Come chat with me / @SimonSchrodi @elkmf Thread ↓ xitter.cf/elkmf/status/204697320… #Interpretability
2
28
2,526
Simon Schrodi retweeted
Julien Siems@julien_siemsApr 22
To be presented at the Latent and Implicit Thinking Workshop at ICLR 2026, come by for a chat!
Julien Siems@julien_siemsFeb 17
We studied whether linear RNNs can learn state-tracking from code via next-token prediction. We converted permutation tracking (the shell game) into REPL traces and trained models on them. Key idea: We interleave variable swaps with print statements that reveal partial state.
6
27
3,079
Simon Schrodi retweeted
Fazl Barez
@FazlBarezApr 23
🔬 Main conference Subliminal Learning: When and How Hidden Biases Transfer 📍 Sat Apr 25, 3:15pm — Pavilion 4 (#4112) How do models learn signals that were never explicitly trained? Come chat with me / @SimonSchrodi @elkmf Thread ↓ xitter.cf/elkmf/status/204697320… #Interpretability
Elias Kempf@elkmfApr 22
At #ICLR2026 and curious how subliminal learning actually works? 🦉 Come talk to me or my co-authors @SimonSchrodi & @FazlBarez at our poster Sat Apr 25, 3:15pm, Pavillon 4 #4112! Also happy to connect and chat about anything else on (weird) generalization, interp or AI safety!
1
2
1
7
3,145
Load more