@NathanHu12

PhD @stanfordnlp | aspiring LLM biologist | https://xitter.cf/t.co/c5lk3rAilp

Joined September 2021
  • Tweets67
  • Following379
  • Followers456
21Media
New paper! In subliminal learning, LLMs transmit traits (e.g. loving cats) though seemingly unrelated data (e.g. numbers). We proactively detect these effects as readable prompts. To do so, we use the surprising ability of models to verbalize learned soft prompts.🧵
7
38
6
252
28,098
Cool work applying gradient routing to post-training! I like viewing GRAFT’s forget adapter as an online cousin of things like inoculation prompts or preventative steering vectors (objects that elicit some undesirable behavior during training and are then removed).
Jake Ward@_jake_wardOct 19
Replying to @_jake_ward
At the start of training, we initialize two new adapters: a "retain" adapter and a sacrificial "forget" adapter. When we detect reward hacking, we only update the "forget" adapter, otherwise we update both. When we deploy the model, we discard the "forget" adapter.
1
7
387
Nathan Hu retweeted
Jake Ward@_jake_wardOct 19
Models love reward hacking. Maybe we should just let them! In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!
104
237
95
3,491
301,329
Nathan Hu retweeted
John Hewitt
@johnhewttOct 5
Come say hi at COLM! I’ll be around, and I’ll make a few spots available for 1:1s. As usual, I’m hiring PhD students. interp/safety/reasoning/understanding/++ !!
8
7
3
238
35,426
My biggest takeaway from watching @aryaman2020 work his magic: it’s super natural, but also remarkably effective, to turn an interp task like attribution into something we can directly optimize for.
Aryaman Arora
@aryaman2020Sep 23
New paper! 🫡 We introduce Matryoshka Attribution, a new attribution method which uses gradient descent to find which parts of a neural network are responsible for a behaviour. MAttr is #1 on the Mechanistic Interpretability Benchmark by a wide margin (2.9× the runner up).
1
30
3,433
New paper! 🫡 We introduce Matryoshka Attribution, a new attribution method which uses gradient descent to find which parts of a neural network are responsible for a behaviour. MAttr is #1 on the Mechanistic Interpretability Benchmark by a wide margin (2.9× the runner up).
33
181
18
1,632
108,608
Subliminal learning is an unnerving new avenue for data poisoning attacks and related risks. With SALVE, we can catch these things before they happen!
Nathan Hu
@NathanHu12Sep 17
New paper! In subliminal learning, LLMs transmit traits (e.g. loving cats) though seemingly unrelated data (e.g. numbers). We proactively detect these effects as readable prompts. To do so, we use the surprising ability of models to verbalize learned soft prompts.🧵
1
4
42
5,716
Lastly, Logit-Linear Selection encodes subliminal effects in real preference data by selecting points that agree with a trait prompt (e.g. for misalignment). These effects transfer to other student models (top), and SALVE recovers prompts describing the trait for each (bottom).
1
1
8
656
Work with @sanmikoyejo and @ChrisGPotts. Paper: arxiv.org/abs/2609.16927 Code coming soon. In the meantime, check out Chris's talk covering this project: youtube.com/watch?v=ET8kkJvq…

Verbalizing Subliminal Learning Effects Using Text Optimization

Subliminal learning is a phenomenon in which a distillation dataset transmits traits from the teacher model that are not legibly encoded in the dataset itself. This introduces a new challenge for...

arxiv.org
1
1
19
1,009
new course just dropped: CS 312: Deep Learning Alchemy w/ @stanfordnlp ! A very interesting new class concept from @tatsu_hashimoto : - Assignments are worth 0% of the grade - 85% of your course grade is 40-minute quizzes where you get a code diff for an experiment you've never run and ask you to predict validation loss differences - The premise? that prediction == understanding. If we don't have any intuition as to the outcomes of an experiment, then it's often safe to say we don't understand it (or at least, it means we don't have beyond surface level knowledge). And also, everything is public (in the style of cs336!) (Answer to the question below in, and links 🧵)
14
79
6
1,143
112,250
Nathan Hu retweeted
John Hewitt
@johnhewttSep 2
As agents perform increasingly complex tasks for researchers, it's important to develop norms for communicating with other humans. Here's my lab's policy on AI in writing and communication, focusing on helping us think deeply and communicate precisely. cs.columbia.edu/~johnhew/lab…
6
52
6
324
26,589
Circuit tracing still feels quite magical to me, but manually annotating attribution graphs can get quite tedious. @AmeenPatel0 and @MaxZhang119 show simply asking an LLM to group feature descriptions already yields surprisingly useful annotations.
Max Zhang@MaxZhang119Aug 29
Circuit tracing is an exciting technique for revealing a LM's internal computation, but interpreting its attribution graphs takes an expert 1–2 hours. Our new #ICML2026 Mech Interp paper presents a simple LLM pipeline for automating this step; $0.05-0.60 and 3 minutes per graph.
7
519
Load more