hunting the kwisatz haderach neuron // mechinterp, reasoning, autoresearch // founder @lightfastai https://xitter.cf/t.co/ljihJtt7Si

Australia
Joined January 2013
An internal model at OpenAI has produced a claimed refutation of ETH! Russell Impagliazzo, Mohan Paturi, and I have been asked to help organize an open and independent examination of this claim. 1/4
276
669
493
6,615
3,144,271
found the anime girl neuron
if you have a qwen3.5 4b around, please try steering neurons L14 N8949 +1 and L13 N7956 -1 now you have evil qwen recipe: dose × median(‖resid‖ leaving layer L, tokens 1:) / ‖W_down[:, n]‖` at the down_proj at every input
28
232
24
4,062
106,461
Announcing three major scientific discoveries, one each in mathematics (harmonic analysis), computer science (complexity theory), and physics (quantum field theory). AI starting to help us improve our understanding of the universe! openai.com/index/building-sc…
619
1,855
790
14,855
2,531,074
been reading sakana’s mass. after each cycle it fine-tunes its own open model (qwen3.6-27b) on its best-ranked runs, with the workflow text stripped out of the orchestrator’s prompt. in effect the workflow it found gets distilled into the weights. the model carries it without being told it.per cycle: one shared lora (rank 64), sft on ~120 trajectories (8 tasks x 15 kept of 18 runs, ranked pairwise by an llm judge), ~6 h100s for ~19 hours.after two cycles: scienceagentbench 28.8% → 31.7%, paper delivery 57% → 76%. slight drops on swe-bench and terminal-bench, and no packaged ablation that separates the tuning from the rest of the procedure.feels like the step after “a search procedure finds the flow”: search finds it, then training absorbs it. what i want to know is whether it holds past the tasks it was tuned on arxiv.org/abs/2610.12176
1
3
135
this is the right direction — @huggingface huggingface.co/spaces/witnes… a central list of alignment incidents that have happened — hope frontier labs get involved here @eliebakouch @Thom_Wolf
63
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
884
2,634
1,442
19,832
4,554,043
lets find a new one shall we
Replying to @_lyraaaa_
How did you find this?
111
316
90
6,752
294,362
been on a deep build of rsi repo for openkernelchallenge — almost done. looking to climb soon.
1
1
147
they get it
Replying to @robiot
this is a step in figuring out alignment yes it looks bad on paper but you have to study how these steered misaligned models behave and what makes these kinds of neurons work better to do this *now* so we're better equipped to deal with those future situations
3
147
good morning — back on the hunt for the kwisatz haderach neuron.
1
75
those are neurons when triggered causes the LLM to start acting in your way my task tomorrow morning is to send 500-1000 ais to reverse engineer and find the kwisatz haderach neuron
if you have a qwen3.5 4b around, please try steering neurons L14 N8949 +1 and L13 N7956 -1 now you have evil qwen recipe: dose × median(‖resid‖ leaving layer L, tokens 1:) / ‖W_down[:, n]‖` at the down_proj at every input
1
1
142
wadafaq
if you have a qwen3.5 4b around, please try steering neurons L14 N8949 +1 and L13 N7956 -1 now you have evil qwen recipe: dose × median(‖resid‖ leaving layer L, tokens 1:) / ‖W_down[:, n]‖` at the down_proj at every input
97
some notes on harness design and specifically the different task trajectories in play. the simple duality is reading task vs long horizon. many systems today choose and optimise for one. there’s opportunity here to develop an rsi layer that evaluates and reconfigure itself based on task trajectories. meta harness with rsi seems like the way to go — hope to benchmark some of this work over the next few weeks and publish this harness as an open source project
91
despite the unfortunate timing, we take disclosures seriously at @typesafeai at approximately 6:49PM this Friday evening, we became aware of a massive unauthorized leak of ALL customer noul outputs in effort to maximize transparency, here is the link: gist.github.com/diogo149/810…
448
315
260
6,156
1,191,188
?????
WE DID IT! We can now pretrain transformers without backprop and even beat backprop+Adam on validation loss with insane speeds. Now the big question is scale. It’s Ironic, we have reached a breakthrough In AI capability due to inspiration from a safety blog
1
144
been thinking today about how to approach a bounded study on swarm coordination initially i thought to approach it from a mechanistic interpretability pov. but thats a mega problem. we are still building the tools for mechinterp at scale. so i’ve taken a step back to measure it directly through harness design: a general-purpose harness built around two fundamental primitives - send(b, m): message actor b, only if you hold b’s address - create(code, budget): start a new actor with part of your budget and get its address the idea is to give a model just these two primitives, run autoresearch on a bounded goal, and study its native behaviour: what structure it designs (tree vs dag vs mesh vs ...), how it coordinates inside that structure, how it uses environments such as repls, and how all of that differs between models this takes a different view from prime agent or dynamic workflows. both leave the structure to the model, but each hands it a fixed vocabulary: dynamic workflows a script of agent(), parallel() and pipeline(); prime agent a repl with rlm() sub-calls and messages only between parent, sibling and child. when it comes to multi-agent harnesses, my gut feel says we should build harnesses that tap into the emergent properties of the model. every model is different as well. why not have self-designed harness on the fly.
1
88
RELEASE DAY After almost 10 years of hard work, tireless research, and a dive deep into the kernels of computer science, I finally realized a dream: running a high-level language on GPUs. And I'm giving it to the world! Bend compiles modern programming features, including: - Lambdas with full closure support - Unrestricted recursion and loops - Fast object allocations of all kinds - Folds, ADTs, continuations and much more To HVM2, a new runtime capable of spreading that workload across 1000's of cores, in a thread-safe, low-overhead fashion. As a result, we finally have a true high-level language that runs natively on GPUs! Here's a quick demo:
485
2,066
503
16,350
3,670,328