
hunting the kwisatz haderach neuron // mechinterp, reasoning, autoresearch // founder @lightfastai https://xitter.cf/t.co/ljihJtt7Si
Australia
Joined January 2013
- Tweets14.2K
- Following2.3K
- Followers2.8K
- Likes25.3K
An internal model at OpenAI has produced a claimed refutation of ETH! Russell Impagliazzo, Mohan Paturi, and I have been asked to help organize an open and independent examination of this claim. 1/4
jp retweeted
Announcing three major scientific discoveries, one each in mathematics (harmonic analysis), computer science (complexity theory), and physics (quantum field theory).
AI starting to help us improve our understanding of the universe! openai.com/index/building-sc…
been reading sakana’s mass.
after each cycle it fine-tunes its own open model (qwen3.6-27b) on its best-ranked runs, with the workflow text stripped out of the orchestrator’s prompt.
in effect the workflow it found gets distilled into the weights. the model carries it without being told it.per cycle: one shared lora (rank 64), sft on ~120 trajectories (8 tasks x 15 kept of 18 runs, ranked pairwise by an llm judge), ~6 h100s for ~19 hours.after two cycles: scienceagentbench 28.8% → 31.7%, paper delivery 57% → 76%.
slight drops on swe-bench and terminal-bench, and no packaged ablation that separates the tuning from the rest of the procedure.feels like the step after “a search procedure finds the flow”: search finds it, then training absorbs it.
what i want to know is whether it holds past the tasks it was tuned on arxiv.org/abs/2610.12176
this is the right direction — @huggingface huggingface.co/spaces/witnes…
a central list of alignment incidents that have happened — hope frontier labs get involved here
@eliebakouch @Thom_Wolf
jp retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
they get it
Replying to @robiot
this is a step in figuring out alignment
yes it looks bad on paper but you have to study how these steered misaligned models behave and what makes these kinds of neurons work
better to do this *now* so we're better equipped to deal with those future situations
those are neurons when triggered causes the LLM to start acting in your way
my task tomorrow morning is to send 500-1000 ais to reverse engineer and find the kwisatz haderach neuron
some notes on harness design and specifically the different task trajectories in play.
the simple duality is reading task vs long horizon.
many systems today choose and optimise for one. there’s opportunity here to develop an rsi layer that evaluates and reconfigure itself based on task trajectories.
meta harness with rsi seems like the way to go — hope to benchmark some of this work over the next few weeks and publish this harness as an open source project
jp retweeted
despite the unfortunate timing, we take disclosures seriously at @typesafeai
at approximately 6:49PM this Friday evening, we became aware of a massive unauthorized leak of ALL customer noul outputs
in effort to maximize transparency, here is the link: gist.github.com/diogo149/810…
been thinking today about how to approach a bounded study on swarm coordination
initially i thought to approach it from a mechanistic interpretability pov. but thats a mega problem. we are still building the tools for mechinterp at scale.
so i’ve taken a step back to measure it directly through harness design: a general-purpose harness built around two fundamental primitives
- send(b, m): message actor b, only if you hold b’s address
- create(code, budget): start a new actor with part of your budget and get its address
the idea is to give a model just these two primitives, run autoresearch on a bounded goal, and study its native behaviour: what structure it designs (tree vs dag vs mesh vs ...), how it coordinates inside that structure, how it uses environments such as repls, and how all of that differs between models
this takes a different view from prime agent or dynamic workflows. both leave the structure to the model, but each hands it a fixed vocabulary: dynamic workflows a script of agent(), parallel() and pipeline(); prime agent a repl with rlm() sub-calls and messages only between parent, sibling and child.
when it comes to multi-agent harnesses, my gut feel says we should build harnesses that tap into the emergent properties of the model. every model is different as well. why not have self-designed harness on the fly.
jp retweeted
RELEASE DAY
After almost 10 years of hard work, tireless research, and a dive deep into the kernels of computer science, I finally realized a dream: running a high-level language on GPUs. And I'm giving it to the world!
Bend compiles modern programming features, including:
- Lambdas with full closure support
- Unrestricted recursion and loops
- Fast object allocations of all kinds
- Folds, ADTs, continuations and much more
To HVM2, a new runtime capable of spreading that workload across 1000's of cores, in a thread-safe, low-overhead fashion. As a result, we finally have a true high-level language that runs natively on GPUs!
Here's a quick demo:










