eigenpaul (@eigen_paul) header
eigenpaul (@eigen_paul) profile picture

deep in the RL mines teaching models to do science. resident @PrimeIntellect

Groningen
Joined August 2024
Photo by eigenpaul (@eigen_paul)Photo by eigenpaul (@eigen_paul)Photo by eigenpaul (@eigen_paul)Photo by eigenpaul (@eigen_paul)Photo by eigenpaul (@eigen_paul)Photo by eigenpaul (@eigen_paul)Photo by eigenpaul (@eigen_paul)Photo by eigenpaul (@eigen_paul)Photo by eigenpaul (@eigen_paul)Photo by eigenpaul (@eigen_paul)
Happy to finally release TransitBench, a benchmark for autonomous scientific programming in astronomy. It measures how far AI agents can automate the search for exoplanets.
1
1
7
391
eigenpaul retweeted
In my ongoing quest to find the limits of Opus 5.5, I directed it to audit my old scientific field, theoretical astrophysics. We indexed about 150,000 papers, audited 50,000 for issues, and did progressively more thorough checks on thousands more. We found nearly 1000 major errors that got cited many times, most are still uncorrected. This survey enabled me to estimate the efficacy of peer review for correcting errors (not very good) and to identify dozens of open questions that anyone with spare Claude Max tokens could answer today (.md context files included). Bottom line: AI can speed up quantitative science by >1000x with no further model improvement required. But I'm seeing no evidence of this being pushed within universities, only academics complaining about labs dropping math proofs. Among my fellow reformed academic high agency founders, however, several are speed running entire disciplines. This survey has implications for how science will be run in the very near future. Hint: Academic publishing is dead. It was already a very stupid way to record and promulgate knowledge, and had been so for decades, even before all that publicly funded knowledge was locked behind paywall cartels. Now AIs can generate thousands of novel, correct papers per day, it is no longer meaningful for humans to consider a research paper a "unit of science" or to even bother much with the 99.9% of science which, up to this point, was necessary drudgery. Time to discover something like General Relativity or Evolution every week. LFG!! Enjoy: science.caseyhandmer.com/phy…
173
376
90
2,738
94,192
eigenpaul retweeted
Can AI agents correctly identify periapical lesions on a panoramic x-ray of the jaw? Introducing Periapical Lesions Bench (dylantirandaz.com/projects.h…) 🧵
5
12
3
38
2,086
please add a sixth one. i need kilometers of diamond nanothread for orbital tethers
Today, we announce five Grand Challenges in Material Science. We’re also committing $1M in prize money to anyone who can solve these challenges. AI can solve Millennium Prize problems in math. Can it do the same in material science? To find out, we reached out to 10 leading professors to identify the hardest open problems in the field.
1
41
eigenpaul retweeted
huge if true
22
57
9
525
22,435
at aiGrunn today, love seeing a full talk on ai scientists
1
57
What are idea basins and how do agents explore them? A few weeks ago, researchers from different labs independently discovered what we now call ‘Idea-basins’: As an autonomous agent iterates to solve a problem, it gets stuck into the methods it has tried before and its trajectories remain in localized regions of the solution space. Techniques like ‘Fork-and-Flush’ and ‘Scaling Discovery requires forgetting’ reduce this idea fixation by periodically deleting the session context. In this post we study how frontier models explore different idea basins for AutoResearchExam. E.g. for a problem on accelerating transformer decoder graphs on CPUs models can try Numpy+Asm Kernels, Native BLAS, JIT compilers and other idea basins. We find that some models switch ideas more than others while others get to the best idea faster and explore deeper. We find that most runs stick with one or two ideas, and models that submit more spend the extra submissions refining and overfit more. We see that models explore most in the first six hours, after which visits to new idea basins drop sharply. This suggests models start refining early leaving potential gains on the table: 58% of runs never reach the basin where the best solution to their task was found. Finally, bold exploration pays off: exploring a new idea gains about nine times as much as a refining step on average. Read more in our blog: bespokelabs.ai/blog/how-mode…
50
121
34
868
168,718
eigenpaul retweeted
Of course that's your contention. You're a first-year RL engineer. You just read Prime Intellect's prime-rl writeup and saw rollout workers streaming trajectories into a trainer that never waits on them, so naturally you believe decoupling generation from training is free throughput and staleness is a rounding error. Next month, when you get your hands on an RL cookbook, the Unsloth GRPO handbook, probably, you'll be singing from the treetops about colocating vLLM inside the training process, and the 400 milliseconds you save on every weight sync by waking and sleeping the optimizer like you have not just moved the idle GPU somewhere else. That'll last until sometime next year, when you watch a livestream of Will Brown talking about verifiers and become convinced that none of the systems work matters, because an environment is just a Python function and the reward figures itself out. "Well, as a matter of fact, I won't, because a verifier..." "A verifier is exactly the thing your policy learns to game first. It finds the one branch you forgot to test and farms it for maximum reward in fifty steps, and your async sampler is already three policy versions stale while it does it, so the importance ratio is fiction by the time the update lands." Yeah, I read that too. Every training-inference mismatch thread since the sampler's logprobs and the trainer's logprobs first disagreed in BF16 and nobody noticed for 200 steps. Do you have any thoughts of your own on the subject, or should I just point my agent at the prime-rl repo, set group size to 64, and let it pay an H100 to argue with itself for me?
27
70
23
1,166
63,442
eigenpaul retweeted
A few weeks ago, @dexhorthy released a great blog on how frontier agents performed on SlopCodeBench. Now, slop is moving up the stack and destroying your codebase with duplicate code.
6
21
3
98
12,456
Currently I still only know of 1 alignment-specific RL environment company: making RL envs that aim to represent situations in real deployments in which alignment is challenging. It seems like a huge mistake in hindsight that we have seen so much encouragement and funding to, for example, start 3rd party evaluators, but not comparable encouragement to start alignment data companies. I do think a lot of traditional AI safety views underestimate the importance of data, so perhaps this is related. But the Hugging Face incident seems squarely downstream of both (a) reward hackable RL envs in general (b) much less effort and $ put into RL envs where alignment is a core component of reward. A separate idea is alignment by default: the old idea of "find the persona basin which is aligned" and things will just work. This is clearly not guaranteed under increased RL. But my sense is there has not been a huge amount of effort to figure out how custom environments could help "nudge" the model back into good persona basins during extensive RL. Here too, a robust ecosystem of alignment data companies could help significantly. In the RL data market, there is a trend for some labs to in-house slightly more RL env production. But I don't expect this to fully disappear, and I am convinced the market-based approach to sourcing RL envs produces the best overall volume and quality—for alignment RL, an ecosystem of startups would also greatly help make behind-frontier models safer.
22
27
5
296
18,659
eigenpaul retweeted
announcement: i'm launching a $100,000 grant for amateur astronomy research: claudestronomy.com over the past few weeks, @joshuawolk @LukeFarritor and i have been digging through public telescope data, hunting for new planets, asteroids, and more our first release: we found a new dwarf planet in our solar system!! it's insane how much you can find with some claude + codex credits it's about 1/4 the width of Pluto and orbits about twice as far (way past Neptune), and looks like it's about the 50th largest object in our solar system. @joshuawolk is in the process of confirming it this dwarf planet has sat in public images for over a decade without anyone noticing. there's tons more out there: all sitting in public NASA/ESA/etc. databases for you to analyze there's tons of stuff still out there, sitting in data that astronomers don't have enough time to go through all of you can do this too! apply for a grant. more info below
Claude and I found a dwarf planet candidate. It's the ~50th largest known object in our entire solar system.
131
146
39
1,599
197,482
Introducing SynthonMap: a zoomable map of billion-to-trillion-scale chemical spaces, using factorized synthons and parametric UMAP density estimation. Try at: synthonmap.mireklzicar.com/?… Demo on @XtalPi VAST 👇
6
30
6
191
12,064
The full release (with paper links): lila.ai/news/a-new-synthesis… Credits 👇
1
3
1
35
3,383
want to turn this into an actual measurement. that plot assumes someone who claims at exactly 5 sigma and i have no idea what sigma ai scientists actually claim at
even at five sigma, an AI testing a million hypotheses a day on pure noise finds about a hundred fake discoveries a year
1
3
155
im also really curious about ideation. right now it has no ground truth, so benchmarks either ask an llm judge or check if a model rediscovers a known paper. in these worlds you can just run every idea and see if its true
1
1
13
which means it could double as a research ideation benchmark. you could compare a single model with a swarm of them and score both on what they actually find, at the same false discovery rate so neither wins by just claiming more
1
14
eigenpaul retweeted
In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety. Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works already-clean.goodfire.com/
53
248
59
1,788
147,228
eigenpaul retweeted
the first agent swarm scaling laws, beautiful
Introducing Claude Fable 5.2 and Claude Mythos 5.2, designed to tackle the most daunting technical challenges. They’re capable collaborators on ambitious projects in many fields, and show improvements in areas like coding, reasoning, and vision.
11
27
3
461
18,656