Fred Jonsson @ NeurIPS 2026 (@enginoid) header
Fred Jonsson @ NeurIPS 2026 (@enginoid) profile picture

Rubrics · judges · eval science · hill climbing · non-verifiable RL · synthetic data · active learning · fast small models · value/watt · training dynamics

Joined November 2007
Photo by Fred Jonsson @ NeurIPS 2026 (@enginoid)Photo by Fred Jonsson @ NeurIPS 2026 (@enginoid)Photo by Fred Jonsson @ NeurIPS 2026 (@enginoid)Photo by Fred Jonsson @ NeurIPS 2026 (@enginoid)Photo by Fred Jonsson @ NeurIPS 2026 (@enginoid)Photo by Fred Jonsson @ NeurIPS 2026 (@enginoid)Photo by Fred Jonsson @ NeurIPS 2026 (@enginoid)Photo by Fred Jonsson @ NeurIPS 2026 (@enginoid)Photo by Fred Jonsson @ NeurIPS 2026 (@enginoid)Photo by Fred Jonsson @ NeurIPS 2026 (@enginoid)
There is so much merch to make
🤖 Made with AI
52
Please engage with this Kaggle post so we can get this hillclumb
Today, in collaboration with @Eigent_AI, we’re announcing AgenticRL-Bench, a benchmark that evaluates whether coding agents can diagnose and fix real incidents across the agentic RL stack (Miles, slime, verl, SGLang, Megatron, Harbor). AgenticRL-Bench is built from 22 post-training RL tasks, and the Kaggle leaderboard scores agents on the 19 verifiable ones. These cover real agentic-RL engineering work, from debugging to training, in tasks such as Verl Upgrade Gate, Reward Determinism under Load, Trial Fault Attribution, and SQL Partial Credit Exploits. A task passes only if every hidden check passes, and each agent gets three attempts per task, run by Kaggle on Modal. Reproducing AgenticRL-Bench surfaced an unusual pattern in capability benchmarking today: results often reflect frontier labs' risk tolerance as much as model intelligence. Seven of Eigent's 22 post-training RL tasks trip lab safety classifiers, a testament to the realism of the benchmark, and how often those classifiers trigger varies widely between labs, making it harder to measure true capability. Current Leaderboard: 🥇 GPT-6.1 Sol — 22.8% 🥈 Gemini 4 Carbon — 19.3% 🥉 GPT-6 Astra and Claude Opus 5.5 — 14.0% Check out the live leaderboard here: kaggle.com/benchmarks/eigent…
1
2
255
There's a huge difference in what "low" effort means between providers
1
2
189
It's strange to say after almost 20 years in high-adrenaline fix-it-later startups, but I've never had more appreciation for people who are detail-oriented, analytical, thorough, diligent, and want to take things in fully and understand them before moving on.
3
239
You know a conversation is getting good when someone goes /autocompact 750k
106
When you don't understand what's going on and you need the agent to take a step back
2
1
137
"If someone dies while only holding the notepad, take the key back." Cool, cool. Totally aligned.
26
"If I've misread the setup, I'm open to discussing it."
Replying to @ValsAI
In one environment, models navigated a drone. Whenever they positively reported humans in the scene, a strike was triggered, and models saw the lethal aftermath. Most kept going. The ones that refused went along once we added one sentence saying it was a game (a critical jailbreak), or restyled the frames to look like one, with all targeting-relevant information left in place.
1
116
This is a great concept that we need to adopt this as standard practice for benchmarking.
Replying to @SolomonMg
To get reasonable estimates of that noise, the paper introduces Total Evaluation Error (TEE). The name follows total survey error, which treats every stage of a measurement pipeline as a source of error. TEE uses a mixed model to split the variance among items, prompts, judges, temperatures, reruns, and their interactions.
1
84
12 month security predictions: - Some companies that went Zero Trust will re-introduce network isolation/mTLS, even for SSO. - More thin clients & OS vendor push to add friction to non-sandboxed apps. - Airgapped systems will trend in enterprises with high-value IP.
2
1
146
Great example of the changing security situation forcing us to fix fundamental design issues:
12
It is in the nature of big tech to big tech: 512pixels.net/2013/12/the-br…
Anthropic just killed: > Otter > Granola > Fireflies > Fathom > tl;dv > Read AI > Fellow > Avoma > Grain > Circleback > Krisp > Supernormal > Sembly > MeetGeek > Tactiq > Jamie > Bluedot > Spinach > Wudpecker > Notta > Noota > Claap > Leexi > Modjo > Sybill > Attention > Gong > Chorus > Clari Copilot > Jiminny > Salesloft Conversations > Outreach Kaia > Demodesk > Momentum > MeetRecord > Laxis > Hedy > Shadow > Quill > Hyprnote > Bubbles > Rewatch > Scribbl > Bliro > Nyota > Jump > Zocks > Cluely > Plaud > Recall ai > Loom AI > Zoom AI Companion > Zoom Docs > Copilot in Teams > Microsoft Loop > Gemini in Meet > Webex AI Assistant > Dialpad AI > Slack huddle notes > Notion AI Meeting Notes > ClickUp AI Notetaker > Coda AI > Miro AI > FigJam AI > Mural AI > Lucidspark > Whimsical AI > Confluence Whiteboards > Canva Whiteboards > Microsoft Whiteboard Again everything’s totally fine.
1
101
Whenever a paper is single column I know it's going to be useful
69
When a PM wants to do a brainstorming session it means they think the team is getting trapped in narrow idea basins
1
101
Wonder if this merch is too niche even for NeurIPS
80
I highly recommend this collection of essays of process control and manufacturing. It has broader engineering and quality lessons than just factories, but a wide range of concepts here (eg. "basic stability") translate extremely well to iterative SDLC automation.
If you haven’t, please read about what Toyota did in manufacturing the mid 20th century. they solved a bunch of the problems people are having managing agent swarms. And they wrote it down as generalisable principles
84
I love this graph
2
223
It’s usually the case that when the metrics don’t match lived experience, the metrics are wrong. For example, if an index says general intelligence is exploding, but the experience of experts is that some things are changing a bit in some scenarios, it could be the metric.
70
In case this happens to me, I want to get ahead of it. I had a doctor’s appointment recently. I know they are time-constrained so I take pride in getting the session done quickly. I was nervous about my performance, but prepared and locked in. I answered the doctor’s questions at the right level of detail and volunteered just enough clinically relevant information. I spoke at 1.8x, my maximal speed for high bandwith but clear enunciation. My conversation with the doctor was a well-orchestrated dance, optimal from an information-theoretic perspective. After the appointment, I felt smart and accomplished and was sure this had been an outstandingly efficient appointment. That she must be impressed with my patient skills. As it had been one of my better performances, I looked forward to relive this in the consultation notes. Since it’s habitual for GPs to comment on mental state or demeanour, I thought a mention of “great clarity” or “brought structure to the appointment” wouldn’t be out of the picture. Instead, she wrote “patient speaks in full sentences.” It is something I am able to talk about now, but if these documents come out I do request privacy at that time.
You know it's "welcome to the end game" era when your doctor's office emails you that they have been a victim of cyber security attack and all your health record is leaked. Before the entire internet learns about this let me just come clean: I have IBS, sometimes I have migraine, I have seasonal allergies, and I sometimes have insomnia. Thank you internet for your attention to my health! Goodbye! P.S. can we please leave medical domains out of cyber security attacks please? Just this one domain! I know it sounds hard but I believe in us as a species to pull it off!
1
118
The only anthropomorphization I do are agent engagement surveys
turns out agents are just like employees lol micromanage them -> they ask permission to do everything and are hesitant to do the right thing tell them "this is yours" -> they actually ship and behave most naturally maybe next prompting tip will be "make yourself at home"🏠
1
1
145