
Research Scientist @Google | Self-improving agents: environments, skills, RSI | Shipped Gemini Deep Research (Enterprise & Max) | Prev: Apple, UCSD.
Sunnyvale, CA
Joined October 2013
- Tweets78
- Following1.1K
- Followers448
- Likes251
Banger paper from Google Cloud AI Research on agents doing professional engineering work.
(bookmark it)
For professional engineering deliverables, the best agent configuration averages 50.6%, using 27M tokens and 63 minutes per task.
They introduce a benchmark called PLE (Physical-Engineering's Last Exam). It has 85 tasks across electrical, civil, structural, and mechanical engineering. Solving a task means producing a billable deliverable, such as a design package a licensed reviewer would sign off on, from a multi-format project corpus with physics and building-code requirements.
Some tasks are written end to end by practicing engineers. Others start from an engineer's problem statement, and an adversarial Creator-Solver agent pipeline builds the project environment, grader, and reference deliverables before experts validate them and make them harder.
Frontier models were tested under seven public agent harnesses. Most existing engineering benchmarks are textbook QA or toy design problems that leading agents already saturate.
arxiv.org/abs/2610.30537
Chat with Paper: academy.dair.ai/papers/ple-p…
14 frontier models × 7 harnesses on our private test set:
🥇 Claude Opus 5.5: 48.3%
🥈 GPT 6 Astra: 45.8%
🥉 Gemini 4 Argon: 40.6%
⚡ Gemini 3.8 Flash: 27.5%
Wildest finding: Harness choice alone swings score by up to 12.1% and cost by 3x ($13 vs $38/task).
Think your harness can beat 48.3%?
Join the PLE Open Competition:
• Bring your harness (hidden test set)
• 13 Google-sponsored models run FREE
• Prizes coming soon
📄 Paper: arxiv.org/abs/2610.30537
🌐 Leaderboard: ple-bench.com
💻 Code: github.com/google-research/p…
PLE Team: @HanRujun, @yufan_zhuang, @ARRay693, Han Yu, Zoey CuiZhu, George Lee, @ming5_alvin, Binz Roy, Varun Sakalkar, Vishy Tirumalashetty, Burak Gokturk, @tomaspfister, @chl260
HLE tested knowledge. ALE tested digital work. PLE tests the work that builds the physical world.
Today we’re releasing Physical-Engineering’s Last Exam (PLE) from Google Cloud AI Research.
All frontier models scored under 50% 🧵
AI writes code, but can it design a power grid, chip fab, or skyscraper?
PLE has 85 long-horizon tasks in Mechanical, Electrical, Civil & Structural engineering.
Agents drop into real project folders (~61 files, outdated revisions as traps) & must pass licensed engineer’s bar.
Agents can remember now. Self-improvement over long horizons is the harder part.
My guest lecture at UC Berkeley this week: memory → lessons → skill curation → better environments → safe self-improvement.
Slides: chenyulee.com/berkeley
We evolved an agent on one benchmark, froze it, and tested it on five it had never seen.
Other self-improvement methods barely transfer, and some get worse.
RRSI improved on every one, up to 24% over its starting score, with ~30% fewer tokens.
Make every change earn its place.
Led by our amazing intern Peng Xia, with a great team from Google Cloud AI Research, UNC-Chapel Hill, Stanford, and WashU.
📄 arxiv.org/abs/2609.24972
💻 github.com/google-research/r…
🌐 regularized-rsi.com
@richardxp888 @HanRujun @ZifengWang315 @anmourchen @yufan_zhuang @yoonholeee @ChengsongH31219 Han Yu, Zoey Cuizhu,
@ming5_alvin @HuaxiuYaoML Burak Gokturk @tomaspfister @chl260
The agent can try any change it wants. But a change only sticks if it:
helps on new problems
is more than luck
is worth the complexity it adds
Changes that stop helping get removed.
We've seen this before. Neural networks overfit too, until regularization taught them to generalize.
Self-improving agents need the same fix.
Introducing RRSI: Regularized Recursive Self-Improvement of Agent Harnesses.









