Research Scientist @Google | Self-improving agents: environments, skills, RSI | Shipped Gemini Deep Research (Enterprise & Max) | Prev: Apple, UCSD.

Sunnyvale, CA
Joined October 2013
Chen-Yu Lee retweeted
Banger paper from Google Cloud AI Research on agents doing professional engineering work. (bookmark it) For professional engineering deliverables, the best agent configuration averages 50.6%, using 27M tokens and 63 minutes per task. They introduce a benchmark called PLE (Physical-Engineering's Last Exam). It has 85 tasks across electrical, civil, structural, and mechanical engineering. Solving a task means producing a billable deliverable, such as a design package a licensed reviewer would sign off on, from a multi-format project corpus with physics and building-code requirements. Some tasks are written end to end by practicing engineers. Others start from an engineer's problem statement, and an adversarial Creator-Solver agent pipeline builds the project environment, grader, and reference deliverables before experts validate them and make them harder. Frontier models were tested under seven public agent harnesses. Most existing engineering benchmarks are textbook QA or toy design problems that leading agents already saturate. arxiv.org/abs/2610.30537 Chat with Paper: academy.dair.ai/papers/ple-p…
13
23
2
123
8,377
14 frontier models × 7 harnesses on our private test set: 🥇 Claude Opus 5.5: 48.3% 🥈 GPT 6 Astra: 45.8% 🥉 Gemini 4 Argon: 40.6% ⚡ Gemini 3.8 Flash: 27.5% Wildest finding: Harness choice alone swings score by up to 12.1% and cost by 3x ($13 vs $38/task).
2
2
247
Think your harness can beat 48.3%? Join the PLE Open Competition: • Bring your harness (hidden test set) • 13 Google-sponsored models run FREE • Prizes coming soon 📄 Paper: arxiv.org/abs/2610.30537 🌐 Leaderboard: ple-bench.com 💻 Code: github.com/google-research/p…
1
2
183
PLE Team: @HanRujun, @yufan_zhuang, @ARRay693, Han Yu, Zoey CuiZhu, George Lee, @ming5_alvin, Binz Roy, Varun Sakalkar, Vishy Tirumalashetty, Burak Gokturk, @tomaspfister, @chl260
1
175
HLE tested knowledge. ALE tested digital work. PLE tests the work that builds the physical world. Today we’re releasing Physical-Engineering’s Last Exam (PLE) from Google Cloud AI Research. All frontier models scored under 50% 🧵
5
7
2
45
3,987
What a PLE task looks like across all 4 disciplines: 🏗️ Structural: Design a roof deck 🗺️ Civil: Lay out a full campus ⚡ Electrical: Size a switchboard & cable schedule ❄️ Mechanical: Design a cooling loop & clash-free 3D equipment hall
1
2
275
AI writes code, but can it design a power grid, chip fab, or skyscraper? PLE has 85 long-horizon tasks in Mechanical, Electrical, Civil & Structural engineering. Agents drop into real project folders (~61 files, outdated revisions as traps) & must pass licensed engineer’s bar.
1
2
372
Agents can remember now. Self-improvement over long horizons is the harder part. My guest lecture at UC Berkeley this week: memory → lessons → skill curation → better environments → safe self-improvement. Slides: chenyulee.com/berkeley
1
3
19
749
The question I left the class with: when should experience and insights go back into the weights? The harness is cheap to edit; weights generalize. Where's the line?
1
40
We evolved an agent on one benchmark, froze it, and tested it on five it had never seen. Other self-improvement methods barely transfer, and some get worse. RRSI improved on every one, up to 24% over its starting score, with ~30% fewer tokens.
1
1
4
524
The agent can try any change it wants. But a change only sticks if it: helps on new problems is more than luck is worth the complexity it adds Changes that stop helping get removed.
1
2
5
488
We've seen this before. Neural networks overfit too, until regularization taught them to generalize. Self-improving agents need the same fix. Introducing RRSI: Regularized Recursive Self-Improvement of Agent Harnesses.
1
1
8
640
Your self-improving agent is probably overfitting. Agents can now rewrite their own prompts, tools, memory, and workflows, round after round. Scores rise on the tasks they practice on. On new tasks, the gains often disappear. Our fix: regularization. 🧵
9
18
1
179
9,376