Robotics + AI System PhD @UCBerkeley
Berkeley, CA
Joined August 2022
- Tweets33
- Following300
- Followers380
- Likes118
Pinned Tweet
We gave Claude and GPT a phone video of a robot wiping. In an hour, each built a full-dynamics digital twin.
Introducing RoboSimStudio 🤖: an open and interactive toolkit for Agentic Robotics, enabling Fable, Astra and Argon AI agents to model sponges 🧽, cables 🧵 and cloth 👔.
Weekend project after codex reset. I took one photo of my desk, then used AI agent to turn it into a Microduck playground 💛 like Minions taking over my workspace.
Built with RoboSimStudio. Everything is open.
I took a picture with my desk with complex outdoor lighting but Real2Sim with Astra can reconstruct the scene and bring the Microducks (thanks to @pollenrobotics, @NVIDIARobotics) to life.
Below is the only input I gave to RoboSimStudio:
Everything is built with RoboSimStudio, our open-source framework for agentic robot simulation developed by @berkeley_ai and @NVIDIARobotics.
Try it yourself 👇
github.com/robosimstudio/rob…
And code for generating the video
github.com/KeplerC/microduck…
Thank you all for the interest and support!!! More features on agentic real2sim will be released in the next couple of days :)
That's genuinely insane! 🤯
An AI agent built a physics twin of a robot task from a phone video in under an hour.
This project RoboSimStudio was built on Newton by researchers from UC Berkeley and NVIDIA Robotics.
Open toolkit, every physics parameter exposed as something a human can slide or an agent can edit.
The demo is a bimanual setup, one arm holding a bar, the other wiping a marker stripe off it with a foam sponge. Input was a smartphone video.
What's more important no CAD, and no calibrated cameras. Claude and GPT-6 Astra each built a simulation of it independently. One finished in 56 minutes.
Then they ran it on the real robot and the bar support moved. In simulation it had been modelled as fixed, so the sponge kept contact. In reality it shifted and contact was lost.
The agent went back and replaced the fixed constraint with a free rigid body. The revised simulation then reproduced the tipping the original could not.
That is the loop actually closing. The real world found the modelling error and the fix went back into the twin.
🔗 Check the project here: lnkd.in/dgXJxAhw
~~
♻️ Join the weekly robotics newsletter, and never miss any news → ziegler.substack.com
Kaiyuan Eric Chen retweeted
Follow Gemini 4 Argon's report, I implemented their robotics harness.
Gemini did a zero-shot bimanual handover. Now open source 🤖👇
Kaiyuan Eric Chen retweeted
😫Tired of spending hours teleoperating robots?
🚀What if you could collect 2.9× more demonstrations with the same human effort?
Introducing TANDEM: our new work that combines autonomous task and motion planning (TAMP) with human teleoperation only when needed. 🧵 1/n
Kaiyuan Eric Chen retweeted
Gemini-4 Argon shows a large improvement in robotics understanding on RoboVista.
It reaches 81.9%, beats GPT-6 Astra and Claude Fable 5.1 in our evaluation with lowest eval cost. Compared with Gemini 3 Flash, it improves across every robotics domain. Example VQA👇
🤖How well do today's VLMs actually understand real-world robotics? 👀
Excited to share RoboVista at #RSS2026 — a systematic evaluation and benchmark for VLMs across diverse, real-world robot applications. Website, dataset and paper: berkeleyautomation.github.io…
Developed by researchers at @UCBerkeley, @GoogleDeepMind, and @Princeton. 🧵👇
The paper, code, examples, and interactive 3D rollouts are all open
🌐 Site, 📄 Paper, 🎮 Demo: robosimstudio.github.io
💻 Code: github.com/robosimstudio/rob…
Thanks to the great team!
@eric_heiden @erwincoumans @milesmacklin #spencer_huang from @nvidia @NVIDIARobotics and
@Shuangyu0_0Xie, @UksangYoo, @XinyueZhuXY , Cael Magner, @alvinyzhu , @talking_kim, and @Ken_Goldberg from @UCBerkeley @berkeley_ai
I will be presenting RoboVista at #RSS2026 today! This joint work with @GoogleDeepMind assess how effectively current VLMs process diverse robotic applications (esp. field robotics!) . Fun fact: most of all annotators already have PhD degrees. Come talk to me at poster session!
🤖How well do today's VLMs actually understand real-world robotics? 👀
Excited to share RoboVista at #RSS2026 — a systematic evaluation and benchmark for VLMs across diverse, real-world robot applications. Website, dataset and paper: berkeleyautomation.github.io…
Developed by researchers at @UCBerkeley, @GoogleDeepMind, and @Princeton. 🧵👇
Graph-as-Policy (GaP) is a new variant of Agentic Robotics from @NVIDIA and @UCBerkeley that builds computation graphs (like ROS) to ensure modularity, manage complexity, and facilitate interpretability.
🧵Open code and paper: graph-robots.github.io/gap/
#Robotics #Automation #AgenticAI #EmbodiedAI
The paper, more demo videos, 8 benchmarks, the skill library, and the code are all open. 🔓
🌐 Site: graph-robots.github.io/gap/
📄 Paper: arxiv.org/abs/2607.05369
Thanks to the great team !
including: @UksangYoo @ruwantw from @BoschGlobal
@guanzhi_wang @Adithya_Murali_ @balakumar_ @yukez @DrJimFan #spencer_huang from @nvidia
@Shuangyu0_0Xie @letian_fu @uynitsuj @talking_kim @sandeep_b24 @HaoruXue from and @Ken_Goldberg from @UCBerkeley
How GaP works:
🗣️ Task Description: Describe a task (e.g., "make popcorn - repeat"), workcell, object and pose range, and provide a set of model-based and model-free skills.
🧠 Multi-Agent Orchestration: A multi-agent harness, built over coding tools like #Claude and #Gemini, decomposes a special prompt and assembles an interpretable computation graph from an open skill library
🔄 Self-Learning in Sim: The graph is rehearsed in simulation, using contact feedback to diagnose its own failures, and rewrites its own structure until performance plateaus.
🦾 Sim-to-Real: The optimized graph is then exported to run on the physical robot.
GaP can provide the interpretability and reliability of classical engineering, but with coding agents doing the authoring and fine tuning. 🤯
