
PhDing @Stanford, AI & robotics. Prev @MetaAI @NVIDIAAI @berkeley_ai🐻
Joined May 2017
- Tweets178
- Following1.4K
- Followers5K
- Likes2.5K
Pinned Tweet
How to learn dexterous manipulation for any robot hand from a single human demonstration?
Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.
Mandi Zhao retweeted
Most robot policies treat time implicitly, using sequence position as a proxy for physical time. But real robot systems rarely run on a fixed clock. Sensors run at different rates, observations arrive with delays, and actions take time to compute and execute. Successful control depends not only on what the robot observes but also on the timing of observations and actions.
We introduce 🕒 Chronos Policy, a diffusion policy that explicitly conditions on observation capture times and requested action execution times. We use time-aware positional encodings to make attention depend on elapsed physical time. Because timestamps let the policy interpret each sequence's timing, we can randomize sensing and action timing during training without making those variations ambiguous. This exposes the policy to a wider range of timing conditions, improving robustness to timing changes at deployment.
Operator-supervised real world RL that improves the success rate on high-precision tasks. Really like the strictly blind eval protocol and statistical rigor mulligan.page/#how-the-evalu…
I've been thinking a lot about making robot learning scalable. To me, that means learning on the job, and using the human help still needed efficiently.
In Mulligan ⛳️, we focus each data-collection round on the initial states where the robot fails. This happens in deployment, with the robot doing the task and a person stepping in when needed.
On 3 real tasks and 2,550 blind evals, it improves on uniform collection at the same budget.
Excited for @larsankile to release this! Maybe now he'll finally have time to golf with us again🤔
Mandi Zhao retweeted
When a robot learns a new behavior, what keeps it from forgetting the ones it already knows?
Prior work, especially on VLAs, shows that rehearsing past experiences can enable strong continual learning. But why does rehearsal work so well, and when does it fail?
We find that a small but ✨special subset✨ of rehearsed data largely determines whether old behaviors are retained or forgotten. We call these examples Memory Anchors. ⚓
Withholding just the top 10% of Memory Anchors from rehearsal increases forgetting by up to 4.5x. Increasing their presence, meanwhile, reduces task forgetting and enables a real robot to learn challenging task sequences.
Website: robot-adaptation.github.io/M…
Paper: arxiv.org/abs/2608.26545
Curious? Read on! 🧵👇 (1/9)
WAMs + force data improves on both pre-training and fine-tuning tasks:
Can we enable robots to develop a sense of touch without forgetting what they learned from large-scale vision-only pretraining?
Introducing MultiSensory World Model (MuSe) 🌍: A new approach for finetuning visuomotor policies on minimal data from new sensor modalities, such as force/torque (F/T)
With Muse, touch learned later improves skills learned earlier — a small amount of F/T data on new tasks improves zero-shot on diverse pretraining tasks that were never supervised with F/T
We believe MuSe provides a practical pathway towards training multisensory foundation models that leverage both abundant vision data, and smaller multisensory datasets 🧵👇
Mandi Zhao retweeted
🌱 How do you control a robot whose body is constantly growing, buckling, and reshaping?
📷 Put 19 cameras on it.
Meet PanoVine, the first autonomous vine robot system.
We distribute 19 cameras along a 6 meters, 7-DoF soft growing vine robot, giving it whole-body visual feedback of both itself and its surroundings. Using these observations, we train an end-to-end visuomotor policy that autonomously navigates branches, slopes, gaps, turns, and obstacles.
🔗 panovine-bot.github.io
Mandi Zhao retweeted
Can we learn whole-body mobile manipulation directly from human demonstrations?
Introducing Whole-Body Mobile Manipulation Interface (HoMMI)
Egocentric + UMI, 0 teleop -> bimanual & whole-body manipulation, long-horizon navigation, active perception
hommi-robot.github.io
Mandi Zhao retweeted
SimToolReal 🤖🔨is a generalist dexterous manipulation policy that works:
* zero-shot sim→real
* zero-shot on novel tools
* zero-shot on novel tasks
It performs grasping, in-hand rotation, & tool-use!
I love the motor & tool interaction sounds 🔊
See @kushalk_ thread for more
Mandi Zhao retweeted
For video generation in robotic applications, looking pretty is usually not enough.
Robot manipulation requires understanding how visual observations and 3D geometry evolve over time under agent actions, with temporal coherence and geometric consistency across camera views.
We study this challenge in our work (recently accepted by @iclr_conf ), 4D Video Generation for Robot Manipulation, which enforces multi-view 3D consistency via geometric supervision to generate spatio-temporally aligned videos.
Mandi Zhao retweeted
Can we quickly improve a pre-trained robot policy by learning from real world human corrections? Introducing Compliant Residual DAgger (CR-DAgger), a system that improves policies performance to close to 100% on challenging contact-rich manipulation problems, using as few as 50~100 episodes of human corrections. Co-lead by @XiaomengXu11 and I, CR-DAgger quickly learns a force-aware residual policy even when the base policy is position-only. CR-DAgger already won the best paper award at the Human2robot workshop at CoRL 2025, and will be presented at NeurIPS tomorrow Dec 3 at poster #2314. Come talk to us if you are interested!
- NeurIPS paper: arxiv.org/abs/2506.16685
- Extended version with more experiments & learnings: compliant-residual-dagger.gi…
- Full code and instructions: github.com/yifan-hou/cr-dagg…
Mandi Zhao retweeted
How can we enable finetuning of humanoid manipulation policies, directly in the real world?
In our new paper, Residual Off-Policy RL for Finetuning BC Policies, we demonstrate real-world RL on a bimanual humanoid with 5-fingered hands (29 DoF) and improve pre-trained policies with ~15-75 minutes of robot interaction.
By learning residual corrections on frozen BC policies using sample-efficient off-policy RL, we achieve significant improvements in sample efficiency, enabling policy finetuning directly on the hardware — to our knowledge, one of the first examples of this on a humanoid with bimanual dexterous hands.
(If you know of other examples, let me know!)
ICPC results are amazing and all, but just today ChatGPT told me, confidently and assertively, that scipy’s Rotation.from_quat does *not* take scalar_first keyword — when it in fact does take it, and this whole paragraph is trash.
Makes me wonder how big the gap is between these models achieving “superhuman coding” and actually doing the simplest tasks reliably.
One amusing takeaway from doing RL in these massively parallelized sim environments is that reward engineering matters more so than ever.
A small detail in reward function could make a huge difference: with 10k+ parallel threads to explore in, the policy Will exploit any caveat and find a shortcut to high rewards and not going with the desired behavior.
The two videos below are only one line difference in implementation: left computes the average distance between the 4 fingertip keypoints and the object, and right is adding the palm as another keypoint in the same function, which quickly leads to the grasping motion and lifts up the object.
Another detail here is how to compute distance-based reward in batched setting: a common equation you will see in papers is ‘rew = exp(-beta * dist)’, but when you have multiple keypoints, i.e. dist has shape (B, N) where B is number of environments and N is number of keypoints, you want to do ‘rew = torch.mean(exp(-beta * dist))’. Because taking the average first, i.e. rew = exp(-beta * dist.mean())) - will make the reward much sparser and harder to achieve.
I’m a big fan of this line of work from Columbia (also check out PhysTwin by @jiang_hanxiao: jianghanxiao.github.io/physt…) they really make real2sim work for the very challenging deformable objects, And show it’s useful for real robot manipulation.
So far it seems a bit limited to object-centric representations, looking forward to this being extended to simulate&render bigger scenes.
Can we learn a 3D world model that predicts object dynamics directly from videos?
Introducing Particle-Grid Neural Dynamics: a learning-based simulator for deformable objects that trains from real-world videos.
Website: kywind.github.io/pgnd
ArXiv: arxiv.org/abs/2506.15680
Code: github.com/kywind/pgnd
Demo: huggingface.co/spaces/kaifz/…
To appear at #RSS2025
Mandi Zhao retweeted
Normally, changing robot policy behavior means changing its weights or relying on a goal-conditioned policy. What if there was another way?
Check out DynaGuide, a novel policy steering approach that works on any pretrained diffusion policy. dynaguide.github.io/ 🧵
Our lab at Stanford usually do research in AI & robotics, but very occasionally we indulge in being functional alcoholics -- Recently we hosted a lab cocktail night, and created drinks with research-related puns like 'reviewer#2' and 'make 6 figures', sharing the full recipes here in case you are looking for a fun summer drink:
The recipes are twists from classic drinks like Palomas and Martinis. The names are inspired by the drink tastes: 'reviewer#2' is spicy with jalapeños, 'in-between-projects' because it's chill and relaxing.
shout out to my co-host @du_maximilian!



















