PhDing @Stanford, AI & robotics. Prev @MetaAI @NVIDIAAI @berkeley_ai🐻

Joined May 2017
How to learn dexterous manipulation for any robot hand from a single human demonstration? Check out DexMachina, our new RL algorithm that learns long-horizon, bimanual dexterous policies for a variety of dexterous hands, articulated objects, and complex motions.
21
103
17
625
122,221
Most robot policies treat time implicitly, using sequence position as a proxy for physical time. But real robot systems rarely run on a fixed clock. Sensors run at different rates, observations arrive with delays, and actions take time to compute and execute. Successful control depends not only on what the robot observes but also on the timing of observations and actions. We introduce 🕒 Chronos Policy, a diffusion policy that explicitly conditions on observation capture times and requested action execution times. We use time-aware positional encodings to make attention depend on elapsed physical time. Because timestamps let the policy interpret each sequence's timing, we can randomize sensing and action timing during training without making those variations ambiguous. This exposes the policy to a wider range of timing conditions, improving robustness to timing changes at deployment.
12
43
5
245
170,067
Operator-supervised real world RL that improves the success rate on high-precision tasks. Really like the strictly blind eval protocol and statistical rigor mulligan.page/#how-the-evalu…
I've been thinking a lot about making robot learning scalable. To me, that means learning on the job, and using the human help still needed efficiently. In Mulligan ⛳️, we focus each data-collection round on the initial states where the robot fails. This happens in deployment, with the robot doing the task and a person stepping in when needed. On 3 real tasks and 2,550 blind evals, it improves on uniform collection at the same budget.
1
2
29
2,741
Excited for @larsankile to release this! Maybe now he'll finally have time to golf with us again🤔
1
237
When a robot learns a new behavior, what keeps it from forgetting the ones it already knows? Prior work, especially on VLAs, shows that rehearsing past experiences can enable strong continual learning. But why does rehearsal work so well, and when does it fail? We find that a small but ✨special subset✨ of rehearsed data largely determines whether old behaviors are retained or forgotten. We call these examples Memory Anchors. ⚓ Withholding just the top 10% of Memory Anchors from rehearsal increases forgetting by up to 4.5x. Increasing their presence, meanwhile, reduces task forgetting and enables a real robot to learn challenging task sequences. Website: robot-adaptation.github.io/M… Paper: arxiv.org/abs/2608.26545 Curious? Read on! 🧵👇 (1/9)
9
57
8
281
50,984
This is really exciting, fully matched glove + hand will make data collection so much more scalable
The natural final evolution of UMI devices will be the human hand form factor. Why teleoperate when you can just wear the end-effector like a glove.
1
6
1
109
13,057
WAMs + force data improves on both pre-training and fine-tuning tasks:
Can we enable robots to develop a sense of touch without forgetting what they learned from large-scale vision-only pretraining? Introducing MultiSensory World Model (MuSe) 🌍: A new approach for finetuning visuomotor policies on minimal data from new sensor modalities, such as force/torque (F/T) With Muse, touch learned later improves skills learned earlier — a small amount of F/T data on new tasks improves zero-shot on diverse pretraining tasks that were never supervised with F/T We believe MuSe provides a practical pathway towards training multisensory foundation models that leverage both abundant vision data, and smaller multisensory datasets 🧵👇
3
4
84
11,734
🌱 How do you control a robot whose body is constantly growing, buckling, and reshaping? 📷 Put 19 cameras on it. Meet PanoVine, the first autonomous vine robot system. We distribute 19 cameras along a 6 meters, 7-DoF soft growing vine robot, giving it whole-body visual feedback of both itself and its surroundings. Using these observations, we train an end-to-end visuomotor policy that autonomously navigates branches, slopes, gaps, turns, and obstacles. 🔗 panovine-bot.github.io
5
20
4
112
21,680
Can we learn whole-body mobile manipulation directly from human demonstrations? Introducing Whole-Body Mobile Manipulation Interface (HoMMI) Egocentric + UMI, 0 teleop -> bimanual & whole-body manipulation, long-horizon navigation, active perception hommi-robot.github.io
12
71
7
338
84,780
Mandi Zhao retweeted
SimToolReal 🤖🔨is a generalist dexterous manipulation policy that works: * zero-shot sim→real * zero-shot on novel tools * zero-shot on novel tasks It performs grasping, in-hand rotation, & tool-use! I love the motor & tool interaction sounds 🔊 See @kushalk_ thread for more
🤖 Can a single robot policy manipulate diverse tools without ever seeing them before? Introducing SimToolReal 🔨 : a generalist dexterous manipulation policy that transfers zero-shot sim→real to unseen tools + unseen tasks All videos are 1x speed (60 Hz control) 🧵👇
2
20
3
112
17,498
Mandi Zhao retweeted
For video generation in robotic applications, looking pretty is usually not enough. Robot manipulation requires understanding how visual observations and 3D geometry evolve over time under agent actions, with temporal coherence and geometric consistency across camera views. We study this challenge in our work (recently accepted by @iclr_conf ), 4D Video Generation for Robot Manipulation, which enforces multi-view 3D consistency via geometric supervision to generate spatio-temporally aligned videos.
8
37
1
312
55,960
Mandi Zhao retweeted
Can we quickly improve a pre-trained robot policy by learning from real world human corrections? Introducing Compliant Residual DAgger (CR-DAgger), a system that improves policies performance to close to 100% on challenging contact-rich manipulation problems, using as few as 50~100 episodes of human corrections. Co-lead by @XiaomengXu11 and I, CR-DAgger quickly learns a force-aware residual policy even when the base policy is position-only. CR-DAgger already won the best paper award at the Human2robot workshop at CoRL 2025, and will be presented at NeurIPS tomorrow Dec 3 at poster #2314. Come talk to us if you are interested! - NeurIPS paper: arxiv.org/abs/2506.16685 - Extended version with more experiments & learnings: compliant-residual-dagger.gi… - Full code and instructions: github.com/yifan-hou/cr-dagg…
4
51
11
223
60,520
How can we enable finetuning of humanoid manipulation policies, directly in the real world? In our new paper, Residual Off-Policy RL for Finetuning BC Policies, we demonstrate real-world RL on a bimanual humanoid with 5-fingered hands (29 DoF) and improve pre-trained policies with ~15-75 minutes of robot interaction. By learning residual corrections on frozen BC policies using sample-efficient off-policy RL, we achieve significant improvements in sample efficiency, enabling policy finetuning directly on the hardware — to our knowledge, one of the first examples of this on a humanoid with bimanual dexterous hands. (If you know of other examples, let me know!)
11
56
9
265
53,300
ICPC results are amazing and all, but just today ChatGPT told me, confidently and assertively, that scipy’s Rotation.from_quat does *not* take scalar_first keyword — when it in fact does take it, and this whole paragraph is trash. Makes me wonder how big the gap is between these models achieving “superhuman coding” and actually doing the simplest tasks reliably.
17
6
3
226
29,401
One amusing takeaway from doing RL in these massively parallelized sim environments is that reward engineering matters more so than ever. A small detail in reward function could make a huge difference: with 10k+ parallel threads to explore in, the policy Will exploit any caveat and find a shortcut to high rewards and not going with the desired behavior. The two videos below are only one line difference in implementation: left computes the average distance between the 4 fingertip keypoints and the object, and right is adding the palm as another keypoint in the same function, which quickly leads to the grasping motion and lifts up the object.
6
23
7
379
33,780
Another detail here is how to compute distance-based reward in batched setting: a common equation you will see in papers is ‘rew = exp(-beta * dist)’, but when you have multiple keypoints, i.e. dist has shape (B, N) where B is number of environments and N is number of keypoints, you want to do ‘rew = torch.mean(exp(-beta * dist))’. Because taking the average first, i.e. rew = exp(-beta * dist.mean())) - will make the reward much sparser and harder to achieve.
4
1
21
2,913
The task environment in the video is built in Genesis, with state-based policy input (object pose, robot joint states). The full reward is a weighted sum of keypoint distance reward and another term that rewards higher object height (hence the lifting up motion), plus an action penalty term.
8
2,324
I’m a big fan of this line of work from Columbia (also check out PhysTwin by @jiang_hanxiao: jianghanxiao.github.io/physt…) they really make real2sim work for the very challenging deformable objects, And show it’s useful for real robot manipulation. So far it seems a bit limited to object-centric representations, looking forward to this being extended to simulate&render bigger scenes.
Can we learn a 3D world model that predicts object dynamics directly from videos? Introducing Particle-Grid Neural Dynamics: a learning-based simulator for deformable objects that trains from real-world videos. Website: kywind.github.io/pgnd ArXiv: arxiv.org/abs/2506.15680 Code: github.com/kywind/pgnd Demo: huggingface.co/spaces/kaifz/… To appear at #RSS2025
2
11
58
7,965
Normally, changing robot policy behavior means changing its weights or relying on a goal-conditioned policy. What if there was another way? Check out DynaGuide, a novel policy steering approach that works on any pretrained diffusion policy. dynaguide.github.io/ 🧵
5
32
3
144
19,154
Our lab at Stanford usually do research in AI & robotics, but very occasionally we indulge in being functional alcoholics -- Recently we hosted a lab cocktail night, and created drinks with research-related puns like 'reviewer#2' and 'make 6 figures', sharing the full recipes here in case you are looking for a fun summer drink:
9
10
2
267
27,200
The recipes are twists from classic drinks like Palomas and Martinis. The names are inspired by the drink tastes: 'reviewer#2' is spicy with jalapeños, 'in-between-projects' because it's chill and relaxing. shout out to my co-host @du_maximilian!
1
1
22
4,511
just realized twitter cropped the images :( here's the full list:
2
10
2,832