Pinned Tweet
Can a robot acquire real-world dexterous manipulation skills from just human videos?
Meet Video2Sim2Real: full-stack autonomous dexterous skill acquisition from a single RGB-D human manipulation video — without robot data or expert intervention.
Project: video2sim2real.github.io/
Paper: arxiv.org/abs/2606.08828
Great efforts! And feel so good for the continued collaboration with Kelin!
Our paper OmniTacTune got accepted to #CoRL2026 !!! Thanks again to our great team! @HaodeZ47056 @HanYunhai @h_ravichandar @RuohanGao1
See you in Austin!
1
4
526
Great work!!!!
Want to scale up realistic and dynamic vision-based tactile data from a single reference tactile image for robot learning with initial contact? Check out our paper, accepted by #CoRL2026
The code is open-sourced with the dataset and checkpoints:
🌐 Website: dongyuluo.github.io/controlt…
📖 Paper: arxiv.org/abs/2505.20498
💻 Code: github.com/DongyuLUO/Control…
More details about the story👇
2
319
After GPT6 helps you create the simulation, how can you turn it real world skills?
See our works below for insights!
Can a robot acquire real-world dexterous manipulation skills from just human videos?
Meet Video2Sim2Real: full-stack autonomous dexterous skill acquisition from a single RGB-D human manipulation video — without robot data or expert intervention.
Project: video2sim2real.github.io/
Paper: arxiv.org/abs/2606.08828
Yunhai Han retweeted
show the robot 1 human video. it builds a simulator, practices the skill there, then tries it on the real robot.
that’s basically the pipeline behind “[7 Jun 2026] Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video.”
the system reconstructs a digital twin from the video, extracts how the hand and objects move, then uses the simulator to fix the parts that don’t transfer cleanly across embodiments.
after that, imitation learning handles noisy real-world perception while residual RL adjusts the finger-level interaction.
they show skills like preparing fruit, seasoning food, arranging toys, passing books, and retrieving trays, all starting from a single human demonstration and no robot demos.
human video → simulator → robot practice → real-world skill is a pretty compelling learning loop.
video2sim2real.github.io/
Yunhai Han retweeted
Thanks so much for the invite! Really enjoyed meeting everyone in the reading club 🤖
🤖 @saturdayrobotic Robotics World Model Reading Club #25
🎤 @ColinYu14116982 (@UofMaryland, @amazon): Contact-Rich Robot Learning from Human Videos and Tactile.
🎤 @LinfengZhaoZLF (@Stanford): Retriever: Closing the Perception-Reasoning-Action Loop for Long-Horizon Robot Manipulation.
🦾 @Scav (@MissionRobotBay)
Contact-Rich Robot Learning: From Human Video → Object Flow → Tactile Residuals → Temporal Robot Agents
Bottleneck: embodiment and contact—not raw model scale—still dominate robot learning.
1️⃣ Human Video: Transfer the Object, Not the Human
Human video is our largest manipulation dataset. The transferable signal is object motion + interaction structure, not human morphology.
Object-centric flow captures embodiment-agnostic translation/rotation/trajectory.
Interaction-centric representations capture contact point, relative motion, and interaction state (approach → contact → manipulation).
GenFlowRL replaces expensive video-prediction rewards with generated object-centric δ-flow: dense flow matching + sparse stage rewards, while conditioning the policy itself.
Language-conditioned dynamics tuning + motion/semantic filtering matter; raw optical flow is too noisy.
Bounding box → keypoints → CoTracker → motion filtering enables scalable cross-embodiment datasets.
HumanEgo shows zero-shot human→robot transfer: ~92.5% average success across 4 tasks with ~30 min human video/task, without robot demos.
🔥 Hot take: the embodiment gap is not primarily visual—it is morphology + contact dynamics. Match the object and interaction, not the human body.
Video generation remains useful for imagination/planning, but compressed object flow is a better closed-loop reward/policy interface.
2️⃣ Tactile: Freeze Vision, Learn the Contact Residual
Vision gives a strong motion prior; touch supplies what vision cannot: force, shear, slip, compliance, and local geometry.
MimicTouch: human tactile demonstrations are dramatically more efficient than teleop (104 vs. 19 traj/hr for SpaceMouse; 83.3% vs. 38.5% success). Multi-modal proprioception + GelSight + audio → representation learning → imitation → real-world SAC residual RL.
OmniTacTune generalizes this idea: attach a policy-agnostic, contact-gated tactile residual to any frozen visual policy—Human-flow, Diffusion Policy, ACT, π₀.₅, etc. Dense rewards combine object flow + tactile grasp/stability + slip/safety signals.
Reported Peg-in-Hole gains: Human-flow 40→100%, Diffusion 30→90%, ACT 25→80%, π₀.₅ 15→75%, within ~40–80 min real interaction.
🔥 Hot take: don't retrain the whole robot policy to learn touch. Freeze the visual prior; learn only the residual corrections that vision cannot see.
3️⃣ ControlTac: Scale Touch Through Physical Latents
Tactile pixels are sensor-specific. The scalable abstraction should be force + contact area + pose + slip.
ControlTac generates realistic tactile images from one reference conditioned on ΔF + contact pose, enabling trajectory augmentation and improving tracking, weighing and imitation. The next step, RoboTac, extends this from static images to action-conditioned tactile dynamics.
🔥 Hot takes:
One real image + physical controls can generate useful tactile distributions.
Marker-free generation + marker recomposition enables practical augmentation.
Contact-gated residuals are more stable than letting touch fight vision everywhere.
Dense flow + grasp + slip rewards beat sparse success signals for contact-rich RL.
Sensor-specific pixels are a historical accident; shared physical features are the scalable latent.
Better perception/data alone cannot close the final hardware/compliance gap. Humans dynamically change stiffness; rigid robots cannot.
🎯 Long-term goal: a tactile world model that imagines contact under candidate actions and schedules compliance before contact—not merely reacts after impact.
4️⃣ Retriever: Robotics Needs a “PyTorch” for Closed-Loop Agents
Long-horizon robotics has another missing layer: the harness connecting System 2 reasoning/VLMs, System 1 skill VLAs, memory and high-rate control.
Retriever proposes Flow + Clock + Sync + Runtime:
Temporal compositionality
Asynchronous multi-rate execution
First-class closed loops
Explicit synchronization
Deterministic replay
Hardware/backend abstraction
30 Hz perception → 2 Hz skill VLA → 200 Hz controller can run concurrently without reasoning blocking control. Belief, replanning and skill-progress triggers close the loop.
🔥 Hot take: robotics doesn't just need another foundation model. It needs a programming model with explicit temporal semantics—one where agents can be composed, debugged, replayed, ported and eventually trained at scale.
The Bigger Bet
The trajectory is surprisingly coherent:
Human video learns WHAT should happen → object/interaction flow bridges embodiments → tactile residuals learn HOW contact feels → tactile world models anticipate contact → temporal runtimes turn these modules into scalable closed-loop agents.
The next frontier may therefore be less about bigger policies and more about better abstractions: object motion, interaction state, physical contact variables, and time itself.
Yunhai Han retweeted
Can robots learn to use tools as skillfully and flexibly as human? 🤖🛠️
Join us at BiDexTool @ #CoRL2026!
📄 Papers
🛠️ Demos & position abstracts
💬 Panel questions
Full details: bimanual-tool.github.io
@HanYunhai @HaonanChen_ @JiahengHu1 @YuxiangMa4 @ToruO_O @ChengXianyi @jiajunwu_cs @NimaFazeli7
Well deserved!
OmniTacTune won the Best Paper Award at the Tactile Sensing for Robotic Foundation Models Workshop at RSS 2026! (tac-for-fm.github.io/rss2026…)
Huge thanks to the coauthors @HaodeZ47056 and @HanYunhai, and our advisors @h_ravichandar and @RuohanGao1.
3
301
Yunhai Han retweeted
1/ 👀Vision tells robots where to go. 👋Touch tells robots about the interactions.
2/ Visual policies from teleoperated robot demos and human videos are scaling fast — but they don't have paired tactile data, so they still fail at the last millimeter of contact-rich manipulation.
❓So here is the question:
How can we adapt tactile feedback into pretrained visual policies?
Humans solve this naturally.
1️⃣ Learn from demonstrations through vision: we understand the task structure and the motion priors.
2️⃣ Then practice with touch: we interact with the world, feel what happens, and refine the motion.
🚀 We introduce OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policies ⬇️
✦ Adapting tactile feedback into existing visual policies
✦ No offline tactile demonstrations
✦ Real-world RL in 40–80 minutes, 5–40% → 85–100%
✦ Works across human flow policies, ACT, DP, π0.5
✦ Works across different tactile representations
🌐 Website: colinyu1.github.io/omnitactu…
📖 Paper: arxiv.org/abs/2607.03723
📷 Video: youtube.com/watch?v=anRD7lhh…
In this demo, we show four challenging contact-rich manipulation tasks and a 40-mins, one-take recording of an online RL training demo.
1/n
Yunhai Han retweeted
Real robot data is expensive. Real robot evaluations are slow.
Excited to share SimFoundry - a system that turns real scenes into sim-ready worlds for training and benchmarking robots at scale -
✅Automated Scene Reconstruction with asset generation
✅Handles clutter, articulated objects, multiple robot embodiments
✅High Correlation Real-to-Sim Evals
✅Zero-shot Sim-to-Real
✅Generates diverse digital cousins
Less manual environment authoring, more scalable feedback for robot learning.
🌐research.nvidia.com/labs/gea…
🧵1/9
Yunhai Han retweeted
Check out our paper on learning forceful manipulation by separating task planning from interaction control!
People are talking about building world models for AI systems. But for world models to be truly useful for robots, they need to model the changing dynamics of the physical world, such as gravity, friction, and external disturbances.
Introducing IMPACT: Internal Model Predictive ConTrol. Inspired by the mechanism in human cerebellum, we propose learning an internal model of environmental dynamics on the fly and adapting to changing dynamics accordingly. This internal model runs at 1000 Hz on real robot hardware!! Controlled experiments in both simulation benchmarks and the real world demonstrate that IMPACT significantly outperforms baseline methods.
Yunhai Han retweeted
Beyond EgoEngine and Video2Sim2Real: Turning Human Videos into Robot Experience: medium.com/@liuyangcen112358…
We evaluate on 7 everyday dexterous tasks, including fruit placement, steak seasoning, toy rearrangement, tissue handover, book passing, and tray retrieval.
In real-world trials with object-pose variations, Video2Sim2Real achieves a 95.7% success rate, significantly outperforming both RL-only and IL-only sim-to-real methods.
More task videos and results are available on our project website.
1
6
413
Takeaways:
- Retargeting a robot trajectory from a human video alone is insufficient; effective execution requires object-centric refinement that leverages hand-object interaction cues extracted from the video.
- Such refinement does not need to be applied continuously over the entire trajectory; instead, it can focus on a small set of key manipulation frames that capture the most critical interaction effects.
- Since this refinement is performed in a digital-twin simulator, reliable real-world deployment also requires effective sim-to-real transfer. To this end, we introduce a decoupled transfer strategy in which global IL adapts to geometric variations, while residual RL handles contact and physics discrepancies.
Big thanks to the team! Yunhai Han, Jianuo Qiu @JianuoQiu, Linhao Bai @lbai46, Ziyu Xiao, Zihang Zeng, Yangcen Liu @Randle_Liu, Zhaodong Yang, Shalin Jain, Wenrui Ma, Jiaqi Fu @JiaqiFu17693, Yuqian Zheng, Manisha Natarajan, Muhammad Zubair Irshad @mzubairirshad, Kenneth Shaw @kenny__shaw, Matthew Gombolay @MatthewGombolay, Zsolt Kira @zsoltkira, and Harish Ravichandar @h_ravichandar.
10
409
Yunhai Han retweeted
Project website: sites.google.com/view/immimi…
Arxiv Link: arxiv.org/abs/2509.10952
Big thanks to: @wshin1726 @HanYunhai @DanielZhenyang @h_ravichandar @danfei_xu
Yunhai Han retweeted
What if one unified method helps robots learn from human videos across many tasks, many robots?
Meet ImMimic: Cross-Domain Imitation from Human Videos via Mapping and Interpolation (CoRL 2025 Oral Presentation🏆) @ICatGT
Check it here sites.google.com/view/immimi…!
















