Rui Yan (@RuiJerryYan) header
Rui Yan (@RuiJerryYan) profile picture

PhD @GeorgiaTech | Robotics

San Diego, CA
Joined December 2024
Photo by Rui Yan (@RuiJerryYan)Photo by Rui Yan (@RuiJerryYan)Photo by Rui Yan (@RuiJerryYan)Photo by Rui Yan (@RuiJerryYan)Photo by Rui Yan (@RuiJerryYan)Photo by Rui Yan (@RuiJerryYan)Photo by Rui Yan (@RuiJerryYan)Photo by Rui Yan (@RuiJerryYan)Photo by Rui Yan (@RuiJerryYan)Photo by Rui Yan (@RuiJerryYan)
Introducing Physical Agent 🥚🦾 We post-trained an open 4B general-purpose agent to control a robot, on four GPUs. Weights are open. It acts by writing tool calls, not an action head. So it is still a general agent, and it learns new robot tasks from one demo in context. 🧵
13
17
4
157
17,106
One 4B model, four B200 GPUs, under 50 GPU-hours. Simulation only for now. Next: real robots and large-scale RL. We are short on GPUs and would love to work with US labs. Page: physical-agent-project.githu… Weights: huggingface.co/GatechLIDAR/P… With @XuJunxiaoxn and Ye Zhao, Georgia Tech
1
6
1,643
The demonstrations contain only successes. In natural rollouts the agent still catches some of its own failed grasps: it writes that it cannot confirm the object is held, opens the gripper, and grasps again.
1
488
The key result is in-context learning. Put one robot demonstration in the prompt, with no parameter update: • 44.8% → 56.2% on 21 perturbed tasks • 17.1% → 31.4% on the hard ones Given the same demo, the VLA baseline falls to 0%. We never trained for this. It is inherited.
1
571
It matches a VLA on robot tasks and is still a general agent. LIBERO: 79.2% vs 80.0% for a VLA baseline from the same model and data. Under task perturbation: 23.3% vs 15.0%. MMLU-Pro 56.8% (61.8% before), MMMU 53.7% (46.8% before). The baseline scores 0 on both.
1
1
758
What is a physical agent? A general-purpose agent that sees with its own vision, reasons in language, and acts through physical tool calls: native tool calls whose arguments are end-effector targets. Like a VLA, it learns from robot data. Unlike a VLA, the action stays native.
1
950
Find it. Reach it. Grasp it. Easy for us, hard for a humanoid. Introducing HuG (#CoRL2026): Learning Humanoid Whole-Body Affordance-Aware Grasping with Active Perception. The robot looks around for a half-hidden object, squats or stretches its whole body to reach it, and grasps. Zero real demos, zero-shot to unseen objects and everyday scenes, from low tables to high shelves. 🌐 xuanbinpeng.com/hug/
13
32
18
149
15,049
Excited to share OpenRoboKit, our series of open-source robotics projects! 🤖 - RoboKit for fast, scalable optimization & simulation github.com/openrobokit/robok… Our RoboKit paper has been accepted for a spotlight at #CORL2026 - RetargetingKit for motion transfer & teleoperation github.com/openrobokit/retar… - GraspKit for end-to-end grasping github.com/openrobokit/grasp… Website: openrobokit.com Check the video (sound on 🎵) for a 16-hour grasping time-lapse in beautiful San Diego, plus a quick look at dexterous retargeting and robot simulation.
18
56
22
297
43,204
We introduce RoboSkills: a minimal, low-cost skillset for agentic robotics 🤖 It supports diverse robotics workflows, such as learning cat’s cradle 🧶 from a human video (Real2Sim) and transferring multi-person motion to characters and humanoids 💃 (Motion Generation). We also support Auto Research, Agent as Policy, and Agentic Simulation. With suitable tools and workflows, we enable cheaper models to solve complex tasks. GitHub: github.com/openrobokit/robos… Website: openrobokit.com/roboskills Watch the video for more examples.
14
41
14
298
38,701
Excited to share RoboSkills 🤖! A minimal, low-cost skillset for Real2Sim, Agent-as-Policy, Agentic Simulation, and much more. Its lightweight design enables lower overhead, faster iteration, and greater flexibility in agentic robotics development. 🚀
We introduce RoboSkills: a minimal, low-cost skillset for agentic robotics 🤖 It supports diverse robotics workflows, such as learning cat’s cradle 🧶 from a human video (Real2Sim) and transferring multi-person motion to characters and humanoids 💃 (Motion Generation). We also support Auto Research, Agent as Policy, and Agentic Simulation. With suitable tools and workflows, we enable cheaper models to solve complex tasks. GitHub: github.com/openrobokit/robos… Website: openrobokit.com/roboskills Watch the video for more examples.
5
32
2,811
Can we scale robot learning without scaling human labor? 📈 Demonstrations, resets, failure recovery, constant supervision—collecting robot data still takes substantial human effort. 🔑 Three keys let agents take on more of the work: 1. Real2Sim2Real: Agents explore and learn in digital twins, where trial and error is safer and cheaper. 2. Autonomous Recovery: Robots learn to recover from unexpected failures instead of waiting for human resets. 3. Human-on-Demand Learning: Multiple robots run in parallel, and humans step in only when autonomous recovery falls short. 🥑 Recova brings these together to get more robot learning out of every hour of human effort. research.nvidia.com/labs/gea… Joint work with @anjjei, Johan Bjorck, @ZhidingYu, @yin_hongxu, @jankautz, @DrJimFan, @yukez, and @Sifei30488L. 👇 Here's how it works 🧵
5
31
11
159
17,962
No train. No teleop. No pre-built map. ASENA keeps self-evolving ↺ with every task. With ASENA, we put coding agents in the engineer's seat: they break down tasks, write and execute robot programs, diagnose failures, and turn experience into reusable skills. The robot doesn't just complete tasks. It builds new capabilities for the next one.
Our G1 robot found its way to the pantry, picked up a Pringles can, and handed it over. No pre-built map. No training for this task. Behind it is a coding agent, writing, running, and fixing the robot's programs as it goes. And it self-evolves: what works is saved as code for the next task, no retraining needed. This is ASENA. Joint work with @Isabella__Liu, @edmund_bu, Johan Bjorck, @yin_hongxu, @zhengyiluo, @jankautz, @DrJimFan, @yukez, and @Sifei30488L. 🧵👇
1
2
33
1,700
Rui Yan retweeted
Our G1 robot found its way to the pantry, picked up a Pringles can, and handed it over. No pre-built map. No training for this task. Behind it is a coding agent, writing, running, and fixing the robot's programs as it goes. And it self-evolves: what works is saved as code for the next task, no retraining needed. This is ASENA. Joint work with @Isabella__Liu, @edmund_bu, Johan Bjorck, @yin_hongxu, @zhengyiluo, @jankautz, @DrJimFan, @yukez, and @Sifei30488L. 🧵👇
11
30
8
139
16,820
If you dont know Yutong, he is the guy always have the rubik's cube in his hands. A lot of work is needed to make this happen in real.
Meet FINGR: a dexterous hand that solves a 2×2 Rubik’s Cube with continuous finger tricks. Here’s how we got the fingers to work together. 🧵
1
4
1
63
13,260
Rui Yan retweeted
Human demonstrations are a scalable source of contact rich manipulation data, and they naturally record contact forces. But robots still can’t execute them directly. Introducing WT-UMI: Tactile-based Whole-Body Manipulation via Force-Supervised Contact-Aware Planning. Accepted to #CoRL2026 ! 🎉 wt-umi.github.io/WTUMI/
2
11
2
85
12,669
Rui Yan retweeted
Transformer Transformer will be presented at #CoRL2026 🚀 See y'all in Austin! Website: transformer-transformer.gith… Video: youtube.com/watch?v=TTyjvPVF… Paper: transformer-transformer.gith…
13
84
23
663
117,865
Children learn to walk by falling. Robots fail too—but we usually throw those failures away. Yet failed rollouts tell us something successful demonstrations cannot: what bad actions actually lead to. So, can robots learn from their failures too? Introducing FACT: Failure-Aware Causal Training for World-Action Models. 🔗website: fact-wam.github.io/ 📄arxiv: arxiv.org/abs/2608.10232 💻 code: github.com/Bariona/FACT
15
43
10
323
77,630
VLA/VAs are doing well on short skills like pick-and-place. But real tasks rarely stop after one action, they require 1) many interdependent steps, 2) progress tracking, and 3) recovery from mistakes. In our paper LoHo-Manip, we address long-horizon manipulation with trace-conditioned VLA planning: a task manager tracks what’s done, plans what remains, and guides execution with visual traces.
1
44
2
239
26,881
Cross-embodiment learning for diverse dexterous hands is an exciting direction. Looking forward to seeing how shared latent representations enable generalization across embodiments.
Ever want to have a single policy to control diverse robots as well as different dexterous hands, or to observe the emergent behavior under cross embodiment training? Introducing our #CVPR2026 paper XL-VLA, Cross-Hand Latent Representation for Vision-Language-Action Models.
1
3
382
Ever want to have a single policy to control diverse robots as well as different dexterous hands, or to observe the emergent behavior under cross embodiment training? Introducing our #CVPR2026 paper XL-VLA, Cross-Hand Latent Representation for Vision-Language-Action Models.
11
42
20
277
91,204