
PhD @GeorgiaTech | Robotics
San Diego, CA
Joined December 2024
- Tweets54
- Following129
- Followers173
- Likes167
Pinned Tweet
Introducing Physical Agent 🥚🦾
We post-trained an open 4B general-purpose agent to control a robot, on four GPUs. Weights are open.
It acts by writing tool calls, not an action head. So it is still a general agent, and it learns new robot tasks from one demo in context.
🧵
One 4B model, four B200 GPUs, under 50 GPU-hours.
Simulation only for now. Next: real robots and large-scale RL. We are short on GPUs and would love to work with US labs.
Page: physical-agent-project.githu…
Weights: huggingface.co/GatechLIDAR/P…
With @XuJunxiaoxn and Ye Zhao, Georgia Tech
The demonstrations contain only successes.
In natural rollouts the agent still catches some of its own failed grasps: it writes that it cannot confirm the object is held, opens the gripper, and grasps again.
1
488
The key result is in-context learning.
Put one robot demonstration in the prompt, with no parameter update:
• 44.8% → 56.2% on 21 perturbed tasks
• 17.1% → 31.4% on the hard ones
Given the same demo, the VLA baseline falls to 0%.
We never trained for this. It is inherited.
1
571
It matches a VLA on robot tasks and is still a general agent.
LIBERO: 79.2% vs 80.0% for a VLA baseline from the same model and data. Under task perturbation: 23.3% vs 15.0%.
MMLU-Pro 56.8% (61.8% before), MMMU 53.7% (46.8% before). The baseline scores 0 on both.
1
1
758
What is a physical agent?
A general-purpose agent that sees with its own vision, reasons in language, and acts through physical tool calls: native tool calls whose arguments are end-effector targets.
Like a VLA, it learns from robot data. Unlike a VLA, the action stays native.
1
950
Rui Yan retweeted
Find it. Reach it. Grasp it. Easy for us, hard for a humanoid.
Introducing HuG (#CoRL2026): Learning Humanoid Whole-Body Affordance-Aware Grasping with Active Perception.
The robot looks around for a half-hidden object, squats or stretches its whole body to reach it, and grasps. Zero real demos, zero-shot to unseen objects and everyday scenes, from low tables to high shelves.
🌐 xuanbinpeng.com/hug/
Rui Yan retweeted
Excited to share OpenRoboKit, our series of open-source robotics projects! 🤖
- RoboKit for fast, scalable optimization & simulation github.com/openrobokit/robok…
Our RoboKit paper has been accepted for a spotlight at #CORL2026
- RetargetingKit for motion transfer & teleoperation github.com/openrobokit/retar…
- GraspKit for end-to-end grasping github.com/openrobokit/grasp…
Website: openrobokit.com
Check the video (sound on 🎵) for a 16-hour grasping time-lapse in beautiful San Diego, plus a quick look at dexterous retargeting and robot simulation.
Rui Yan retweeted
We introduce RoboSkills: a minimal, low-cost skillset for agentic robotics 🤖
It supports diverse robotics workflows, such as learning cat’s cradle 🧶 from a human video (Real2Sim) and transferring multi-person motion to characters and humanoids 💃 (Motion Generation).
We also support Auto Research, Agent as Policy, and Agentic Simulation. With suitable tools and workflows, we enable cheaper models to solve complex tasks.
GitHub: github.com/openrobokit/robos…
Website: openrobokit.com/roboskills
Watch the video for more examples.
Rui Yan retweeted
Excited to share RoboSkills 🤖! A minimal, low-cost skillset for Real2Sim, Agent-as-Policy, Agentic Simulation, and much more.
Its lightweight design enables lower overhead, faster iteration, and greater flexibility in agentic robotics development. 🚀
We introduce RoboSkills: a minimal, low-cost skillset for agentic robotics 🤖
It supports diverse robotics workflows, such as learning cat’s cradle 🧶 from a human video (Real2Sim) and transferring multi-person motion to characters and humanoids 💃 (Motion Generation).
We also support Auto Research, Agent as Policy, and Agentic Simulation. With suitable tools and workflows, we enable cheaper models to solve complex tasks.
GitHub: github.com/openrobokit/robos…
Website: openrobokit.com/roboskills
Watch the video for more examples.
Rui Yan retweeted
Can we scale robot learning without scaling human labor? 📈
Demonstrations, resets, failure recovery, constant supervision—collecting robot data still takes substantial human effort.
🔑 Three keys let agents take on more of the work:
1. Real2Sim2Real: Agents explore and learn in digital twins, where trial and error is safer and cheaper.
2. Autonomous Recovery: Robots learn to recover from unexpected failures instead of waiting for human resets.
3. Human-on-Demand Learning: Multiple robots run in parallel, and humans step in only when autonomous recovery falls short.
🥑 Recova brings these together to get more robot learning out of every hour of human effort.
research.nvidia.com/labs/gea…
Joint work with @anjjei, Johan Bjorck, @ZhidingYu, @yin_hongxu, @jankautz, @DrJimFan, @yukez, and @Sifei30488L.
👇 Here's how it works 🧵
Rui Yan retweeted
No train. No teleop. No pre-built map. ASENA keeps self-evolving ↺ with every task.
With ASENA, we put coding agents in the engineer's seat: they break down tasks, write and execute robot programs, diagnose failures, and turn experience into reusable skills.
The robot doesn't just complete tasks. It builds new capabilities for the next one.
Our G1 robot found its way to the pantry, picked up a Pringles can, and handed it over. No pre-built map. No training for this task.
Behind it is a coding agent, writing, running, and fixing the robot's programs as it goes. And it self-evolves: what works is saved as code for the next task, no retraining needed. This is ASENA.
Joint work with @Isabella__Liu, @edmund_bu, Johan Bjorck, @yin_hongxu, @zhengyiluo, @jankautz, @DrJimFan, @yukez, and @Sifei30488L. 🧵👇
Rui Yan retweeted
Our G1 robot found its way to the pantry, picked up a Pringles can, and handed it over. No pre-built map. No training for this task.
Behind it is a coding agent, writing, running, and fixing the robot's programs as it goes. And it self-evolves: what works is saved as code for the next task, no retraining needed. This is ASENA.
Joint work with @Isabella__Liu, @edmund_bu, Johan Bjorck, @yin_hongxu, @zhengyiluo, @jankautz, @DrJimFan, @yukez, and @Sifei30488L. 🧵👇
Rui Yan retweeted
Human demonstrations are a scalable source of contact rich manipulation data, and they naturally record contact forces. But robots still can’t execute them directly.
Introducing WT-UMI: Tactile-based Whole-Body Manipulation via Force-Supervised Contact-Aware Planning. Accepted to #CoRL2026 ! 🎉
wt-umi.github.io/WTUMI/
Transformer Transformer will be presented at #CoRL2026 🚀 See y'all in Austin!
Website: transformer-transformer.gith…
Video: youtube.com/watch?v=TTyjvPVF…
Paper: transformer-transformer.gith…
Rui Yan retweeted
Children learn to walk by falling. Robots fail too—but we usually throw those failures away.
Yet failed rollouts tell us something successful demonstrations cannot: what bad actions actually lead to.
So, can robots learn from their failures too?
Introducing FACT: Failure-Aware Causal Training for World-Action Models.
🔗website: fact-wam.github.io/
📄arxiv: arxiv.org/abs/2608.10232
💻 code: github.com/Bariona/FACT
Rui Yan retweeted
VLA/VAs are doing well on short skills like pick-and-place. But real tasks rarely stop after one action, they require 1) many interdependent steps, 2) progress tracking, and 3) recovery from mistakes.
In our paper LoHo-Manip, we address long-horizon manipulation with trace-conditioned VLA planning: a task manager tracks what’s done, plans what remains, and guides execution with visual traces.
Cross-embodiment learning for diverse dexterous hands is an exciting direction.
Looking forward to seeing how shared latent representations enable generalization across embodiments.
Ever want to have a single policy to control diverse robots as well as different dexterous hands, or to observe the emergent behavior under cross embodiment training?
Introducing our #CVPR2026 paper XL-VLA, Cross-Hand Latent Representation for Vision-Language-Action Models.
1
3
382
Rui Yan retweeted
Ever want to have a single policy to control diverse robots as well as different dexterous hands, or to observe the emergent behavior under cross embodiment training?
Introducing our #CVPR2026 paper XL-VLA, Cross-Hand Latent Representation for Vision-Language-Action Models.




















