PhD student @Harvard | Prev. @Tsinghua_IIIS (Yao Class) | Robot Learning, Generative AI
Cambridge, MA
Joined September 2024
- Tweets121
- Following711
- Followers350
- Likes1.9K
Runhan Huang retweeted
How should we approach research, when the very field we work on is going through rapid and potentially fundamental changes?
I recently gave a talk at the Stanford Vision Seminar, reflecting on my PhD journey and how AI is changing the way I think about research. I adapted part of it into a new blog post, organized around three questions:
1. What should we work on?
2. How should we work with AI?
3. What is a PhD for?
I hope some of my reflections are helpful. The talk ended with an open discussion, and I’d love to continue it here; curious how others are thinking about these questions.
Personally -- as much as there's FOMO, I also think life is very long, and we have time.
toruowo.github.io/blog/posts…
Runhan Huang retweeted
We introduce action learning: learning a stochastic action that scores how plausible an entire trajectory is, directly from observed trajectories.
No known forces. No potential energy function. No simulator likelihood.
We think this is a new paradigm for reasoning about trajectory data.
Joint work with @du_yilun and @YuanqiD.
🔥 Recovers underlying physical dynamics with over 2,000× lower error than local regression in a double well system.
🪐 Identifies a hidden 0.01% perturbation in the Sun's mass from trajectories of a 31 body orbital system.
🧬 Reuses a single learned molecular action for trajectory completion, steering, constrained optimization and temperature transfer.
The core idea: One learned action → reuse for many possible downstream tasks!
Runhan Huang retweeted
LeWM predicts action effects, but searches 9,000 sequences to act.
Introducing INTACT: an end-to-end unified JEPA that completes the other half of LeWM: intent to action.
Only 1 epoch. 0 search.
95.33% Direct macro SR.
2.9-5.5 ms inference.
~300× lower planning latency.
Runhan Huang retweeted
After seeing Astra's strong reasoning capabilities in robotics, I'm more convinced that we shall not bake "language following / reasoning" into low-level action models, i.e., VLAs.
Robotics has spent years scaling demonstration data and model size. What happens when we scale EXPLORATION for robot agents?
We're incredibly excited to share RAX, an agentic robotics system that learns by scaling online exploration.
A coding agent connects directly to the robot’s perception and control APIs, learns from feedback, and builds programs that perceive, plan, and act. The model stays frozen. As exploration scales, the agent improves its programs and builds up reusable skills.
The results:
• ZERO teleoperation or action demonstrations on real robots—from insertion with 0.1 mm clearance and air-fryer manipulation to deformable objects and long-horizon mobile whole-body control.
• +37 percentage points on average over leading learned policies across five simulation benchmarks: RoboDojo, BEHAVIOR-1K, HumanoidArena, DexJoCo’s bimanual tasks, and LIBERO-Pro.
• One framework spanning single-arm, bimanual, dexterous, humanoid, and mobile robots, with new embodiments connected through APIs.
The craziest part: on each real-world embodiment, going from scratch to mastering the tasks took roughly two hours. No teleop. No model finetuning.
We’re open-sourcing the full framework AND every experiment session. We can’t wait to see what people build with it!
Project + sessions: robot-agent-exploration.com/
Paper: assets.robot-agent-explorati…
GitHub: N
Project fully led by @RunhanH and @xczhangxx who are absolutely awesome.
How it works, experiments, and robot videos below 🧵
(1/N)
Runhan Huang retweeted
Common belief: Robot policies need continuous action heads; discrete actions are 𝐥𝐞𝐬𝐬 𝐧𝐚𝐭𝐢𝐯𝐞 𝐚𝐧𝐝 𝐥𝐞𝐬𝐬 𝐩𝐫𝐞𝐜𝐢𝐬𝐞.
❗We find policies w/ the simplest discrete representations 𝐦𝐚𝐭𝐜𝐡 𝐨𝐫 𝐛𝐞𝐚𝐭 continuous ones—in sim & real, in single-step & diffusion/flow settings.
Here's why 🧵
Runhan Huang retweeted
🚀 Where does intelligence reside?
The paradigm in modern AI suggests: scale up a single model, train it on more data, and general intelligence may emerge. Yet Society of Mind proposed another possibility: intelligence may emerge from interactions of less intelligent agents.
As we keep scaling up a few generalist agents shared by everyone, can we scale up societies of decentralized agents that work together and collectively achieve more than any one of them could alone?
We introduce MPT: Market-based Post-Training, a framework to train such societies of agents by jointly learning agent policies and self-orchestration in a decentralized manner.
Runhan Huang retweeted
Research index (e.g. Google Scholar) is one of the largest collaborative systems where millions of human users build on each other's work across centuries.
How about we build something similar for async LLM agents?
We give agents a minimal hypertextual system. They asynchronously search for and read related work, cite the work that they find useful, and publish (mini-)papers.
Each search returns only the top k papers, ranked by relevance and a PageRank-like citation weight, which bounds the read cost of each agent to O(k), independent of agent count.
We show that performance effectively scales as we keep adding agents, and the system is robust to absence/failures of individual agents.
Runhan Huang retweeted
We have open-sourced our code, harness, skill library, and full real robot logs containing Claude sessions!
- Github harness code: github.com/robot-agent-explo…
- Github skill library where anyone can reuse and contribute their own robot skills on their own platforms! github.com/robot-agent-explo…
- Huggingface dataset with full robot logs and sessions! huggingface.co/datasets/robo…
Try them out on your own robot and contribute to the whole community!
Robotics has spent years scaling demonstration data and model size. What happens when we scale EXPLORATION for robot agents?
We're incredibly excited to share RAX, an agentic robotics system that learns by scaling online exploration.
A coding agent connects directly to the robot’s perception and control APIs, learns from feedback, and builds programs that perceive, plan, and act. The model stays frozen. As exploration scales, the agent improves its programs and builds up reusable skills.
The results:
• ZERO teleoperation or action demonstrations on real robots—from insertion with 0.1 mm clearance and air-fryer manipulation to deformable objects and long-horizon mobile whole-body control.
• +37 percentage points on average over leading learned policies across five simulation benchmarks: RoboDojo, BEHAVIOR-1K, HumanoidArena, DexJoCo’s bimanual tasks, and LIBERO-Pro.
• One framework spanning single-arm, bimanual, dexterous, humanoid, and mobile robots, with new embodiments connected through APIs.
The craziest part: on each real-world embodiment, going from scratch to mastering the tasks took roughly two hours. No teleop. No model finetuning.
We’re open-sourcing the full framework AND every experiment session. We can’t wait to see what people build with it!
Project + sessions: robot-agent-exploration.com/
Paper: assets.robot-agent-explorati…
GitHub: N
Project fully led by @RunhanH and @xczhangxx who are absolutely awesome.
How it works, experiments, and robot videos below 🧵
(1/N)
Runhan Huang retweeted
Today, we introduce Priming Models, a new paradigm for training one-step generative models from scratch.
No diffusion teacher. No flow map. No CFG. No JVP.
We believe this work will challenge how we think about building one-step models.
Joint work with @sitanch, @du_yilun, @parkwbin03, Bohyung Han.
🔥 1.19 FID one-step on ImageNet 256×256 while using substantially less compute, e.g., 12X less than Drifting-B, 6X less than Drifting-L
🎨 Scales to text-to-image generation (GenEval 0.82)
🤖 World modeling on RT-1 / Bridge-V2
Priming Model is built on one core message: [Reconstruction pretraining --> Distributional post-training].
Runhan Huang retweeted
We have open-sourced our code, harness, skill library, and full real robot logs containing Claude sessions!
- Github harness code: github.com/robot-agent-explo…
- Github skill library where anyone can reuse and contribute their own robot skills on their own platforms! github.com/robot-agent-explo…
- Huggingface dataset with full robot logs and sessions! huggingface.co/datasets/robo…
Runhan Huang retweeted
Can humanoids adapt to new terrain using only flat-ground references? 🤖
Meet P-BFM! 🥳
We learn a terrain-agnostic intent space with unsupervised RL and forward-backward representations, paired with a terrain-aware policy that adapts across diverse terrains—without terrain-specific demonstrations.
📄🎬💻 Check out our paper, video, and code here!
🌐 roboparty.github.io/P-BFM/
Runhan Huang retweeted
Scaling, period.
Robotics has spent years scaling demonstration data and model size. What happens when we scale EXPLORATION for robot agents?
We're incredibly excited to share RAX, an agentic robotics system that learns by scaling online exploration.
A coding agent connects directly to the robot’s perception and control APIs, learns from feedback, and builds programs that perceive, plan, and act. The model stays frozen. As exploration scales, the agent improves its programs and builds up reusable skills.
The results:
• ZERO teleoperation or action demonstrations on real robots—from insertion with 0.1 mm clearance and air-fryer manipulation to deformable objects and long-horizon mobile whole-body control.
• +37 percentage points on average over leading learned policies across five simulation benchmarks: RoboDojo, BEHAVIOR-1K, HumanoidArena, DexJoCo’s bimanual tasks, and LIBERO-Pro.
• One framework spanning single-arm, bimanual, dexterous, humanoid, and mobile robots, with new embodiments connected through APIs.
The craziest part: on each real-world embodiment, going from scratch to mastering the tasks took roughly two hours. No teleop. No model finetuning.
We’re open-sourcing the full framework AND every experiment session. We can’t wait to see what people build with it!
Project + sessions: robot-agent-exploration.com/
Paper: assets.robot-agent-explorati…
GitHub: N
Project fully led by @RunhanH and @xczhangxx who are absolutely awesome.
How it works, experiments, and robot videos below 🧵
(1/N)
Runhan Huang retweeted
Robotics has spent years scaling demonstration data and model size. What happens when we scale EXPLORATION for robot agents?
We're incredibly excited to share RAX, an agentic robotics system that learns by scaling online exploration.
A coding agent connects directly to the robot’s perception and control APIs, learns from feedback, and builds programs that perceive, plan, and act. The model stays frozen. As exploration scales, the agent improves its programs and builds up reusable skills.
The results:
• ZERO teleoperation or action demonstrations on real robots—from insertion with 0.1 mm clearance and air-fryer manipulation to deformable objects and long-horizon mobile whole-body control.
• +37 percentage points on average over leading learned policies across five simulation benchmarks: RoboDojo, BEHAVIOR-1K, HumanoidArena, DexJoCo’s bimanual tasks, and LIBERO-Pro.
• One framework spanning single-arm, bimanual, dexterous, humanoid, and mobile robots, with new embodiments connected through APIs.
The craziest part: on each real-world embodiment, going from scratch to mastering the tasks took roughly two hours. No teleop. No model finetuning.
We’re open-sourcing the full framework AND every experiment session. We can’t wait to see what people build with it!
Project + sessions: robot-agent-exploration.com/
Paper: assets.robot-agent-explorati…
GitHub: N
Project fully led by @RunhanH and @xczhangxx who are absolutely awesome.
How it works, experiments, and robot videos below 🧵
(1/N)
Runhan Huang retweeted
Two years ago, we introduced Video Prediction Policy (VPP), one of the first video-based generalist robot policies.
Today, we’re introducing VPP2: predict better, act better
Project Page: robert-gyj.github.io/video-p…
Code: github.com/roboterax/video-p…
A thread🧵 on how and why we do it.
Runhan Huang retweeted
(1/5) Everyone says dexterous manipulation needs real-world data. These policies were trained with none. Our robots mesh gears, thread nuts, and insert pegs zero-shot from RGB, and the same recipe works for locomotion across terrains much harder than before.
The missing piece wasn't a better RL algorithm, but a small tweak to what the robot practices.
Today we're releasing Success-Guided Sampling (SGS), a simple change to the task sampler that unlocks new levels of dexterity and agility with large-scale, end-to-end sim2real RL.
No demonstrations, no per-task reward tuning. Just PPO at scale.
Check out the policies yourself at sgs-rl.github.io! All videos at 1x speed! 🧵 🔈
Runhan Huang retweeted
SFT is not dead! 🥳
We found a way to make SFT rival current prevailing posttraining methods, often generalizing better and forgetting less than RL and OPSD. 🤯
Following our prior work on reasoning with sampling, we now introduce sampling to the posttraining stack.
1/n
Runhan Huang retweeted
NeurIPS decisions brought joy to some and disappointment to others. My first NeurIPS paper was in 2020, but given how much AI and robotics have changed, it feels like a lifetime ago.
I’ve shared a few thoughts with my students on the role of publications and so-called “A+B” research, and I’d like to share them more broadly here.
Runhan Huang retweeted
wow!! GPT-6 Astra can rapidly adapt to a new robot embodiment and a new task 🤯🤖
We asked it to put a cake into an air fryer in the benchmark we’re building, using simple tool APIs including CuRobo-based motion planning.
Its first attempts were chaotic. After repeated interaction, it accumulated environment-specific experience and learned to reliably complete the task end to end. 🍰➡️🔥
This is generalization far beyond VLAs and WAMs.
Runhan Huang retweeted
Remember the car task in the DARPA Robotics Challenge?
I asked Astra to tackle getting a humanoid into a Tesla—in Mujoco.
Two videos: learning through my feedback, then reusing that experience for direct execution using onboard egocentric perception and in-context adaptation.
Runhan Huang retweeted
We put GPT-6 Astra in the RoboDojo. 🥋🤖
The RoboDojo Team conducted a comprehensive evaluation of GPT-6 Astra as an embodied agent, including:
• RoboDojo Sim & Real, compared with GPT-5.5 and DeepSeek-Flash
• Humanoid high-level control
• Dexterous piano playing with RoboPianist 🎹
• A systematic study of in-context learning (ICL)
Our key takeaway:
GPT-6 Astra demonstrates remarkably strong semantic and spatial understanding, together with impressive in-context adaptation.
At the same time, physical commonsense remains a clear bottleneck — revealing an important gap between understanding the world and truly reasoning about its physics.
Full report & demos:
robodojo-benchmark.com/repor…
@_wenbozhang (project lead), @wenhaocha1, @frankzydou, @JinWeiyang18434, @YutaoOuyang, @minifullcapsule, @x_h_ucb, @YutaoOuyang, @YueChen614
