Visiting PhD student at @StanfordSVL. Working with Prof. @drfeifei. PhD student of NUS. Prev. Tsinghua Univ. Roboticist.
Stanford
Joined August 2021
- Tweets556
- Following1K
- Followers855
- Likes723
Pinned Tweet
๐คRobots need world models for visual planning, but they don't need to imagine every frame.
Introducing DreamCan: Sparse Visual Planning for Long-Horizon Robot Manipulation.
Project: dreamcan.github.io/
Code: github.com/HeegerGao/DreamCaโฆ
Paper: arxiv.org/abs/2610.24981
Chongkai Gao retweeted
Planners operate on task timescales. Feasibility changes on control timescales.
Bridging that gap takes a high-frequency mid-level layer. Introducing EPO: Energy-Based Policy Orchestration #CoRL2026
EPO gives each low-level policy an energy: how feasible a goal is, and how well the policy will reach it. Energies are differentiable and comparable across policies, so EPO optimizes the policyโgoal pair at control rate.
Unlike a learned MoE router, EPO is compositional: adding a skill means training one new energy, nothing else. It's also more precise and switches policies 7ร less often, pushing the precisionโrobustness frontier.
๐ Paper: arxiv.org/abs/2611.02546
๐ Project: yewr.github.io/epo
๐ป Code: github.com/YeWR/EPO
@wangyenjen @elvis_hsieh77 @frankzydou @ShiboZhaoSLAM @pabbeel @du_yilun @MIT_LISLab
Chongkai Gao retweeted
Alright, alright, alrightโฆ
CoRL registration is completely maxed out, with hundreds of people on the waiting list. Please stop flooding my inbox for tickets ๐
You can still join the fun! Our partners are hosting social events all over town. We gathered the list of public events here:
corl.org/program/socials
Many donโt require a conference badge to attend. Letโs turn Ausitn into a Robotics Party Town!
CoRL is now sold out. We have an incredible program lined up. Can't wait to see y'all in Austin next week!
corl.org/program
ALT Conference on Robot Learning 2026
Chongkai Gao retweeted
๐ฆพ We're generating dexterous robot demonstrations from human data at scale ๐ Check out our work at Meta Reality Labs Research
๐ฅ Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation
๐ Website: yy-gx.github.io/GNR/
๐ Report: arxiv.org/abs/2610.12440
Chongkai Gao retweeted
Does spatial understanding translate into reliable robot action?
Introducing BEHAVIOR-Spatial benchmark: Evaluating Spatial Understanding for Robotics.
Built on BEHAVIOR-1K, our benchmark connects three levels of spatial reasoning across shared scenes, objects, and relations:
๐ง BEHAVIOR-Spatial-VQA (Understand): answer spatial questions
๐ฏ BEHAVIOR-Spatial-IR (Ground): point to the instructed target
๐ค BEHAVIOR-Spatial-Action (Act): execute manipulation and navigation tasks
Across 3,256 questions, 1,240 grounding queries, and 24 robot tasks, we examine when spatial perception translates into robot behavior.
Our key test: keep the training scene fixed and change the instruction. Does the robot switch targets?
Our baseline commits to the wrong target in 70% of these probes. Explicit target cues help, but reliable instruction following remains challenging.
๐ฌ Overview below. Benchmark data, training demonstrations, evaluation tools, and findings in the thread. ๐งต
Chongkai Gao retweeted
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! ๐งต
Blog: rikiyat.github.io/blog/weighโฆ
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-spโฆ
github.com/RikiyaT/lm_undโฆ
huggingface.co/Qwen/Qwen3.5-0โฆ
huggingface.co/RikiyaT/Qwen3.โฆ
Chongkai Gao retweeted
Long-horizon, whole-body robot policies still fail during deployment. How do we best close this last-mile gap? ๐ค
๐จ Introducing I3L: Iterative Interactive Imitation Learning for Whole-Body Mobile Manipulation
Website: i-3l.github.io/
Paper: arxiv.org/abs/2610.17748
Chongkai Gao retweeted
GPT and Claude now operate real robot arms, and they generalize remarkably well. What they lack is speed and low-level control. We introduce FACT, the first cerebellum for robots: high-frequency tactile reasoning, learned from scaled human data with both motion priors and applied forces. Brains will keep getting smarter and keep getting replaced, but the cerebellum couples with whichever brain comes next.
1, Built like a brain and a cerebellum. A coarse planner (the brain) takes images, instructions, robot state, past tracks, and predicts future video, tracks and actions. A force-action refiner (the cerebellum) runs fast, jointly predicting fingertip forces and refines actions using new touch feedback.
2, We collect EgoForce, a human manipulation dataset pairing egocentric video and hand motion with calibrated tactile forces. It spans ~2,000 hours, with 40,000 episodes covering 1,000 tasks performed by 1,000 demonstrators.
3, We pretrain FACT, a multimodal robot System 1 policy, on EgoForce plus robot data, then fine-tune with just 10 robot demos + 90 human demos per task. Across 10 real-world tasks, including pipetting, tool use and playing instruments, FACT outperforms ฯ0.5 and T-Rex, with gains up to 60 points over T-Rex.
4, Human force supervision makes a substantial difference. Adding force labels to human pretraining unlocks tasks that were previously almost impossible, such as shoelace tying:
Test-tube cap opening: 0% โ 80%
Flip-cap opening: 10% โ 50%
Shoelace tying: 0% โ 20%
FACT connects human contact experience to robot execution-from force supervision during pretraining to predicting and responding to contact on the robot.
๐ Paper: arxiv.org/abs/2610.19155
๐ Website: fact-1-0.github.io/
Code and dataset coming soon!
๐งต Thread ๐
Chongkai Gao retweeted
Human videos bring scale and motion priors to robot pretraining. But video mainly captures human states, leaving intended actions implicit. Applied forces help fill that gap. Introducing FACT: learning dexterous manipulation from human video and force. ๐งต
Really cool demo!
(๐ต๐๐๐๐๐๐๐ข๐๐ ๐๐ข๐ ๐๐: ๐ ๐๐๐ก๐ก๐๐ ๐๐ข๐๐ก ๐๐๐๐ฆ๐๐ ๐๐ฆ ๐๐ ๐๐๐ ๐กโ๐ ๐๐๐๐๐ก ๐๐ก๐ ๐๐๐ ๐ถ ๐ธ๐๐๐๐ฆ ๐กโ๐ ๐ฃ๐๐๐๐ ๐ค๐๐กโ ๐ ๐๐ข๐๐ ๐๐! ๐)
Ever dreamed of a robot that could play the harp ๐ผ, tie your shoelaces ๐, or even perform delicate wet-lab experiments ๐งช?
The key is ๐ฅ๐๐๐ซ๐ง๐ข๐ง๐ ๐ง๐จ๐ญ ๐ฃ๐ฎ๐ฌ๐ญ ๐ก๐จ๐ฐ ๐ก๐ฎ๐ฆ๐๐ง๐ฌ ๐ฆ๐จ๐ฏ๐, ๐๐ฎ๐ญ ๐ก๐จ๐ฐ ๐ญ๐ก๐๐ฒ ๐๐๐๐ฅ ๐๐ง๐ ๐๐จ๐ง๐ญ๐ซ๐จ๐ฅ ๐๐จ๐ซ๐๐๐ฌ. We introduce ๐
๐๐๐, a ForceโActionโTrack foundation model that learns from 2,000 hours of force-rich human demonstrations in our EgoForce dataset, bringing us closer to robots with truly human-like dexterity! ๐ฉท๐๐
arxiv.org/abs/2610.19155
Bigggggg shoutout to @ZhiCao01756604 !
Chongkai Gao retweeted
๐ Actions that condition world models are highly heterogeneous: forces on objects, camera movements, hand motions, robot arms, each with its own meaning and structure. So today's action-conditioned world models are usually built one action type at a time, each with its own conditioning design.
We introduce UniAction, a unified action-conditioning framework: one world model driven by all of these actions, and even by mixes of them in a single generation. Trained jointly, it outperforms the action-specific experts. Great work led by amazing @BowenXue_ .
More detailed threads below๐
๐งต 1/10
Chongkai Gao retweeted
Thanks Yann for the comment! We also have an upcoming work that operates in latent state :) But I see them as tradeoffs that offer different capabilities in practice.
Regardless of representation (latent, 2D, 3D, etc), the goal for any "imagining" in planning is ultimately to evaluate candidate actions or policies, so the other half of the story is how imagined states can be scored. Image-based representation conveniently shares this interface with VLMs and other pretrained visual tools (like SAM). But they certainly have their downsides too compared to latent representation, like search efficiency (which is actually important for finding novel solutions!)
One piece of advice from Toru's talk that really stuck with me: be brave. It matters so much nowadays.
How should we approach research, when the very field we work on is going through rapid and potentially fundamental changes?
I recently gave a talk at the Stanford Vision Seminar, reflecting on my PhD journey and how AI is changing the way I think about research. I adapted part of it into a new blog post, organized around three questions:
1. What should we work on?
2. How should we work with AI?
3. What is a PhD for?
I hope some of my reflections are helpful. The talk ended with an open discussion, and Iโd love to continue it here; curious how others are thinking about these questions.
Personally -- as much as there's FOMO, I also think life is very long, and we have time.
toruowo.github.io/blog/postsโฆ
Chongkai Gao retweeted
AlphaGo and coding agents need reasoning, so does robotics, which is arguably orders of magnitude harder. But what is the right representation to perform reasoning?
Excited to introduce DreamCan, led by @GChongkai ! DreamCan performs sparse visual planning, reasoning over sparse visual frames that are both **reachable** by the robot and **makes progress** towards the task goal, and it can learn this capability from action-free suboptimal videos!
Check out Chongkaiโs post for more details ๐งตโฌ๏ธ
๐คRobots need world models for visual planning, but they don't need to imagine every frame.
Introducing DreamCan: Sparse Visual Planning for Long-Horizon Robot Manipulation.
Project: dreamcan.github.io/
Code: github.com/HeegerGao/DreamCaโฆ
Paper: arxiv.org/abs/2610.24981
13/13 ๐งช Ablations show why the balance matters.
LIBERO-Long success:
All three terms: 91.2%
Without reachability: 78.0%
Without task utility: 74.2%
Without spacing: 83.8%
Useful progress and feasible transitions work best together.
Huge thanks to all collabrators! @wenlong_huang @HaonanChen_ @linshaonju @du_yilun @jiajunwu_cs @RuohanZhang76 @drfeifei