PhD student @UTCompSci | Learn to understand ourselves and build intelligence.🤖🧠👁️
Austin, TX
Joined July 2023
- Tweets564
- Following3.1K
- Followers823
- Likes9.7K
Pinned Tweet
🤖Co-training is everywhere (sim↔real[e.g. GR00T, LBM], human↔robot[e.g. PI, EgoScale], even non-robot data[e.g. PI, LBM).
But why does it work? How can we improve it further?
Taking sim-and-real imitation learning in diffusion/ flow-based models as the test bed, we performed a rigorous mechanistic analysis, drawing on theoretical insights and multi-layered experiments.
😮Key insight: it’s all about representations.
- Alignment → enables transfer
- Discernibility → enables adaptation
⚖️Both are necessary — it's better to have more aligned representations, but the model must be able to discern the domains. We term this as structured representation alignment.
⬇️Let’s take a deep dive into that:
Paper: arxiv.org/pdf/2604.13645
Website: science-of-co-training.githu…
Yu Lei retweeted
Introducing GR00T 2: our most advanced, open System One model for robotics. It runs entirely on your robot’s onboard compute, fully untethered, and you own every weight of it. GR00T 2 features:
1. Long-horizon tasks: dexterous manipulation and whole-body control chained over 20+ steps and 5+ minutes, such as performing chemistry lab experiments or assembling model airplanes end to end.
2. X-Adapt: adapting from easy data sources (human wearables and simulation) and deploying zero-shot to diverse real robots.
3. Unified Cognition: native integrations with frontier agents for advanced physical reasoning, robust error recovery, and safety monitoring.
It's an open-source buffet! We are releasing not only the model weights and X-Adapt recipes (coming December), but also our agentic framework, simulation engine, annotation engine, hardware design, and even training data (100K hours, that's ~11 yrs of experience), all available today!
Everything that moves will be autonomous. Anyone can become a roboticist.
Lock in, build with us!
Deep dive in thread:
Yu Lei retweeted
Fable 5.2 shows substantially improved skills in controlling physical robots, even on tasks it has never explicitly been trained to do.
On RoboDojo, a benchmark with a simulated two-arm robot, it outperformed other models and most AI systems built to control robots.
Yu Lei retweeted
Meet GR00T 2, our new generation of world-action models (WAMs): it imagines how the physical world will unfold and generates the actions that get it there. One model runs on parallel grippers, dexterous hands, and humanoids.
research.nvidia.com/labs/gea…
Excited to welcome y'all coming to Austin to attend the amazing #CoRL2026 🤠.
Looking forward to catching up with old friends and meeting new ones!
Been thinking about representation learning and (mental) simulation in the era of agentic robotics, multimodal pre/co-training, and deep learning science.
Please reach out if you'd like to chat!!
Insightful study as always.
Robot arms can be extremely heavy compared to the objects they are asked to manipulate, especially with their metal links and highly geared motors.
But is that actually a problem for robot learning, or are learning algorithms powerful enough to handle it? Are we paying any hidden costs for using heavy robots? Here's what we found 🧵
Yu Lei retweeted
In sim-and-real co-training, robots learn from a few real demos plus ample sim data. But which sim data should we generate?
MetaGenSim meta-learns what to simulate: 2.5× the real-world success of uniform sim data, matching expert hand-tuning.
meta-gensim.github.io
#CoRL2026
Yu Lei retweeted
The science of pretraining deserves more attention!
What makes a good pretrained model? How to evaluate the it? When should we stop training? And how do we optimize pretraining for downstream SFT or RL?
Some answers may be counterintuitive, e.g., the best-performing checkpoint may not be the best starting point for downstream learning.
If you're curious now, check out @s_zhanyi's work “Beyond Validation Loss” 👇
Can more pretraining improve a robot policy’s success rate, yet makes it worse for porttraining? 🤔
In our new work, we find that it can. Continued supervised pretraining of generative robot policies can improve reliability at the expense of coverage, and this tradeoff affects downstream posttraining.
Read on to see why this happens and how we can preserve coverage. 1/n 🧵
Yu Lei retweeted
Back in ~Dec 2025, I really wanted to work on mobile manipulation, but robots were either too expensive or not performant enough.
So we built our own!
Excited to finally share something I’ve had so much fun working on:
1. Pixel: a co-designed full stack system for learning mobile manipulation.
2. Pixel-UMI: environment aware whole body manipulation without robot data.
w/ @JasonJZLiu
(1/N)
Yu Lei retweeted
Can more pretraining improve a robot policy’s success rate, yet makes it worse for porttraining? 🤔
In our new work, we find that it can. Continued supervised pretraining of generative robot policies can improve reliability at the expense of coverage, and this tradeoff affects downstream posttraining.
Read on to see why this happens and how we can preserve coverage. 1/n 🧵
Yu Lei retweeted
i've reproduced the results from the MIT blog post today on weight-space programming. but one additional result i have to add: this is a new, step-change emerging capability that's appeared in just the latest set of models.
Yu Lei retweeted
We introduce 3D World Simulator, an interactive world model that runs at 50+ FPS on an RTX 5090.
Our model predicts in a unified 3D representation space, capable of generating:
• Novel view appearance
• Multi-view consistent RGB-D
• 3D geometry
• Spatially aligned tactile signals
Play with the world model now at jwang771.github.io/3dworldsi…!
You definitely want a world model that do efficient test-time-adaptation. Please check out @yluo_y’s nice work!
🌏 World models need to capture a scene’s hidden physics properties, e.g., mass or friction, to generate predictions consistent with physical reality.
🚀 Excited to introduce PhysiCalWorld: In-Context Physics Calibration for World Models! We calibrate world models through an interaction clip, grounding future predictions in scene-specific dynamics.
🏷️ Supervision: trained without physics labels
⚡ Efficiency: minimal additional inference cost
⚛️ Generalization: extrapolation beyond the physics ranges seen during training
physical-world.github.io/
More below 🧵
1
2
453
Yu Lei retweeted
The Platonic Representation Hypothesis is now a Theorem ◼️
Train JEPAs separately, on any modalities: they provably learn the same representation up to one rotation.
The result? A 109M JEPA-Text model that aligns with vision models better than 72B LLMs.
w/ @ylecun 🧵
Yu Lei retweeted
Introducing DreamFAR, a world model trained from scratch using only robot data, without internet pretraining. It supports:
🎮 Real-time teleoperation
⏱️ Minutes of long-horizon rollout
🧩 Generalization across 1,000+ tasks
yimingdou.com/DreamFAR
🧵 (1/8)
Yu Lei retweeted
This is Genie 4, our most realistic and interactive world model yet. 🌐
Explore dynamic environments in real time to see how the scene responds, sounds, and how the people in it move, react, and speak.
Yu Lei retweeted
🎉 After one year of robot pretraining, we learned one thing the hard way: data is everything.
Today we're excited to release FOMI, a Force-feedback Manipulation Interface for teaching contact-rich manipulation!
Inspired by @SongShuran's UMI, FOMI keeps handheld, robot-free data collection and adds what operators have been missing: you can feel what the gripper touches ✋ No force sensors, no motors.
Two key results:
🔑 29% more successful demos than UMI in the same time, and 18% more than GELLO in teleoperation
🔑 π0.5 fine-tuned on about an hour of FOMI data per task reaches 78.4% average success on four contact-rich tasks
Let's check out our demos 👇
Yu Lei retweeted
What can a robot hand learn from seconds of human motion?
It turns out: dexterous skills that generalize!
Excited to share AdaDex: Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking.
From one short hand–object clip per skill, AdaDex learns an adaptable dexterous skill, with skills spanning functional grasping, object reposing, in-hand reorientation, and throw-and-catch.
Project page: yuxuank.com/AdaDex/
🧵👇 (1/8)