Building Physical AI @DecisionFacts — teleop & UMI demo data, sim-first VLA training (ACT · SmolVLA · GR00T). University of Michigan
SF Bay Area, California
Joined July 2008
- Tweets306
- Following121
- Followers56
- Likes128
Pinned Tweet
Great to see SO-101 Bench, @h1ckok. Cheap sim you can trust is the unlock for low-cost robots.
A complementary experiment: we replayed real SO-101 teleop (cup nesting, 200 episodes) in open-source MuJoCo. Sim joints stay within ~1° of the real arm on average.
Would nesting make a good next task? Dataset 👇
huggingface.co/datasets/Deci…
@huggingface @LeRobotHF
Most robot demos only show the clean take.
This SO-101 run opens a drawer, places surgical forceps and closes the drawer:
❌ 4 missed handle grasps
✅ drawer opens on try 5
⚠️ forceps land crooked
🔁 robot re-grasps and fixes it
At @DecisionFacts we keep all of it in the data. That's what teaches policies to recover.
#PhysicalAI #robotics
3
118
Free sim is the easy part now. The bottleneck has moved to real demos clean enough to validate sim against: synced, with consistent resets and labeled failures. That’s where we spend our time.
Under the leadership of Jensen, everything NVIDIA has built for robotics, apart from chips, is free.
- Isaac Sim and Isaac Lab for simulation, free and open source
- Newton, a new physics engine built with DeepMind and Disney, fully open source
- GR00T, recently launched open foundation model for humanoids
- World model Cosmos for generating training video of the physical world
- Jetson Orin Nano Super at just $249 which is pretty reasonable for hardware
- RoboDojo which is now on Isaac Lab-Arena to test two-arm manipulation in sim
Really hope the CEO transition doesn’t affect the value that the company has been giving to robotics.
Sriram Gopalan retweeted
Introducing GR00T 2: our most advanced, open System One model for robotics. It runs entirely on your robot’s onboard compute, fully untethered, and you own every weight of it. GR00T 2 features:
1. Long-horizon tasks: dexterous manipulation and whole-body control chained over 20+ steps and 5+ minutes, such as performing chemistry lab experiments or assembling model airplanes end to end.
2. X-Adapt: adapting from easy data sources (human wearables and simulation) and deploying zero-shot to diverse real robots.
3. Unified Cognition: native integrations with frontier agents for advanced physical reasoning, robust error recovery, and safety monitoring.
It's an open-source buffet! We are releasing not only the model weights and X-Adapt recipes (coming December), but also our agentic framework, simulation engine, annotation engine, hardware design, and even training data (100K hours, that's ~11 yrs of experience), all available today!
Everything that moves will be autonomous. Anyone can become a roboticist.
Lock in, build with us!
Deep dive in thread:
CoRL week so far:
GR00T 2 learns from human video,
Gemini Robotics 2.5 reports 96% on a factory pilot,
and an Apple paper has a general model beating demo-trained VLAs with zero demos.
None of them talk much about what happens in the other 4%. For anyone deploying: is recovery from a failure what you actually evaluate on?
CoRL 2026 Day 1🤖 @junfanzhu98's Hot🔥 Takes
10 takeaways: shortcut-solvable benchmarks, VLAs that don't listen, video as policy, sim-to-real, time-aware memory, and self-improving agents.
#CoRL2026 @corl_conf
73
96% on parts sorting is solid. On a line, the 4% matters more than the 96%. Does it fail in a way it can detect and recover from, or does it fail silently? I’d love to see that breakdown.
Google announced a new Gemini Robotics 2.5 model for physical AI.
> This model was built with support from the Gemini 4 Argon model.
> In a POC with Apptronik and Mercedes-Benz, Gemini Robotics 2.5 helped the Apollo 2 robot pick and organize car parts for assembly with 96% accuracy.
24
Strong result for instruction changes and pick-and-place. I’d want to see contact-rich tasks before calling it, though. Something like twisting a bottle cap needs tight closed-loop control at a rate a big model in the loop struggles to hit. What control frequency did AMP1 run at?
New Apple paper shows that a general-purpose AI model like Claude Opus 5.5, given camera views and direct control of a robot arm, handled objects more reliably than robot models trained on recorded demonstrations.
New Apple paper shows that a frontier AI model directly controlling a robot arm beats specialist robot policies, with no demonstrations or task training.
Most robot manipulation work trains specialist policies, like vision-language-action models, on demonstrations, and they break easily when instructions change.
Their harness, AMP1, gives a model like Claude Opus 5.5 or Gemini 4 Argon camera views, code execution, and direct motion commands in a loop. It has no built-in grasping or placing skills, so the model invents its own strategies.
On RoboLab-120, AMP1 succeeded in 85.6% of episodes zero-shot, against 49.2% for the best VLA and 42.9% for the best world-action model. On a real Franka arm it succeeded in 104 of 108 trials, though the robot waits much of each episode for the model.
17
Sriram Gopalan retweeted
JUST IN: @VerneRobotics released V0.5, an open-weight model for industrial robots.
It predicts both the next video frames AND the robot’s actions.
( 📌 Weights and code are on Hugging Face. 👇 )
Best thing:
the weights are public, so other teams can fine-tune and use it.
On the LIBERO and RoboTwin 2.0 benchmarks it currently ranks as the strongest open-weight model from a US company. In cluttered and randomized scenes it outperformed Physical Intelligence’s π0.5 by a clear margin.
Both models were trained only on clean scenes.
In demos it loaded drinks and snacks onto a cart and then moved the cart, coordinating a mobile base and both arms, after less than two hours of post-training data. It also recovered when objects were removed or dropped.
A separate high-level layer (ChatGPT Astra) directed multiple V0.5 robots through multi-step logistics and packing tasks.
Verne offers help fine-tuning the model on customer demonstrations for production use.
And... later today I will have CEO and Co-Founder @neil_nie_ on the 22Astronauts podcast, so stay tuned!
📌 Weights and code are on Hugging Face: huggingface.co/collections/v…
Model and inference code: github.com/vernerobotics/v0.…
——
Weekly robotics and AI insights.
Subscribe free: 22astronauts.com
1/ We trained SmolVLA on our own SO-101 teleop data and filmed an unedited real-robot rollout.
It hesitates for 30 s, puts the cup next to the target instead of in it, then wanders for 80 s.
Here's what each failure taught us 🧵
2/ Hardware: a Jetson in 15 W mode with synchronous inference means the arm runs 50 actions without looking, then freezes while the next chunk is computed. Async inference or a GPU server fixes most of the stop-and-go.
3/ Setup: the rollout prompt said "red cup" but the training data said "solo cup". One camera sits on a desk-lamp boom. Small mismatches like these put the policy out of distribution.
4/ Training: there was no held-out eval, and no demos of a "done" state. Recovery data was thin too: an SO-101 benchmark measured a 3% recovery rate for SmolVLA.
5/ More episodes ≠ a working robot. Coverage of failures, recoveries and real deployment conditions matters more.
Model: huggingface.co/DecisionFacts…
Async inference in @LeRobotHF would have fixed most of the stop-and-go in 2/. @foin0ir you called it yesterday: recovery demos are the gap. @drdonv1 worth reading before your π0.5 run on the SO-101.
6
@foin0ir made a good point yesterday: measure how often a slipped grasp needs a human. Here’s what that looks like in our data…” A tag that credits someone’s idea gets replies; a list of tags at the end gets ignored.
1
30
LeRobot v0.7 lets you redirect a robot mid-run: /subtask, /vqa, /autosteer.
The model side is ready for this. The data side mostly isn’t. Steering only works well if the training episodes have clean subtask boundaries, and most demo datasets ship one task string per episode.
Who’s annotating subtasks at scale, and how?
𝗟𝗲𝗥𝗼𝗯𝗼𝘁 𝘃𝟬.𝟳.𝟬 𝗶𝘀 𝗟𝗜𝗩𝗘 🚀🤖
From terabytes of data to prompting robots! Check out all what's new in the video below and join the 50,000+ developers & 300+ organizations in the LeRobot ecosystem 🤗
1
30
This week’s robotics feed has one theme: robot-free data. Pixel-UMI, WACL learning whole-arm skills from human demos, ego2robot turning human video into data for 15 robots, and a paper using an LLM agent as the teleoperator in sim.
From the data-collection side, my take: these cut the cost of the what, but contact timing and force still come from a robot or a sensored gripper.
Where does teleop still earn its cost in 2027?
FOMI dropped today: a force-feedback take on UMI-style handheld demos.
arxiv.org/abs/2610.19839
My read: vision-only demos hit a ceiling on contact-rich tasks. On our UMI bottle-opening data the hard part isn’t the reach, it’s the twist, and the camera barely sees it.
If you’re collecting demos right now: are you logging force/torque yet, or still vision + gripper width?
#umi #PhysicalAI @trossenrobotics @foxglove
1
32
Data-collection rule we keep relearning: don’t delete the failed grasps.
A demo where the operator slips, re-centers and regrasps teaches the policy something 10 clean demos can’t: what to do once it’s already wrong.
We tag those as failure→recovery pairs and train on them on purpose.
The operator missed the first grasp on this SO-101 cup-nesting run. We kept it, because the re-center and regrasp is exactly what the policy needs to see.
@chris_j_paxton
G0.5 just landed in LeRobot, and the most interesting part isn’t the 2B size.
Before each action chunk it predicts the current subtask and boxes around the objects involved. That’s a data spec as much as a model: you need demos with subtask boundaries and object grounding aligned to the right frames.
Most teleop datasets have neither. Are you labeling subtasks in your robot data yet?
G0.5 from Galaxea is now available in LeRobot! 🤖
G0.5 is a 2B vision-language-action model built on Qwen3.5.
Before each action chunkmit predicts:
- the current subtask ("pick up the red cube") from the overall task
- bounding boxes around the objects involved
- then the actions, which are conditioned on that reasoning
LeRobot's annotation pipeline can also generate plans, memory and visual Q&A into your datasets, opening the door to longer, more complex multi-step setups for future policies!
Thanks to the Galaxea team for the open release!
Links to the blog, docs and our walkthrough in the comments 👇
3
143
Sriram Gopalan retweeted
In @TolokaAI’s tests, GPT-6 Astra controlled our bimanual Trossen AI robot straight from the camera feed. When a grab missed, it tried again. Sometimes it changed its mind halfway through.
Writeup: buff.ly/lCbduz7
Check out our hardware: buff.ly/iCkYMVI
Before we spend a single real-robot hour on a new policy, we replay the teleop demos in MuJoCo.
SO-101 pipe sorting, ep 2: real (left) vs. physics sim driven by the same joints (right). <1° mean tracking error; the biggest gaps are on the fastest moves.
Data 👇 huggingface.co/datasets/Deci…
Sriram Gopalan retweeted
LLM vs. WAM: Astra vs. Dream-1. Round 3.
Cup stacking. Same arms. Same task. Zero-shot.
A: 1:06
B: 0:41
Both finish. Which one is the LLM?
A or B? Defend your pick.
#Robotics #EmbodiedAI
Meta reportedly built a humanoid prototype with Fourier in about three months.
That timeline says the body is no longer the hard part. The hard part is the thousands of hours of clean demonstrations, with gripper state and contact, that a body needs before it’s useful.
Who solves that first: glasses video, teleop farms, or sim?
Meta builds humanoid robot with Chinese startup Fourier, The Information says reut.rs/4ecTdip
Everyone is racing to scale robot data: open-source UMIs, paid chore-filming apps, teleop farms.
What at @DecisionFacts we've learned collecting teleop and UMI demos:
Volume is the easy part. What matters is the 10–20% of episodes where something goes wrong and the operator recovers. Those make a policy robust, and they’re usually the ones that get deleted.
Do you keep your failure episodes, or throw them out?
@trossenrobotics #UMI #Robotics #LeRobot #PhysicalAI