Contracted AI retweeted
[1/2] “The Task Selects the Hand”
Today we introduce a method for synthesizing families of gripper morphologies using reinforcement learning.
Starting from a single first-person video of an everyday task, we extract the hand, object, and contact trace; reconstruct the scene into a physics-ready simulation environment; and compose and train a gripper for that task, rather than forcing the task onto an arbitrary gripper. Every candidate that succeeds in simulation is then subjected to static finite element analysis under the working load, and any hand that cannot carry it is vetoed. The method closes with a printable CAD file describing the resulting gripper.
The four finger hand produced in this blog, PR-HAND-F4J2, did not exist before the video.
Link below 🧵
Contracted AI retweeted
One brake job could contain hundreds of robot training examples.
@contractedai has released the first volume of its Embodied Cognition research with the @physicreality team, exploring how real human work can be converted into training data for robots.
They recorded an expert mechanic and broke the work down into fine-grained actions including where to reach, which hand to use, what to look at next and how to react when something unexpected happens.
The pipeline goes considerably further than raw video:
-> atomic action annotations with reasoning traces
-> stereo views and metric depth
-> 3D scene reconstruction
-> human-to-robot motion retargeting
-> counterfactual demonstrations with individual conditions changed
One finding stood out: the mechanic could predict moments that frontier models would struggle to recognize.
Even when models could reason correctly after being explicitly prompted, they often failed to notice those moments themselves.
That turns expert workers into more than demonstration sources. They can also help identify exactly where models need better data.
Dyna-2 for instance was recently pre-trained on 1M+ hours of egocentric human video and showed that increasing human-video scale improved performance on previously unseen robot data.
An hour containing hundreds of expert judgments could be a very different training asset from an hour of repetitive manipulation.
🎥 Credits: @contractedai
The same pairs train world-action models: a WAM predicts what happens next if you act. That takes two filmed futures of one scene: the action taken, the action changed, everything else held constant.
Both futures are real, filmed, and verified end to end.
1
4
430
Every pair doubles as an eval: name the change and models answer; stay silent and see who raises it.
Underneath: calibrated stereo, metric depth, 3D reconstruction, atomic annotations, expert reasoning traces, all on the same jobs. The spatial suite is Physical Reality's work.
1
4
365
Full study, all fifteen situations, worked failures, and the filmed pairs, open in a browser:
contracted.ai/research/21398…
Volume 2 is the training work, and scaling what works.
5
324
Can an embodied reasoning model do a brake job?
We ran 9 VLMs through an expert mechanic's exam on his own repair, with @physicreality. In theory they know what to do. Noticing and adapting on their own is where they fall short.
This is where deployment would fail.
Our first research collaboration!
[1/2] In collaboration with @ContractedAI:
"Knowing, but not Noticing"
- We show that frontier ER models (which robot foundation models dominantly initialize from) consistently fail to know 'what to do next' in a chained task;
- how to measure that failure quantitatively using data sourced from a novel branching/ 'counterfactual'-style ego data collection regime; and
- propose a means to produce post-training data to fill in those gaps, based on what context ablations via harnessing reveal the model already knows, but simply fails to conjure reflexively on its own.
Link in thread 🧵



