Dhanvi

currently waiting over 8 hrs for my pi0.5 model fine tuning to be scheduled on shared GPUs from school. Seeing that everyone shipped an online version of WAMs this week, decided to go back to one of the fundamental papers and understand the basics: DreamGen by @NVIDIARobotics . day 4/365 of robot learning. DreamGen is a pipeline to make open-world generalization possible. This means we one day wouldn't have to teach a robot every skill specifically, just one general skill it can apply to many things. For instance, pouring a cup of water and pouring a jug of water into a vase are almost the same task, and DreamGen helps a robot treat it like such. DreamGen is divided into 4-steps: 1. Fine tune the video world models on a target robot, learn the dynamics of the robot 2. Prompt the models with initial frames to generate robot videos that behaviors in the target domain, but also new behaviors in new environments. AKA generate videos in the desired target space + enable new behaviors in new spaces. 3. Extract "pseudo robot actions" via latent action model, or inverse dynamics model. 4. Use the videos labeled with pseudo actions, named as neural trajectories, to help with downstream visuomotor policy learning. AKA first tune video world models to understand the robot they're in, then have it imagine the robot performing a task, and then label the videos with pseudo actions to teach the robot a skill. They lowered the barrier of entry required for data. You only need around 10 teleoperated examples per task. You do need thousands of demos up front (even of just one task, like pick-and-place) for the video model to learn the robot's dynamics. However, the researchers note that it's computationally intensive (240k RoboCasa samples took 54 hrs on 1500 L40 GPUs). This cuts down the need for teleop data but replaces it with compute. A couple questions I had - is it possible for DreamGen to create and subsequently train on hallucinated videos with poor physics in them? What if we introduced a step that verified each video's physics? Since DreamGen found success rising in linearly against the log of the generated trajectories, at what point does a GPU beat a person with a teleop rig? How precise can we get the hallucinated videos to be? One problem I'm facing with the SO-101 is the gripper being 1-2cm off of the pickup, and botching the entire task.

Reinforcement learning has been interesting on the PushT game... day 3/365 of robot learning I've tried 4 different reward systems for teaching a model to win at PushT via RL. The threshold to cross is 95% coverage, yet every reward system seems to fail it. Running 2 million steps per reward system on an M2, here are the results: Attempt 1: Reward increases linearly as coverage gets closer to 95% → ~80% coverage, wins up to 1 in 20. Winning ends the game, so hovering near 90% earns more points than actually winning. Attempt 2: Same as above, but −1 for every move, so winning is the only way to stop paying → ~73%. It rarely got close enough to 95% to find out that winning stops the penalty. Attempt 3: Curved reward (also −1 per move). The last few % before 95% pay the most, since the reward grows steeply near the top → ~10%. The reward was so small below 50% that it never learned anything. Attempt 4: Half coverage, half how close the T's position + angle are to the goal (−1 per move) → ~87% coverage, passing 90% in over two-thirds of its practice games. Still only wins up to 15% when tested. It's consistently getting between 80-93% and then stopping... It gets close but rarely actually finishes and succeeds. Through the process I also learned about SAC which rewards the model for staying a bit random so it keeps exploring, rather than settling on a strategy too early. Pretty cool stuff. Any thoughts on other reward mechanisms I can try to help this learn better? Or is the solution simply more steps?

Starting on a journey to get good at robot learning, today is day 1/365. s/o @praneethsamine for the idea how far can a diffusion policy's denoising steps be cut on PushT before it starts to lose? Reduced denoising makes planning faster. Results showed roughly 19x faster with no measurable drop. Context: PushT is a game where a round pusher has to shove a T-shaped block onto a T-shaped target. A game counts as a win if the block covers at least 95% of the target within 300 moves. The number of steps N is how many rounds of refinement the model does each time it plans. Each plan covers its next 8 moves. I tested this today. 100 games each time, with the pretrained HF Lerobot Diffusion Policy. I had the following step levels: 100 steps, 50 steps, 25, 10, and 5. Ran it on my M2 MacBook, DDPM scheduler. Results: - 100 steps: 73% success rate. This is the baseline, 4.3 s per plan. - 50 steps: 65% success, 2.2 s per plan. - 25 steps: 60% success, 1.1 s per plan. - 10 steps: 68% success, 0.43 s per plan. - 5 steps: 70% success, 0.22 s per plan, 19x faster than the baseline. Not a new idea: the Diffusion Policy paper (Chi et al., 2023) already runs 10 DDIM steps on real robots. Difference: I kept LeRobot's default DDPM scheduler, and 5 steps held up. Caveat: most losses were near-misses that ran out of moves by the 94-95% threshold. A small drop can't be ruled out by 100 games. Next: taking this to my senior project, a LeRobot arm moving a die. Does 5 steps hold up on real hardware? full 500 runs in the replies