yup, curriculum learning is probably as old as RL :D People usually think of RL as "the agent collects its own data", but the robot can't really pick its own data to solve a hard exploration problem, and see useful gradient signal, if the chance of seeing success is super low (in a sparse reward setting). Which then leads to using denser rewards so that the gradient signal is not impossibly rare to stumble upon. Which then leads to tons of reward tuning and task-specific curriculums. Which then leads to suboptimal policies which are optimizing wrt a soup of mishmashed objectives. Which turns out leads to an inability to scale. So the details of SGS that matter are: 1. Expose a ton of "task configurations" (initial state, goal) to the robot, so that the difficulty of the set of tasks the robot can actually pick from is actually diverse. 2. Instantiate a very simple, automatic, adaptive curriculum based on success rate of each of these task configs (SGS) so that the robot doesn't waste experience in tasks that are neither too easy nor too hard. So the surprising thing is that this simple instanciation lets us train policies with a sparse reward formulation, at much higher scale than previously possible, to achieve really hard robotic tasks :D

(1/N) Who says you can't do contact-rich, precise manipulation with sim-to-real RL? No demonstrations, no retargeting, no tactile sensors, no crazy reward tuning. Just 1x, reactive policies trained with scaled up RL. Excited to finally release Success-Guided Sampling (SGS)! More info in @octi_zhang's thread and more thoughts of my own below! 🧵 Website here: sgs-rl.github.io