Computational Cognition

Why can #ReinforcementLearning (RL) achieve results beyond #OptimalControl (OC) in many real-world robotics control tasks? We investigate this question in our paper published today in @SciRobotics, available Open Access: science.org/doi/epdf/10.1126…. Video: youtu.be/HGULBBAo5lA RL's most impressive achievements are beyond the reach of existing optimal control-based systems. However, less attention has been paid to the systematic study of fundamental factors that have led to the success of reinforcement learning or have limited optimal control. This question can be investigated along the optimization method and the optimization objective. Our results indicate that RL outperforms OC because it optimizes a better objective: OC decomposes the problem into planning and control with an explicit intermediate representation, such as a trajectory, that serves as an interface. This decomposition limits the range of behaviors that can be expressed by the controller, leading to inferior control performance when facing unmodeled effects. In contrast, RL can directly optimize a task-level objective and can leverage domain randomization to cope with model uncertainty, allowing the discovery of more robust control responses. This work is a significant milestone in agile robotics and sheds light on the pivotal roles of RL and OC in robot control. Beyond the fundamental study, our work contributes an RL-based controller that delivers the highest performance ever demonstrated on an autonomous racing drone. Our drone achieved remarkable performance: peak acceleration greater than 12 g and peak velocity ~108 km/h, all within minutes of training with a standard workstation. Kudos to my coauthors @realyunlong, @roaguiangel, the pilots @captainvanover, @bmsthomas, and our supporters @ERC_Research, @m_schaepman, @UZH_en @UZH_Science, @UZHspacehub, @nccrrobotics, @aerialcore