One of the dreams of RL is to train models that can quickly learn how to act in any RL environment, even ones they’ve never seen before. This new model is a step in that direction:

Sarah asks interesting questions!

From the preface to Øksendal, perhaps relevant to the present day:

If we end up in a world where many more companies are training very strong models, it will be important to have open source / public training tools that result in aligned models

A provable box!

Maybe you do need giant models after all?

Early signs of GDP growth from AI:

Fixing all software vulnerabilities is very important for the coming cyber apocalypse!

Frontier language models are quite strong robot policies! There are many cool demos floating around twitter, but we wanted to run a larger systematic test on real robots. Some interesting results on the trade-offs between thinking harder and acting slower, recommend reading the post!

Very cool!

Some of the largest-scale funding available for safety projects!

Exciting to see more companies building entirely independent pretraining stacks!

What I think is most interesting about our model is how general the pretraining technique was (which we’re calling goal-conditioned pretraining). There wasn’t anything specific to Minecraft in it, or even about keyboard and mouse. Pretraining is where models get smartest. That’s where the most learning about the world happens, where we add the most information into the weights. We want a technique that allows models to learn how to accomplish goals through pretraining. You can think of video as an RL trajectory that has only observations (it doesn’t have actions or rewards). Video is a little bit special in this way (e.g. you can’t do this with text.) What to do about the missing actions and rewards? One way to deal with missing rewards is to reframe the problem. Rather than maximizing numerical rewards, we use goal-conditioned RL, where the goal is to learn how to reach any state. For video this means prompting the model with images. One of the things that’s great about goal conditioning is that we can do *hindsight-relabeling*, where we use frames from later in a video as goals for the earlier part. (It’s called that because we take a trajectory that may have been a failure for one goal and relabel it as a success for whatever actually happens.) Next, we’re planning on scaling this up to much more diverse videos, and we’re hoping to get a model that can achieve goals in any video game, as well as control robots in the real world. There is a wide space of ideas to experiment with here that have never been scaled up before. If you’re interested in working on these models, feel free to reach out :)