djhardcore

I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data. Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less? Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.

I believe everyone should have access to superintelligence, and I wrote a long piece about Meta's philosophy and values for building a positive future for everyone. meta.com/thefutureisforevery…

Investors are betting billions of dollars that robotics will experience a Giant Leap. Meaning: robots are not useful today, but throw enough GPUs, models, data, and PhDs at the problem, and you’ll cross some threshold on the other side of which you will meet robots that can walk into any room and do whatever they’re told. The Giant Leap view is sexy. It holds the promise of a totally unbounded market – labor today is a ~$25 trillion market, constrained by the cost and unreliability of humans; if robots become cheap, general, and autonomous, the argument goes that you get Jevons Paradox for labor - available to whichever team of geniuses in a garage produces the big breakthrough first. This is the type of innovation that Silicon Valley loves. Brilliant minds love opportunities where success is just a brilliant idea away. My friend @evanbeard is betting that progress will happen by climbing the gradient of variability. That robotics will progress towards general usefulness in small steps. The logic is clear: - Robotics is bottlenecked on data. - The best data is the data your robots collect actually doing things. - The best strategy, then, even if it's not the sexiest, is to get paid to collect that data, learn, and iterate. This is where the vast majority of value lies, and the real path to our abundant robotic future. For the first co-written essay in not boring world, Evan and I write about the robots.

The ultimate life hack is the ability to quickly reset and recover. From a bad interaction. From a bad day. From a missed workout. From a poor decision. You can start over whenever you want. You can't always control what happened, but you can control how long you carry it.