James Sanders ⚛🌍

Say AI one more time, and we’ll end 2024 for good. It’s been a wild ride. The year went on like a diffusion model - we watch our sci-fi visions gradually denoise and materialize. So let’s celebrate - one line each, lightning round. You ready? 6 chapters: Robot Hardware, Embodied AI (robot brains), Compute Hardware, Video Gen & World Modeling, LLMs, AI4Science. > Chapter 1. Robot Hardware We are the last generation without advanced robots everywhere. Everything that moves will be autonomous. 1) Rise of high-end humanoids: the world is built for us. All the facilities, appliances, and tools are designed around our form factor. - Tesla Optimus: very few humanoid companies have the courage to show live interactive demo in the wild. Tesla did it at "We, Robot" event. Gen 3 hand, 22 DOF, ahead of the game; - 1X Neo: friendly neighborhood humanoid, aiming to deploy massively at homes; - Boston Dynamics e-Atlas: the heavy-duty champion, 360-degree joints unlock some insane gymnastics; - Figure: fast iteration speed from prototype to car factory deployment; - Fourier Intelligence: GR-1 robots are one of the few in mass production (1000s) and being shipped worldwide; - Clone: "Westworld"-style design of biomimetic muscles and tendons, a fresh new perspective on how humanoids can materialize. And a lot others springing up, too many to count. 2) Rise of inexpensive robot hardware: cheaper than cars, highly scalable, very affordable to most middle classes in the near future. - Unitree G1 humanoid: costs ~$40K, weighs 77 lbs and stands 50 in tall. Small but mighty in agility; - Unitree B2-W: robot dog with 4 wheels that out-maneuvers most animals on earth; - ALOHA: dirt cheap, open-source Stanford robot with 2 grippers. Can be teleoperated to do complex motions like making an omelette or folding laundry. 3) Apple Vision Pro: plays an interesting role in robotics as a data collection device. Parses your head & hand pose in real time, and controls the robot to mimic your actions. > Chapter 2. Embodied AI (Robot Brain) Some self-plugs here because this is my home domain ;) 1) Tesla FSD v12: the largest Physical AI data flywheel in history, compressed into a powerful photon-to-action neural net. A paradigm shift in the decade-old autonomous driving problem. 2) NVIDIA launches Project GR00T, a moonshot initiative to build the AI brain for general-purpose robots. Jensen walks on stage at the SAP center with 10 humanoid robots in the background. 3) HOVER: our team trained a 1.5M foundation model that learns how to coordinate the motors of a humanoid, capturing the "subconscious processing" that our cerebellum does every millisecond. 4) DrEureka: our team trained a robot dog to balance and walk on top of a yoga ball purely in simulation, and then transfer zero-shot to the real hardware. An LLM automatically writes the reward function and tweaks the parameters, so we can watch Netflix and still get work done. 5) pi0, from the startup Physical Intelligence: a robot vision-language-action (VLA) model that performs impressive multi-step tasks like laundry folding. Uses ALOHA setup for cheap data scaling. 6) OpenVLA: a Stanford work on open-source VLA model, trained on Open X-Embodiment (OXE) dataset that aggregates robot motion trajectories from labs around the world. > Chapter 3. Compute Hardware 1) Scaling up: NVIDIA introduces Blackwell architecture, a new beast in town. DGX GB200 crosses 1 Exaflop compute in a single rack. 2) Scaling down: Jetson Nano Super, 67 TOPS of AI computing in a $249 mini box, designed for running small LLMs on edge devices like robots. It is NVIDIA's raspberry pi moment! 3) Google Willow Chip: quantum computers are GPUs over the multiverse. Solved Random Circuit Sampling (RCS) in 5 min that would take 10 septillion yrs for a regular computer. > Chapter 4. Video gen & world modeling 1) Sora: announced in Feb, stunned the world, eventually released in Dec, and lost some charm due to the long wait. The first of its kind in high-res, long video generation. Sora is a text-conditioned, soft simulation of the visual world. The model learns intricate rendering and intuitive physics all by some denoising and gradient maths. 2) Veo: as OpenAI delays the release, Google stages a great comeback with more accurate physics and fine-grained object dynamics. 3) Action-driven world models: - GameNGen: yes, you can run DOOM literally anywhere, even inside a diffusion model; - Oasis: yes, you can run Minecraft anywhere, even inside a diffusion model; - GENIE-2: run even more games with joystick controls inside a diffusion model. 4) World Labs (startup led by Fei-Fei Li): stunning demo of a generative 3D foundation model with strong geometric consistency. > Chapter 5. LLMs >> Capability frontier 1) Claude Sonnet-3.5: few are expecting No. 1 from Anthropic, but the mighty Sonnet beats everyone's expectations. 2) Gemini 1.5 pro, 10M context length: learns to translate English to Kalamang (< 200 speakers in New Guinea) purely from context using 500 pages of tutorial, a dictionary, and 400 parallel examples. Learning happens on the fly in fast neural activations instead of slow gradient descent. 3) o1 (project "Strawberry"): popularizes inference-time scaling and figures out a way to deploy widely. As Sutton said in the Bitter Lesson, there're only 2 techniques that scale indefinitely with compute: learning & search. It's time to shift focus to the latter. 4) o3: comeback of reinforcement learning. Give me a reward function, and I shall move the world. The essence of o3 is about relaxing a single-point RL super intelligence (such as AlphaGo) to cover more points in the space of useful problems (such as math and coding). 5) The true AGI test is to complete this sequence: 4o -> o1 -> o3 -> (?) >> Human-AI interface is a criminally underrated topic. LLM capability envelope is way ahead of UIUX design to unlock the full potential of such capability. 6) Rise of real-time speech models, such as GPT-4o (advanced voice mode). It's a one-stage distillation of a traditionally 3-stage process: audio->text (Automatic Speech Recognition or "ASR"), text->response (LLM), response->audio (Text to Speech, or "TTS"). 7) NotebookLM: one of the best reimagining of UX for LLMs: podcast-ify *anything*. Lowers barrier to entry: no need to prompt, no need to read. Just upload a file and listen to 2 streams of tokens unfold. >> Open-source community 8) llm.c: Andrej Karpathy crunches tokens in naked C. No python, no frameworks, no dependencies. Just raw math talking to bare metal. He writes poetry in code. 9) Llama-3: the power of GPT-4 in the palm of your hands. 10) DeepSeek: biggest dark horse in OSS LLM this year. Shows that resource constraints force you to reinvent yourself in spectacular ways. > Chapter 6. AI4Science 1) Nobel Prize in Physics: today, we are all physicists. AI PhDs - read my hitchhiker's guide to rebranding. Machine learning is now "statistical mechanics". You are eligible for the top award now. 2) Nobel Prize in Chemistry: I double dare you, say Nobel one more time ... jokes aside, congrats to Demis and the AlphaFold team. A breakthough of a lifetime. Solve AI first, and then use AI to solve everything else! 3) Neuralink: someone with spinal cord injury is now able to browse the web, live stream, play Civ VI, and even Mario Kart on a Nintendo Switch by controlling a mouse with his mind. Neuralink maps the biological neural pulses to artificial neural pulses, which then maps pixels to cursor actions. So underrated. > Closing Thought There are many, many great AI moments that I couldn't possibly enumerate in this post. Can't believe it's already been 2 years since the debut of ChatGPT, and 7 years since the Transformer paper. Lots of hype, promises, and fears. I try my best to improve the signal-to-noise ratio for my audience. I'll end with Wilson's quote: "The real problem of humanity is the following: We have Paleolithic emotions, medieval institutions and godlike technology." The good news is, we get to build the technology, which means we also get to define the role it plays in our institutions, societies, and cultures. AI is an inevitable part of humanity's future. It can be a tremendous force of good if we choose to apply the right force. Happy new year everyone!

What makes up the abstract concept of an apple? We read the word "apple" as a string, see 2D pictures online, 3D shape in real life, and moving apples in videos. We touch the apple, feel its geometry in our palms and texture through the rich tactile sensation on our fingers. Do all these different modalities converge to the same representation space, given sufficient learning capacity? After all, they are all shadows of one "true reality" projected onto our different senses. I really like this study paper from MIT, "The Platonic Representation Hypothesis". The authors show that highly capable LLMs and vision models learn very similar representations, even though the modalities are never explicitly co-trained. Concretely, the experiments compare the similarity between strings "apple" and "orange" with the similarity between a picture of an apple vs orange. These two turn out to agree with each other in a wide selection of off-the-shelf models. Reminds me a bit of the famous word vector experiment: f("king") - f("queen") ~= f("man") - f("woman"). But this time, the equation holds across modality, datasets, and completely separate training pipelines. I would love to see this study extended to more modalities: how about 3D point clouds? Speech? Sound of a lion vs a bird? Tactile signals on a robot hand? Can we train meta-models that transfer to *unseen* modalities at test time, since it's just another projection of reality from a different angle?