Sang Phan (@plsang) header
Sang Phan (@plsang) profile picture

long long ago; /* in a galaxy far far away */

Tokyo
Joined May 2011
Photo by Sang Phan (@plsang)Photo by Sang Phan (@plsang)
Sang Phan retweeted
Over the past week, I’ve been overwhelmed by the love and support of friends, industry colleagues, and people I've never met. Thank you to everyone who has shown kindness to my family and me. It means the world to me. I came to the United States 53 years ago, and becoming an American citizen is one of the greatest gifts and proudest moments of my life. NVIDIA has been my life’s work. Building it into the company it is today alongside my colleagues has been a privilege and an honor. Nowhere but in America could a company like NVIDIA be possible. I love our country, and I love NVIDIA. I did not live up to the standards expected of everyone at NVIDIA, and I have made serious mistakes in my personal life. I created problems for the Board that I deeply regret. I own my mistakes completely, and I am committed to cooperating with the Board’s independent investigation. Since my departure, I’ve seen an outpouring of calls from NVIDIA’s employees, investors, business partners, and customers for me to return as CEO. Many continue to reach out to me directly as the company navigates complex technology, market, competitive, and geopolitical dynamics. NVIDIA is a once-in-a-generation company. If the Board of Directors decides that my return is in the best interest of our shareholders, our employees, the company, and America, I would be honored to answer these calls and return as CEO. Thank you again to everyone who has shared their good wishes. I am profoundly humbled by your support. To my family: I love you, and I will spend the rest of my life being worthy of yours. Jensen
51,091
106,970
22,242
984,108
40,644,341
Sang Phan retweeted
Mechazilla catches Starship! Congratulations to the entire SpaceX team on this incredible achievement!
6,799
27,314
9,091
140,972
17,193,444
this is wild 🤯🤯🤯 just paired openai operator with replit agent to build an app watch as two ai agents team up, exchange credentials, and start testing ai agent 🤝 ai agent the future is here and it’s insane!
338
977
296
8,508
1,823,255
It's only been 1 day of CES 2025, and the announcements have already been incredible. The 10 most impressive reveals of CES 2025 so far: 1. A 360° AI-powered body scanning health mirror that can scan your heart, weight, and metabolic health
670
5,670
855
45,850
14,609,238
Sang Phan retweeted
Introducing Gemini 2.0 Flash Thinking, an experimental model that explicitly shows its thoughts. Built on 2.0 Flash’s speed and performance, this model is trained to use thoughts to strengthen its reasoning. And we see promising results when we increase inference time computation!
123
466
180
3,742
1,470,963
Introducing Willow, our new state-of-the-art quantum computing chip with a breakthrough that can reduce errors exponentially as we scale up using more qubits, cracking a 30-year challenge in the field. In benchmark tests, Willow solved a standard computation in <5 mins that would take a leading supercomputer over 10^25 years, far beyond the age of the universe(!).
2,810
11,802
5,622
74,385
19,276,197
Sang Phan retweeted
The tower has caught the rocket!!
44,113
130,192
22,600
1,102,388
144,884,358
The most common question about 3blue1brown is how I animate videos, so I made a video to give a peek behind the scenes. youtu.be/rbu7Zu5X1zI
56
554
68
5,483
302,885
Interested in 3D object detection and egocentric vision? We open sourced a small yet challenging dataset called Aria Everyday Objects (AEO). We use it as one of the tasks for benchmarking a new class of Egocentric 3D Foundation Models we are working on: arxiv.org/abs/2406.10224
12
148
14
759
56,755
🔥llm.c update: Our single file of 2,000 ~clean lines of C/CUDA code now trains GPT-2 (124M) on GPU at speeds ~matching PyTorch (fp32, no flash attention) github.com/karpathy/llm.c/bl… On my A100 I'm seeing 78ms/iter for llm.c and 80ms/iter for PyTorch. Keeping in mind this is fp32, with no flash attention yet, and slightly stale PyTorch (2.1.0). - It is a direct implementation of the training loop and backpropagation in C/CUDA. - It compiles and runs instantly. No more "hit run then wait for tens of seconds for unknown reasons", for mountains of inscrutable abstractions to build a Universe. - It deletes the need for the Python interpreter and a deep learning library. - It allocates all the memory a single time at the start. - It's pretty cool. How: Getting this to work required us to write a lot of custom CUDA kernels, and doing this manually (instead of using Tensor ops of aten/PyTorch and torch.compile etc.) is a bit like programming in assembly. And you spend quality time looking at more assembly (CUDA PTX/SASS). But this also means we get to hyperoptimize the code and possibly explore optimizations that torch.compile might find difficult to, which is awesome. Examples of optimizations that went in over the last few days: - we're being clever with our memory consumption in the backward pass, only using a few buffers we need to propagate the gradients, saving memory capacity. - one fused classifier kernel does the last layer forward pass, the loss, and kicks off the backward pass. - many improvements to all the kernels involved, including e.g. gains from carefully constraining execution within the autoregressive mask in attention - cuBLAS(Lt) calls for all heavy lifting matmuls, and fused bias accumulation Big credits to two CUDA experts who appeared from somewhere on the internet to help this open source project, ngc92 and ademeure. We're hanging out of Github and Discords of CUDAMODE and my NN Zero to Hero. Next steps: - more optimizing of our (fp32) kernels, and especially switch to flash attention. - mixed precision training (fp16 to start). - multi-gpu training (DDP to start). - data & evals to set up a proper GPT-2 training runs - 🚀 repro GPT-2 (1.6B) training run. - more modern architectures etc. (Llama 3?) - writing, videos, exercises on building all of this from scratch. Figure 1: eye candy: timing profile of the kernels (one layer). NVIDIA cutlass kernels with solid compute throughput taking up a lot of the running time => nice.
129
506
68
5,010
641,116
The hottest new programming language is English
2,297
8,865
1,964
72,880
12,901,772
Sang Phan retweeted
The best linear algebra course out there. PERIOD. For Free! MIT's Professor Gilbert Strang. Go through these videos, and you'll never ever have a problem with linear algebra again! ocw.mit.edu/courses/18-06-li…
87
1,456
93
8,476
Very impressive---indeed, awe-inspiring---AI demos this last week, e.g., from OpenAI (image generation) and Google (text generation). These demos seem to convince many people that current AI is getting closer and closer to human-level intelligence. 🧵 (1/8)
29
291
67
1,335
📢📢📢 Happy to introduce that "Kubric: a scalable dataset generator" was accepted at #CVPR2022. arxiv.org/abs/2203.03570 – Do you work on video and/or 3D vision from images? – Having a hard time applying the "scientific method" to your research? Look no further 🧵👇
13
175
10
950
If you think you know about trees, look at this.
320
11,091
946
53,061
The ongoing consolidation in AI is incredible. Thread: ➡️ When I started ~decade ago vision, speech, natural language, reinforcement learning, etc. were completely separate; You couldn't read papers across areas - the approaches were completely different, often not even ML based.
298
1,585
251
7,865
Sang Phan retweeted
Reconstructing objects in 3D is an important #computervision problem with many AR/VR applications. We're releasing Common Objects in 3D (CO3D), a new data set of nearly 19,000 videos capturing objects along with highly accurate 3D reconstructions. More: ai.facebook.com/blog/common-…
6
124
20
526
I am pleased to announce that the camera ready version of my new textbook, "Probabilistic Machine Learning: An Introduction", is finally available from probml.ai. Hardcopies will be available from MIT Press in Feb 2022.
43
733
43
3,513