Pinned Tweet
> Some companies hire heavily out of Twitter, some hire from communities such as GPU Mode or NanoGPT speedrunning.
To Nathan's point, I am leading an open source workgroup within the @GPU_MODE community (#teenygrad channel) in order to develop a deep learning systems course with an MIT-licensed book, codebase, and lectures which develops your own deep learning framework teenygrad from scratch which can run nanogpt. The project has access to some compute thanks to to the @LambdaAPI research grant (thank you @chuanli11)
This project has been a labor of love the past few months, bridging a must-needed pedagogical gap from micrograd to tinygrad. The SITP book develops teenygrad framework step by step, from a numpy clone, to a pytorch1 clone, to a pytorch2 style compiler. The SITP philosophy subscribes to the same views as @karpathy on education: it's a technical problem whose solution requires a ramp with empathy:
> ..education is the very difficult technical process of building ramps to knowledge...I feel like education is..a tangle of understanding and you're trying to lay it out in a way that creates a ramp where everything only depends on the thing before it.
The project's primary challenge for better and for worse has been the breadth of scope. A lot of time was spent "curriculum engineering", and we are now just getting to implementing accelerated cpu and gpu kernels with automatic differentiation in earnest, but it has a good line of sight towards fusion compilation using tinygrad's RISCy IR. The good news for you is that now is a perfect time to help the workgroup, and to come join in learning from the best. There are some heavy hitters here led by @marksaroufim, @m_sirovatka, @a1zhang, @gaunernst and more.
Links below ⬇️
My raw thoughts on the job market -- both for those hiring and those searching -- at the cutting edge of AI.
interconnects.ai/p/thoughts-…
the functional apis that underlie the stateful optimizers are being publicized with torch 2.15 via torch.optim.functional(). use case is optimizing with lower precision optimizers such as bf16 for adamw before the api get's settled. neat 🤓
youtube.com/watch?v=m1CA-saS…
docs.pytorch.org/docs/main/o…
zipping up the back of the book: for automatic differentiation sitp is going to use the same functional notation used in
1. sussman and wisdom's fdg (focusing on the lie and covariant derivative for general relativity)
2. spivak's calculus on manifolds (focusing on stokes' green, and div theorems)
3. frostig's guest chapter in murphy
with derivative as linear operator afforded by borscht.func() api with grad, jvp, and vjp.
—but!, under the hood borscht is going tape-based operator overloading because it is explicitly bridging to torch. not the jax style source transformation. i will have to read the torch source for how torch.func() sits on top of the dispatcher and engine. this is the best i can do to keep the lambda calculus alive given the constraints.
torch.nn's class Module() and torch.optim class Optimizer() are indeed quite hackable :-)
torch.Tensor and torch.Tensor.backward() is natively written in cpp with the dispatcher, a10's native_functions.yaml, and engine's derivative.yaml, so have a little more work to do there :-(
revisiting this tweet storm on "what would a revolutionary rewrite of torch look like" qt as the teaching autograd borscht is being set up properly, particularly the tradeoffs in a mixed source deep learning framework between python and c++, this exchange that spun off from elon's training stack rewrite helps clarify a lot.
—i.e one of the core tensions/pareto curves that exist in the minds of torch's core team is productivity vs performance, which will continue to exist after agents (the segfault remark with $100m training run), and then the original qt lists a possible compromise is keep native extension modules but make them modular via cpython, and have a python implementation as well. maybe if research engineers were all rustaceans we could rewrite torch to be pure rust, which perhaps if you were writing a training stack in-house, is something you might do .. matklad.github.io/2023/03/28…
for sitp's pedagogic goals, it seems nice to follow this. that way borscht can not only use torch as oracle, but also it's own python implementations.
zipping up the back of the book: for automatic differentiation sitp is going to use the same functional notation used in
1. sussman and wisdom's fdg (focusing on the lie and covariant derivative for general relativity)
2. spivak's calculus on manifolds (focusing on stokes' green, and div theorems)
3. frostig's guest chapter in murphy
with derivative as linear operator afforded by borscht.func() api with grad, jvp, and vjp.
—but!, under the hood borscht is going tape-based operator overloading because it is explicitly bridging to torch. not the jax style source transformation. i will have to read the torch source for how torch.func() sits on top of the dispatcher and engine. this is the best i can do to keep the lambda calculus alive given the constraints.
jdz retweeted
"Surgical" questions that are meant to precisely test a particular misconception, say. It will get to the right ballpark and then sometimes *completely* flub it. But if I correct it, it's good at picking up from there and suggesting variants, etc.
possibly something like this (in red). also, check out the syllabus' puns in blue :D
— banged them all out in just a few minutes xD the inner zillenial needed to come out after too much serious writing in sitp's exposition
everyone knows the feeling you get when you see someone from your alma matter (water water water) in the bay area/on twitter. but being part of the kz family now unlocks a whole new level to that. go dana!
this is not only a go kz post. i'm also exploring turning the curriculum sitp.ai into a multimodal one, motivated by working on the labs. i thought to myself, if the lectures and textbook already shows the math/code for attention and transformer, what is the lab going to consist of? i.e what's sitp's test set?
—and so picked up a microduck and looking into lerobot with vla's. maybe sitp's labs which implement borscht can implement a subset of the calls lerobot makes to torch?
everyone knows the feeling you get when you see someone from your alma matter (water water water) in the bay area/on twitter. but being part of the kz family now unlocks a whole new level to that. go dana!
this is not only a go kz post. i'm also exploring turning the curriculum sitp.ai into a multimodal one, motivated by working on the labs. i thought to myself, if the lectures and textbook already shows the math/code for attention and transformer, what is the lab going to consist of? i.e what's sitp's test set?
—and so picked up a microduck and looking into lerobot with vla's. maybe sitp's labs which implement borscht can implement a subset of the calls lerobot makes to torch?
Cuteness overload!! 🐥💛
Sharing my overview of MicroDuck by Pollen Robotics and Hugging Face here as well!
If you find this little robot as adorable as I do, pre-orders are open at pollen-robotics.com/ 🚀
sidenote: sitp as a course has slightly different constraints compared to something like the amazing stanford336 which provides the conceptual lecture and asks learners to complete procedural assignments.
—sitp wants to "just show learners the code" literary programing style similar to karpathy's makemore and nanogpt. but it also wants labs for learners to have the chance of discovery for themselves.
i like the model created 3b1b style videos that are coming out. can we rl them to encourage perspectives/socratic questioning?
