Chen Lu (@_Chen_Lu_) profile picture

now works on computers, used to work on math

Joined March 2021
Photo by Chen Lu (@_Chen_Lu_)
Composer 2 is pretty good!
We're releasing a technical report describing how Composer 2 was trained.
1
29
1,561
LTM-2-Mini is our first model with a 100 million token context window. That’s 10 million lines of code, or 750 novels. Full blog: magic.dev/blog/100m-token-co… Evals, efficiency, and more ↓
169
417
389
2,647
1,600,005
I wrote a UNet diffusion model in pure CUDA: github.com/clu0/unet.cu This project was inspired by @karpathy 's llm.c (github.com/karpathy/llm.c). I also learnt a lot about CUDA kernels from @Si_Boehm 's Matmul blog (siboehm.com/articles/22/CUDA…). (1/3)
20
159
11
1,354
270,217
Most of the effort so far was spent getting the whole model to work. There is still a lot of room for optimization. Current per iteration timings on a single RTX 4090: - this repo: 143ms - PyTorch: 66ms - PyTorch with torch.compile: 59ms (2/3)
1
6
1
64
15,546
The main targets for optimization are the forward and backward passes of convolutions, which are currently written in a matmul-like fashion. (3/3)
1
3
63
10,829