Otsar

Introducing BackLite: Attention Backpropagation Acceleration Using Dynamic Sparsity 👀 👊Blog post: moonmath.ai/posts/introducin… 👊Code (open source!) : github.com/moonmath-ai/BackL… 👊Integration example to nanochat: github.com/karpathy/nanochat… It is well known that the attention matrix is highly sparse. Several works have used this sparsity to speed up the forward pass. What if we could also use it to speed up the backward pass? BackLite is a novel algorithm designed to dynamically discover and exploit the sparsity inherent in attention to skip computation while mathematically approximating the gradients through the attention layer. Our idea: Simply track the sparsity in the attention matrix during the forward pass and use it to skip computation during the backward pass. Under the hood: 🌊 Uses the forward pass to track attention matrix tile weights at negligible overhead 🌊 Builds a mask by skipping tiles with cumulative weight less than a threshold 🌊 Skips masked tiles during backward 👉 Same forward, same model, fewer backward FLOPs Drop-in kernel replacement, tested on LLMs and video diffusion models, especially good for long sequence lengths 💪 Disclaimers: Image shows nanochat leaderboard *IF* @karpathy/ @OxyKodit will merge our PR. Yes, there's still much work to do on the code and tests to run. Contributions/questions are welcome.