11/13 To learn new tasks safely, our transfer method (DEFT) finds redundant Core neurons and merges them. This frees up "empty" capacity that gets reclaimed by the plastic Slack! ♻️ We then apply an architectural mask that cuts any connections pointing from the adapting Slack to the frozen Core.

10/13 For that, we partition the network using the Hilbert norm. Neurons with a relatively large norm are frozen to protect foundational knowledge (the "Core"), and the remainder are treated as a highly plastic "Slack".

9/13 HOPE progressively shrinks the network by picking the action with the lowest projection error. We even scaled this up to pruning a block of multiple layers (residual blocks) using the same geometric principles. But this goes beyond compression! We can use this for Continual and Transfer Learning.

7/13 Usually, you would approximate the integral using finite data by passing massive datasets through the network. But HOPE evaluates it analytically in closed form! Since neurons perceive inputs as 1D Gaussians (by the Central Limit Theorem), we can build an exact continuous surrogate using BatchNorm stats.

6/13 So, a single neuron gives a rank-1 operator, and a pair gives a rank-2 operator. Under this view, the true capacity of a neuron is simply its Hilbert norm! But how do we evaluate that integral associated with the inner product?

5/13 These neural functions form a Hilbert space, which is a complete inner product space. The inner product is defined as the integral over the input distribution. If we stack these vectors side by side, we get operators, akin to matrices!

3/13 Also context and data distribution matter a ton too. Two neurons might have the exact same weight norm, but one could be "dead" if the data distribution only ever pushes it into the negative zone of its ReLU. We really need a better metric.

Excited to share about what I’ve been working on over the past year: quantifying the capacity of a neuron😅 This led to a mathematical framework we call HOPE, which lets us rigorously deconstruct what deep networks might have learned. Paper: arxiv.org/abs/2607.21366

The end of an era. Dimitri Bertsekas, a pioneer of mathematical optimization, has passed away. So many of us relied on his textbooks to get through graduate school. His work shaped the minds of a generation.

An exciting moment for AI in math! A new paper by a team of mathematicians, including Terence Tao tackled 67 pure math problems and in several cases found solutions that improved on the best-known human results. arxiv.org/abs/2511.02864