Calc Consulting

and one more subsection, I will add to the paper before I leave for vacation this week Here I will describe two ideas 1. Why the average layer alpha is a good metric, even when the individual layer alphas are off 2. The difference between mode seeking and geometry, seeking and how we can gauge layers this way.

In finalizing the RG paper, I realize that I can connect the spectral flow directly to a a dilation beta function for each individual layer arising directly in BackProp Once I’m sure this is correct. I’ll add it to the paper and submit an updated version. Comments welcome

At least prop 40 failed The communists are pushed back at least a couple years

Check out JevFight A cool game where 2 Jev models fight it out jevfight.com/

I am frequently asked, how wel does weightwatcher work on larger, more complex models like foundation-scale LLMs. Here's an older result from LLama3, where we plot the weightwatcher PL exponent alpha, averaged for each layer type (K, Q, V, ...) You can see that the theory works best for the K and Q matrices inside the Attention blocks We see similar results on other foundation-scale models like Falcon, Qwen, etc, although the results are best for LLama3 We are trying to understand why it works so well here and not so much in other layers.

𝗥𝗲𝗻𝗼𝗿𝗺𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻, 𝗘𝗺𝗲𝗿𝗴𝗲𝗻𝗰𝗲, 𝗮𝗻𝗱 𝗪𝗵𝘆 𝗗𝗲𝗲𝗽 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗪𝗼𝗿𝗸𝘀. Turns out, BackProp looks remarkably like defining a Wilsonian RG beta function for each layer. One of the deepest ideas in theoretical physics is that complex, emergent behavior can arise from simple microscopic rules. Kenneth Wilson's Nobel Prize-winning renormalization group (RG) theory gives us a mathematical framework for understanding how this happens across scales. We've been developing a spectral RG theory of deep learning, motivated by empirical observations of heavy-tailed weight matrices, scale invariance, and the remarkable tendency of well-trained neural networks to approach marginal spectral behavior near α ≈ 2. But here's something surprising. You don't even need to look at the weight spectra to see an intriguing connection to RG. Just look at the basic equations of backpropagation. In Wilsonian RG, a beta function describes how a coupling changes with logarithmic scale: β(g) = dg/d(log b) Now consider a neuron with preactivation z = wx. Backpropagation gives us the exact identity: w ∂L/∂w = z ∂L/∂z = ∂L/∂log|z| That's precisely the mathematical form of a logarithmic scaling derivative. And in a multilayer neural network, something even more interesting happens: → Each layer has its own beta-like logarithmic derivative. → These quantities are coupled across layers through the chain rule. → At stationary points of training, the corresponding dataset-averaged quantities vanish. → The resulting mathematical structure resembles a system of coupled RG beta functions. This is a striking connection hiding in plain sight. Of course, identifying these derivatives does not by itself establish a Wilsonian coarse-graining transformation. But it gives us a concrete mathematical starting point for investigating how coupled layerwise RG flows might emerge during learning. And that brings us back to the bigger question: Can we understand deep learning as a theory of emergence, in which different neural-network layers develop coupled, scale-dependent structures, potentially approaching their own fixed points? Our spectral RG theory approaches this question empirically, through the eigenvalue distributions of trained weight matrices. The backpropagation equations suggest another route toward understanding the underlying dynamics. We're working to bring these two perspectives together. I've put together a thread walking through the mathematics, starting with Wilsonian RG and deriving the backpropagation identities step by step Stay tuned for more

Learn more at weightwatcher.ai weightwatcher.ai/rg_theory.h…

𝗦𝗽𝗲𝗰𝘁𝗿𝗮𝗹 𝗨𝗻𝗶𝘃𝗲𝗿𝘀𝗮𝗹𝗶𝘁𝘆 𝗼𝗳 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 11/11 The spectral RG view may extend from neural networks to XGBoost and other learners approaching fixed points. Spectral universality asks whether different learning rules converge toward similar effective correlation structures. Public arXiv link to follow.

𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗿𝗲𝗰𝗼𝘃𝗲𝗿𝘆 𝗳𝗿𝗼𝗺 𝗼𝘃𝗲𝗿𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 10/11 We also fix an overfit checkpoint. In this preliminary experiment, continuing with WW-PGD recovers more test accuracy than the AdamW and Muon continuations shown.

𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝘀𝗽𝗲𝗰𝘁𝗿𝗮𝗹 𝗰𝗼𝗻𝘁𝗿𝗼𝗹 9/11 On MNIST, adding WW-PGD to AdamW moves layerwise exponents toward 2 while keeping test accuracy close to the baseline across five runs. This tests whether we can steer the spectrum using WW-PGD while preserving predictive performance.

𝗧𝘂𝗿𝗻𝗶𝗻𝗴 𝘁𝗵𝗲 𝘁𝗵𝗲𝗼𝗿𝘆 𝗶𝗻𝘁𝗼 𝗮𝗻 𝗶𝗻𝘁𝗲𝗿𝘃𝗲𝗻𝘁𝗶𝗼𝗻 8/11 This suggests an optimizer intervention: WW-PGD. WW-Projected Gradient Descent After a base optimizer step, select a spectral window, nudge its shape toward α = 2, preserve its trace-log before blending, and mix the correction back into the weights. You can test this now using the Python package: ww-_pgd

𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝘁𝗵𝗲 𝗽𝗿𝗼𝗽𝗼𝘀𝗲𝗱 𝗰𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻 7/11 Trained checkpoints let us test the RG picture directly. Near α = 2, we compare energy across logarithmic bands, the ECS with the fitted power-law tail, and their normalization scales. Together, these measurements turn the theory into a concrete experimental program.