Calc Consulting

and one more subsection, I will add to the paper before I leave for vacation this week Here I will describe two ideas 1. Why the average layer alpha is a good metric, even when the individual layer alphas are off 2. The difference between mode seeking and geometry, seeking and how we can gauge layers this way.

In finalizing the RG paper, I realize that I can connect the spectral flow directly to a a dilation beta function for each individual layer arising directly in BackProp Once I’m sure this is correct. I’ll add it to the paper and submit an updated version. Comments welcome

𝗔 𝗦𝗽𝗲𝗰𝘁𝗿𝗮𝗹 𝗥𝗲𝗻𝗼𝗿𝗺𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗚𝗿𝗼𝘂𝗽 𝗧𝗵𝗲𝗼𝗿𝘆 𝗼𝗳 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 Yeah, it’s taken me almost a decade of work to get here! My new paper is finally live on arXiv. This has been a labor of love, beginning with research at UC Berkeley and continuing independently with wonderful collaborators. I spent much of this summer writing it up with substantial help from ChatGPT. It’s my first strongly AI-enabled research paper. I believe we’re on the verge of a scientific revolution in AI. Yet we still don’t have a satisfactory theory explaining WHY deep learning works. Why do remarkable capabilities emerge? Why do models generalize or overfit? Why do optimizers work? How does learning organize correlations? 𝗪𝗵𝗮𝘁 𝘁𝗵𝗲 𝗽𝗮𝗽𝗲𝗿 𝗽𝗿𝗼𝗽𝗼𝘀𝗲𝘀 We bring together Heavy-Tailed Self-Regularization (HTSR), our Semi-Empirical Theory of Learning (SETOL), and optimization research into a proposed Renormalization Group (RG) framework. 𝗦𝗽𝗲𝗰𝘁𝗿𝗮𝗹 𝗥𝗚 𝗳𝗹𝗼𝘄𝘀: Truncation, rescaling, and changes of measure defining effective flows and candidate fixed points. 𝗠𝗮𝗿𝗴𝗶𝗻𝗮𝗹𝗶𝘁𝘆 𝗮𝘁 α ≈ 𝟮: Connecting scale invariance, the self-averaging boundary, and learned correlations. 𝗚𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Understanding model quality, overfitting, and correlation traps through spectral statistics. 𝗪𝗪-𝗣𝗚𝗗: Using these ideas to develop better optimizers. A separate paper is coming soon! Our broader hypothesis is: 𝗦𝗽𝗲𝗰𝘁𝗿𝗮𝗹 𝗨𝗻𝗶𝘃𝗲𝗿𝘀𝗮𝗹𝗶𝘁𝘆 𝗼𝗳 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴. 𝗔 𝗣𝗲𝗿𝘀𝗼𝗻𝗮𝗹 𝗡𝗼𝘁𝗲: This is also an homage to my late PhD advisor, Karl Freed, a world expert in RG, and my undergraduate physics professor Kenneth Wilson, whose pioneering RG work earned him the Nobel Prize. I tried to bring everything I learned from that old-school scientific education to bear on modern AI. A huge thanks to Christopher Hinrichs for helping develop SETOL; Mirco Milletari a for helping develop the early ideas; Hari Prakash for his indispensable contributions to WW-PGD and our understanding of Correlation Traps , and Serena Peng for her work on early HTSR research. I couldn’t have done this without them! Ultimately, I developed WeightWatcher and its theory to help clients train, fine-tune, and evaluate better AI models. WeightWatcher has now exceeded 500,000 downloads. This is a first draft of a proposal, not the final word. I welcome feedback, positive or negative, and hope you find it useful!

Some context may help here This idea came to me when writing 5.6 of the more recent paper weightwatcher.ai/rg_theory_d… where I was considering how to explain the choice of normalization on the ERG Trace-Log condition in terms of a layer-wide rescaling of each layer

The book of diagrams hundreds of open shell Feynman diagrams, all handwritten out A souvenir from ancient times

This is my ChatGPT history on the BS you complained about. I asked ChatGPT to clean up the text and format it in Unicode so the equations would be readable Seriously, you are a prick

🖕 arxiv.org/abs/2507.17912

🖕 arxiv.org/abs/2507.17912

I am frequently asked, how wel does weightwatcher work on larger, more complex models like foundation-scale LLMs. Here's an older result from LLama3, where we plot the weightwatcher PL exponent alpha, averaged for each layer type (K, Q, V, ...) You can see that the theory works best for the K and Q matrices inside the Attention blocks We see similar results on other foundation-scale models like Falcon, Qwen, etc, although the results are best for LLama3 We are trying to understand why it works so well here and not so much in other layers.