@Fraccagnetta

Theoretical physicist (bow tie included), inherently out of equilibrium. Studying data structure and deep learning at https://xitter.cf/t.co/zyWe5FLbRg

 Trieste, Italy
Joined May 2022
  • Tweets121
  • Following535
  • Followers571
14Media
Whatever you make of already-clean.goodfire.com/, it has brought compositionality and generalisation back into the spotlight. I’d like to see more rigorous work testing these ideas. The Random Hierarchy Model is one example. By controlling the data's compositional structure, we recover several key features of deep learning: advantage of depth, scaling laws, linear representations and circuits. Well-motivated toy models are invaluable: they help us explain how these phenomena fit together and make falsifiable predictions about real data. Fun fact: @nickcammarata and colleagues’ earlier “Zoom In” work inspired the RHM, back when we were still thinking about cats and dogs.
2
15
123
4,017
Model: journals.aps.org/prx/abstrac… Connection to real data via diffusion: pnas.org/doi/10.1073/pnas.24…, openreview.net/pdf?id=0GzqVq… Language modelling and scaling laws: openreview.net/pdf?id=NaCXcU…, journals.aps.org/pre/abstrac… Creativity: openreview.net/pdf?id=1OUEnf… Parsing: openreview.net/pdf?id=mJgkPA… Provable depth separation: arxiv.org/pdf/2601.19756 Latent world models: arxiv.org/abs/2605.27734 Platonic representation Hyp: arxiv.org/pdf/2610.07168

How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model

A hierarchical model of high-dimensional data reveals how deep neural networks leverage their multiple layers to reduce the data dimensionality and learn from a finite set of examples.

journals.aps.org
1
13
479
Francesco Cagnetta retweeted
Surya Ganguli
@SuryaGanguliSep 23
We are having our first @SimonsFdn yearly meeting on the Physics of Learning and Neural Computation Sept 24th-25th and all are welcome to join on this zoom: simonsfoundation.zoom.us/j/9… We have an excellent set of speakers. Here is the agenda: simonsfoundation.org/event/s… Join our webinar starting Sept 24th 9:30am ET to learn how the structure of data, architectures, and learning dynamics conspire to empower AI to learn, scale, compute, imagine, reason and collaborate! See physicsoflearning.org/ for more about our collaboration.
2
58
11
286
42,473
Finally someone said it
Vlado Boza
@bozavladoSep 20
I don't think we should attribute 60k papers submitted to ICLR to AI. It's a natural evolution of bad incentives, and AI is the tool that fulfills them with minimal effort. When administration and funding agencies count the number of published papers, everybody figures out the "minimal publishable unit" and splits one solid paper into four crappy ones. When top AI professors demand NeurIPS publication from applying master's students, each student tries to win the publication lottery with something small and insignificant. Many papers I read in the past year could have been a tweet with an attached GitHub repository of experiments. We got 15+ pages of write-ups about one-line changes (think something like DyT). Or a brutal mumbo-jumbo mixture of techniques to get improvement in the 3rd decimal place (in many disciplines this is called p-hacking, but we kind of ignore that). Don't blame AI, blame the system.
17
3,227
Keep an eye on this one!
Adam Shai
@adamimosSep 9
Today, we are launching Belief Updates, the Simplex research blog, with two posts. Read the welcome post here: simplex.pub/welcome/ And see our post about nonergodicy in the quote. We are starting Belief Updates for a few reasons: 🧵👇

Welcome to Belief Updates

Launch note for Belief Updates, the Simplex research blog.

simplex.pub
10
677
Francesco Cagnetta retweeted
Chris J. Maddison
@cjmaddisonSep 2
During my career, the value of ideas has collapse to nearly nothing, while the value of careful experiments has gone way up.
9
15
2
250
17,173
📢 2nd Workshop on Principles of Generative Modelling (PriGM) at NeurIPS 2026 Paris: sites.google.com/view/prigmn… Deadline: September 5th, AoE

PriGM@NeurIPS2026

Workshop summary Machine learning theory has long focused on classical supervised learning settings, where a model is trained on input–label pairs drawn from a well-defined data distribution, with...

sites.google.com
4
9
710
Francesco Cagnetta retweeted
Elisabetta Cornacchia@Elisabetta68885Aug 5
📢Workshop on Principles of Generative Modeling (PriGM) at NeurIPS 2026 Paris: sites.google.com/view/prigmn… Submission Deadline: September 5th, AoE
1
4
6
651
Francesco Cagnetta retweeted
Eric J. Michaud
@ericjmichaud_Jul 24
I gave a lightning talk at the mechanistic interpretability workshop at ICML this year. Thanks to the organizers, who encouraged us to share our high-level takes; my talk ended up being quite high-level and personal indeed. Blog post version: ericjmichaud.com/interp-purp…
7
37
6
303
32,492
Francesco Cagnetta retweeted
Surya Ganguli
@SuryaGanguliJul 23
Our new paper brilliantly lead by @itamarlandau "A predictive theory of experimental design for inferring neural population geometry in large-scale recordings." biorxiv.org/content/10.64898… Our theory can say a lot about the geometry of new data *before* it is collected, by extrapolating from past data. We quantitatively predict: 1) How neural dimensionality, and the reliability of neural correlations and individual neural PCA modes grows with neurons and trials. This allows one to design experiments before the data is collected. 2) We find a "blessing of dimensionality" whereby recording more neurons actually allows us to record *fewer* trials while still reliably inferring neural population geometry. This opens the door to new types of experiments with many more complex trial types. 3) We develop scaling laws for neural prediction using masked autoencoders - a key technology for building foundation models in neuroscience. In the simple setting of linear autoencoders on a single session, we find power law behavior of prediction performance and autoencoder size with neurons and trials, and we trace these power law exponents for neural prediction to power law signals in collective neural modes of the brain. 4) We test our theory across multiple species (mouse, monkey, human) and recoding modalities (electrophysiology, calcium imaging, and fMRI). See @itamarlandau's excellent thread for more information: xitter.cf/itamarlandau/status/20… Also, yet another fun collaboration with Mark Schnitzer!

A predictive theory of experimental design for inferring neural population geometry in large-scale...

Ongoing technological advances will lead to recordings with progressively increasing numbers of neurons, while trial counts may only increase modestly. The analysis of such large-scale data increas...

biorxiv.org
Itamar D Landau@itamarlandauJul 23
Neuro experiments capture 10,000+ neurons but often with only a ~100s of trials. So the data is ~100s points in a 10,000-D space. Can we trust the extracted "population geometry"? This motivates our new paper: a predictive theory of experimental design. @SuryaGanguli 1/13
1
30
1
223
21,956
Francesco Cagnetta retweeted
Antonio Orvieto
@orvieto_antonioJul 18
lots of people asking me "oh did @Kimi_Moonshot K3 actually give you a proof?". This is not the point. very rarely an actual research question (at least not in pure maths) looks like an Erdos problem. it is more like a feeling that something must be there. figuring out a method of attack and the right methodology to make the vibe quantitative, that's the real magic. Once my question looks like an Erdos problem, 95% of the hard stuff is done.
6
100
7,661
Francesco Cagnetta retweeted
cova
@covacutJul 13
how does AI think? ✨
27
30
2
334
18,149
Tonight @icmlconf: predict LLM scaling without even training. #ICML2026 We derive the data-limited scaling exponent from measurable language statistics (token correlations and next-token entropy) and test it across models. 📅 Thu Jul 9, 5:00 PM, #1404
1
6
44
3,120
Important update since the first arXiv, prompted by questions from @yoavgo @BlackHC: could the entropy exponent be a Transformer artefact? We remeasured it with Mamba and n-gram models and found the same arxiv.org/pdf/2602.07488v3. Strong evidence that it is a property of the data.
1
2
10
377
Francesco Cagnetta retweeted
Yasaman Bahri
@yasamanbbJul 8
Our work is appearing in ICML '26, accepted as a Spotlight paper! Come by our poster on Thu July 9 in the evening session - my co-authors will be presenting: icml.cc/virtual/2026/poster/…
Yasaman Bahri
@yasamanbbFeb 18
In our new preprint, we explain how some salient features of representational geometry in language modeling originate from a single principle - translation symmetry in the statistics of data. arxiv.org/abs/2602.15029v1 With @dhruvakarkada, @DanKorchinski, Andres Nava, and @MatthieuWyart.

Symmetry in language statistics shapes the geometry of model...

Although learned representations underlie neural networks' success, their fundamental properties remain poorly understood. A striking example is the emergence of simple geometric structures in LLM...

arxiv.org
5
29
3,618
Francesco Cagnetta retweeted
Surya Ganguli
@SuryaGanguliJul 5
Check out all the amazing work from our @SimonsFdn Collaboration on the Physics of Learning and Neural Computation (physicsoflearning.org/) presented at the main meeting of @ICMLconf #ICML2026 Tuesday Efficient Learning of Compositional Targets with Hierarchical Spectral Methods,Hugo Tabanelli, Yatin Dandi, Luca Pesce, and Florent Krzakala icml.cc/virtual/2026/poster/… CompleteP for RL: Maintaining Feature Learning When Scaling Deep Reinforcement Learning M Ganesh Kumar, Adam Lee, Blake Bordelon , Cengiz Pehlevan icml.cc/virtual/2026/poster/… Universal One-third Time Scaling in Learning Peaked Distributions Yizhou Liu, Ziming Liu, Cengiz Pehlevan, Jeff Gore icml.cc/virtual/2026/poster/… Wednesday A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Models, Leonardo Defilippis, Florent Krzakala, Bruno Loureiro, Antoine Maillard icml.cc/virtual/2026/poster/… Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws Fabrizio Boncoraglio, Vittorio Erba, Emanuele Troiani, Yizhou Xu, Florent Krzakala, Lenka Zdeborová icml.cc/virtual/2026/poster/… A Solvable High-Dimensional Model Where Nonlinear Autoencoders Learn Structure Invisible to PCA While Test Loss Misaligns With Generalization Vicente Mendes, Lorenzo Bardone, Cédric Koller, Jorge Medina Moreira, Vittorio Erba ⋅ Emanuele Troiani, Lenka Zdeborova icml.cc/virtual/2026/poster/… Deep networks learn to parse uniform-depth context-free languages from local statistics Jack T. Parley, Francesco Cagnetta, Matthieu Wyart icml.cc/virtual/2026/poster/… Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling Indranil Halder, Cengiz Pehlevan icml.cc/virtual/2026/poster/… On the Existence of Consistent Adversarial Attacks in High-Dimensional Linear Classification Matteo Vilucchio, Lenka Zdeborova, Bruno Loureiro icml.cc/virtual/2026/poster/… Robust Stochastic Gradient Posterior Sampling with Lattice Based Discretisation Zier Mensch, Lars Holdijk, Samuel Duffield, Maxwell Aifer, Patrick Coles, Max Welling, Miranda C. N. Cheng icml.cc/virtual/2026/poster/… Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability Shobhita Sundaram, John Quan, Ariel Kwiatkowski, Kartik Ahuja, Yann Ollivier, Julia Kempe icml.cc/virtual/2026/poster/… Thursday Deriving Neural Scaling Laws from the Statistics of Natural Language Francesco Cagnetta ⋅ Allan Raventos ⋅ Surya Ganguli ⋅ Matthieu Wyart icml.cc/virtual/2026/poster/… Symmetry in language statistics shapes the geometry of model representations Dhruva Karkada, Daniel Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri icml.cc/virtual/2026/poster/… A Random Matrix Perspective on the Consistency of Diffusion Models Binxu Wang, Jacob A Zavatone-Veth, Cengiz Pehlevan icml.cc/virtual/2026/oral/71… Hyperparameter Transfer with Mixture-of-Expert Layers Tianze Jiang, Blake Bordelon, Cengiz Pehlevan, Boris Hanin icml.cc/virtual/2026/poster/… Analytic Bijections for Smooth and Interpretable Normalizing Flows Mathis Gerdes, Miranda C. N. Cheng icml.cc/virtual/2026/poster/… Efficient RL Training for LLMs with Experience Replay Charles Arnal, Vivien Cabannnes, Taco Cohen, Julia Kempe, Remi Munos icml.cc/virtual/2026/poster/… Embedding Trust: Semantic Isotropy Predicts Nonfactuality in Long-Form Text Generation Dhrupad Bhardwaj, Julia Kempe, Tim G. J. Rudner icml.cc/virtual/2026/poster/… What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoT Yunzhen Feng, Julia Kempe, Cheng Zhang, Parag Jain, Anthony Hartshorn icml.cc/virtual/2026/poster/… From Kepler to Newton: Inductive Biases Guide Learned World Models in Transformers Ziming Liu, Surya Ganguli, Andreas Tolias icml.cc/virtual/2026/poster/…

Home | Physics Of Learning

physicsoflearning.org
32
3
140
27,993
At #ICML2026 this week, with a personal update: I’ve joined simplexaisafety.com/. I’ll keep pushing towards a science of intelligence: how LLMs acquire the latent structure of the world, and what this implies for representations and scaling. I’m especially excited to do this with a team that has thought so deeply about these questions. If you’re around and want to chat, come say hello!
1
1
55
1,953
Francesco Cagnetta retweeted
Alessandro Favero@alesfavMay 29
AI needs vastly more data than we do. One idea might close the gap: don't predict raw signals (tokens), predict your own abstract latent representation (JEPA, data2vec). With @DanKorchinski @MatthieuWyart, on a toy model, we prove how much that helps: the gap is exponential. 🧵
14
76
7
525
53,841
Francesco Cagnetta retweeted
Surya Ganguli
@SuryaGanguliMay 22
My new article "Toward a science of intelligence: unifying physics, neuroscience and AI" amacad.org/publication/daeda… published in the Daedelus journal of @americanacad Its part of a special issue on AI+Science with many amazing contributors lead by James Manyika amacad.org/daedalus/ai-scien…

Toward a Science of Intelligence: Unifying Physics, Neuroscience & AI

Artificial intelligence stands poised to transform our society, yet we hardly understand how it works. A synthesis of physics, neuroscience, and AI can fulfill an urgent need: to build a new, unified...

amacad.org
12
72
4
351
43,774
Load more