Very excited by this year's Analytical Connectionism Summer School! A dream lineup of speakers on the topic of language acquisition in minds and machines Bursaries available to cover costs Aug 17 – Aug 28, 2026 Gothenburg Details: analytical-connectionism.net…

Finally, we demonstrate gradient descent sometimes naturally evolves along the connecting paths between saddles iteratively, yielding saddle-to-saddle dynamics. We identify two distinct mechanisms: timescale separation between directions or units, depending on the architecture.

We then show that saddles are connected by gradient descent paths (invariant manifolds). Along these paths, a larger network behaves like a smaller one, retaining the same simplicity during a saddle-to-saddle transition.

We first show that saddle points are ubiquitous in the loss landscape: fixed points of smaller networks can be embedded as saddle points of larger networks, yielding a nested hierarchy of saddles. These saddles exist in any network that contains a sum of repeated units.

Why don’t neural networks learn all at once, but instead progress from simple to complex solutions? And what does “simple” even mean across different neural network architectures? Sharing our new paper @iclr_conf led by Yedi Zhang with Peter Latham arxiv.org/abs/2512.20607

Not all pairs of representations which can merge are actually expected to merge. Thus, the final learned automaton may look different for different training runs, even when in practice they implement an identical algorithm. (10/11)

The theory predicts mergers can only occur given enough training data and small enough initial weights, resulting in a phase transition between an overfitting regime and an algorithm-learning regime. (9/11)

Since these pairs share outputs, mergers do not affect the automaton's computation. With enough mergers, the automaton becomes finite, fixing its behavior for long sequences. If the training data uniquely specifies the task, this results in full generalization. (8/11)

Using intuitions based on continuity, we derive local interactions between pairs of representations. We find that pairs of sequences which always agree on target outputs after receiving any possible additional symbols will merge representations under certain conditions. (7/11)

We find two phases: -An initial phase where the RNN builds an infinite tree and fits it to the training data, reducing only the training loss. -A second merging phase, where representations merge until the automaton becomes finite, with a sudden drop in validation loss. (6/11)

To understand what is happening in the RNN, we extract automata from its hidden representations during training, which visualize the computational algorithm as it is being developed. (5/11)