Akos Kadar retweeted
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
Akos Kadar retweeted
How do you tell if authors fabricated their research results? Sometimes: they just tell you. Read, e.g., this caption carefully.
Another banger of a post by my friend Nihar B. Shah on his blog, where he found at least 20+ such #ICLR2027 submissions:
researchonresearch.blog/2026…
Akos Kadar retweeted
Human-in-the-loop (HITL) is widely recommended for oversight & safety, yet current AI agent design requires humans to keep up with unreasonable volume & complexity, promotes fatigue & disengagement, and degrades the very humans skills it relies on. 1/ arxiv.org/abs/2608.23642
Akos Kadar retweeted
Sharing some deeply personal and existential reflections on scientific identity, the changing culture of AI research, the erosion of intellectual autonomy, and the growing disconnect between scientific rigor, professional recognition, and what organizations actually reward.
As usual, feedback and debate are welcome and actively encouraged!
mariya.fyi/posts/stranger-in… – at San Francisco, CA
Akos Kadar retweeted
We trained an LLM with FlashAttention-3 in BF16 🔥 Training looked healthy for a long time. Then the grad norm shot up 1000× 📈 and the loss ended 0.2 nats above FP32 attention. Not a single NaN 😶
The cause is in the attention backward 🧵
📄 arxiv.org/abs/2609.34272
Akos Kadar retweeted
wait what?? faculty in post-LLM AI are farting out 10 phds/year and Richard Sutton had 16 graduating students in his entire career?? this is amazing. i respect him so much more now
The PhD thesis of my 16th PhD student, Fernando Hernandez Garcia, is now available.
Title: Selective Reinitialization Algorithms for Preventing Plasticity Loss in Artificial Neural Networks
Url: incompleteideas.net/papers/F…
Abstract:
In this dissertation, I study systems based on artificial neural networks that learn from nonstationary data. Learning from non-stationary data requires continual adaptation of the system, and is often referred to as continual learning. Developing systems capable of continual learning is a longstanding goal in artificial intelligence. In deep learning, the field concerned with designing and training deep neural networks, this goal remains elusive.
This dissertation addresses a fundamental challenge in continual learning: loss of plasticity, a phenomenon where artificial neural network systems progressively lose their ability to learn from new data. While the phenomenon has been noted several times over the last three decades, it has remained understudied until recently. The work in this dissertation constitutes the first systematic demonstrations of plasticity loss, highlighting its persistence and importance.
In this work, I provide a systematic demonstration of plasticity loss in a wide variety of deep learning systems. Across systems based on fully-connected networks, convolutional networks, residual networks, and vision transformers, plasticity degrades when learning continually. Notably, even systems employing normalization techniques, residual connections, and regularization—design choices that improve training stability—remain susceptible to plasticity loss. This evidence establishes that plasticity loss is pervasive and that deep learning systems trained with backpropagation are not suitable for continual learning.
I explore the idea of selective reinitialization to prevent plasticity loss. This idea has been used in the past for improving generalization performance in deep learning systems. However, its use for preventing plasticity loss is a recent innovation pioneered by the continual backpropagation algorithm. The algorithm periodically sets new values for units in the network, making initialization a continuous process rather than a one-time operation. I demonstrate the effectiveness of continual backpropagation in preventing plasticity loss across a wide variety of settings. These demonstrations establish that plasticity loss, while pervasive, is not inherent to deep learning systems.
I generalize continual backpropagation through an algorithm I call selective unit reinitialization. This general algorithm has three key components: a utility measure that ranks units by importance, a pruning criterion that selects which units to reinitialize, and a reinitialization method that assigns new values to the selected units. Continual backpropagation involves a specific choice of utility measure, pruning criterion, and reinitialization method. However, the general algorithm is not limited to the choices used in continual backpropagation. I present a study of how different choices for the three components of selective unit reinitialization affect its effectiveness at maintaining plasticity. This study establishes selective unit reinitialization as a general approach that can be tailored to each learning system to maintain plasticity.
Finally, I propose a different approach to implementing selective reinitialization that operates at the weight level. I call the corresponding algorithm selective weight reinitialization. Reinitializing at the unit or weight level involves different trade-offs. Unit reinitialization minimally disrupts the network outputs, preserving stability during learning. However, the definition of a unit varies across network architectures, requiring additional engineering to make the approach effective. Weight reinitialization, in contrast, can be readily applied to any arbitrary network architecture, but can substantially affect network outputs, reducing training stability. The reduced learning stability can be remedied with L2 regularization, which stabilizes selective weight reinitialization but introduces an additional hyperparameter. Both selective unit and weight reinitialization successfully maintained plasticity across the systems tested, providing flexible approaches for different architectural and engineering constraints.
Fernando is now a research scientist at Zyphra.
Akos Kadar retweeted
10/ Read the full paper and my deep dive here.
Paper: arxiv.org/abs/2607.05609
Review: arxiviq.substack.com/p/to-re…
Do you agree that we focus too much on avoiding forgetting? Let's discuss.
#MachineLearning #AI
Akos Kadar retweeted
Slides from my ICML tutorial "Is numerical optimization theory irrelevant to machine learning practice in 2026?":
cs.ubc.ca/~schmidtm/Document…
(Updated to fix some typos, incorporate feedback, and add some things I did not have time for. Will link to the video here when posted.)
Akos Kadar retweeted
Is Muon as good as they say? We looked beyond training speed and found a hidden cost: Muon loses the simplicity bias of older optimizers like gradient descent — and this matters for generalization.
Akos Kadar retweeted
Quite enjoyed reading the proofs in this paper.
For anyone who doesn't know, you're no longer using SGD when doing multiple passes over your data. Most papers studying SGD essentially assume infinite amount of data, which isn't always the case in practice. The real algo is RR.
Akos Kadar retweeted
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs.
Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [arxiv.org/abs/2606.19317]
Akos Kadar retweeted
At this point you should not trust any claims that @SakanaAILabs makes. Between the blatantly false claim in this tweet and the series of fraudulent “AI Scientist” papers they put out last year, my default assumption is that the opposite of what they say is true.
Introducing Sakana Fugu: A full multi-agent orchestration system accessible via a single model API.
Our ‘Fugu Ultra’ model matches the performance of Fable and Mythos, delivering frontier capability without the risk of export controls.
Try it: sakana.ai/fugu 🐡
Akos Kadar retweeted
Transformer training often proceeds through sudden changes: new capabilities appear, attention heads specialize, many heads remain redundant.
We build a solvable model explaining staged head specialization through symmetry breaking and signal structure. arxiv.org/abs/2603.03993
Akos Kadar retweeted
Congratulations to Google on open-sourcing Gemma Diffusion!
I want to give a shout-out to a group of really talented Cornell students who developed in the lab a lot of the new ideas that we see in this model:
@mariannearr -- Block diffusion is what enables Gemma Diffusion to generate arbitrary length sequences and support KV caching.
@mariannearr @SchiffYair -- Efficient encoder-decoder diffusion (E2D2) extends block diffusion and is part of what makes Gemma really fast, speeding up inference by running a smaller decoder model.
@SchiffYair @ssahoo_ @Guanghan__Wang -- Uniform diffusion LMs (UDLMs) are the family of discrete diffusion models that underlie Gemma and define its noise process and training objective. This work builds on our earlier simplified losses in MDLMs.
@ssahoo_ -- Uniform diffusion supports built-in error correction and is especially effective with distilled fast samplers like the ones introduced in Duo.
This is a great overview of Gemma Diffusion: newsletter.maartengrootendor…
Check out the students' papers below:
Akos Kadar retweeted
LLMs learn by predicting tokens. World models (JEPA, data2vec) learn by predicting their own abstractions. Which needs more data? For data with hidden hierarchy, we prove the gap is exponential. arxiv.org/pdf/2605.27734
Akos Kadar retweeted
As Nick says, we’re excited about the potential for leveraging cognitive science to improve user simulators, then use them to evaluate and train models that collaborate better with real humans.
I'm hiring (including but not limited to postdocs), let me know if interested in this direction!
New paper! LLM memory keeps improving, but this makes them *worse* as user sims. If we want to build models that can, e.g., simulate realistic students to train chatbots to be better teachers, then these models need to be able to forget like humans do
📄: arxiv.org/abs/2605.25680
Akos Kadar retweeted
Why does deep learning generalize? What does weight decay really do? Can algorithmic information theory address these questions?
In my latest preprint, I give a proof that the minimum neural weight norm matches the minimum program length (aka Kolmogorov Complexity), up to a logarithmic factor. In other words, the neural network with the smallest possible weight norm (that fits the data) must encode the shortest program (that fits the data).
The result only holds for fixed-precision neural nets: infinite precision nets can store infinite information with finite (small) weights.
arxiv.org/abs/2605.10878
Akos Kadar retweeted
Language Models Need Sleep
"Transformer-based large language models are increasingly used for long-horizon tasks; however, their attention mechanism scales poorly with context length. To handle this, we study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights before clearing its key-value cache."
"increasing sleep duration N for our models improves performance, with the largest gains on examples that require deeper reasoning."
Akos Kadar retweeted
general Intelligence requires rethinking exploration
arxiv.org/abs/2211.07819




































