Pinned Tweet
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
Can a language model be based purely on self-conditioning? 🤔
Yes! Meet LAW 👋 (a.k.a. Loopholing All the Way!)
🧵1/8
David González-Martínez retweeted
🚨 New paper!
What if LLMs could reason like we do when solving Sudoku: keep a few candidates per cell and iteratively narrow them down?
Introducing Binary Discrete Diffusion for language and reasoning tasks: it denoises sets of candidate tokens using binary representations! 🧵
David González-Martínez retweeted
🚨🚀 Once again great summary of our work by @askalphaxiv
👇
"Depth Memory"
Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth.
Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream.
On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels.
alphaxiv.org/abs/2610.18809
David González-Martínez retweeted
What if information passing between Transformer layers were treated as memory reads and writes? "Depth Memory" proposes LDM, a fixed-size memory updated via recursive least squares, and reports lower perplexity than PreLN, mHC and AttnRes at 400M and 1.5B scale (authors' experiments).
arxiv.org/abs/2610.18809
David González-Martínez retweeted
This next paper is a bit quirky, but here goes:
Introspection is hard to study in both humans and LLMs. But, we do have internal senses that are verifiable, like proprioception (the sense of where your arm is right now).
What would proprioception look like in language models?
As a digital analogue to the sense of position, a model could verbalize which expert activated in a certain layer for any previous token.
It turns out that LLMs can learn a range of such proprioception skills where they understand and report on their internals. We can calibrate and score these reports, and evaluate whether training for proprioception changes model behaviors.
David González-Martínez retweeted
Great summary of our paper, Depth Memory!
We
1. introduce a framework for existing depth mixing methods and
2. propose a new depth mixer that outperforms AttnRes and mHC.
Check it out! 🚀 @_davidgonmar @tk_rusch @Shiwei_Liu66
"Depth Memory"
Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth.
Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream.
On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels.
alphaxiv.org/abs/2610.18809
David González-Martínez retweeted
"Depth Memory"
Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth.
Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream.
On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels.
alphaxiv.org/abs/2610.18809
David González-Martínez retweeted
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
Can reasoning finish while latent states keep moving?
Happy to share Quotient Recurrent Reasoner (QRR)!
We let symmetry-generated rotations persist while task-relevant dynamics converge faster in quotient space, reaching 97.9% on Sudoku-Extreme with 78% fewer loops than FPRM.
David González-Martínez retweeted
MesaNet rotated 90 degree?
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
delta rule rotated by 90°
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
We’re bringing natural-language control to any scientific generator.
We are very excited to introduce LLM-to-Flow Distillation (LFD), a method to adapt scientific generators to high-level natural-language discovery goals.
Starting from textual goals like “design peptides with promising anticancer activity” or “design small molecules binding to the D2 dopamine receptor”, LFD distills high-level goal-related knowledge from LLMs into scientific generators through iterative DPO on LLM-generated sub-goals.
🔥 LFD does not need task-specific proxy rewards, works with any frontier LLM via API-access, and outperforms state-of-the-art reward-guided methods such as FlowGRPO and DiffusionNFT on held-out computational evaluations. (1/6)
David González-Martínez retweeted
mHC, iHC, oHC, xHC, and a zoo of AttnRes variants... An LLM architect with FOMO from the daily explosion of new papers? 🫠
Fear not! You can't afford to miss ScalingDepth — our community hub for keeping up with the ever-evolving world of residual connections!
We'll keep tracking and sharing the latest research, methods, and papers from the community 🚀
And it's an OPEN community! Missed something? PRs and DMs are always welcome! Got a cool blog, post, idea, or unexpected finding? We'd love to hear it! ScalingDepth hosts a dedicated space to showcase community insights and discoveries.🌍
Built by the community. Driven by curiosity. Pushing the limits of depth! 🔥
🌐 keyu-wang-2002.github.io/Sca…
David González-Martínez retweeted
Depth Memory augments residual streams with compressed, queryable memory, outperforming PreLN, mHC, and AttnRes on perplexity at 400M and 1.5B parameters. Check out the paper! 😍
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
🥇 LDM beats AttnRes and mHC on DepthBench, and it’s the method that best turns extra depth into lower loss.
David González-Martínez retweeted
Introducing Depth Memory 🧠
One framework that unifies residual methods like mHC and AttnRes. They turn out to be special cases of a single view: how layers write to and read from a memory across depth.
Our new depth mixer, Linear Depth Memory (LDM), consistently beats both at minimal overhead. 👇
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
Unifying attention residuals and mHC by introducing a memory stream along depth 🤯 can be instantiated with a linear attention mechanism. And it’s fast!
Such an exciting project led by our students @_davidgonmar and @philna00
👇
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
Our new preprint Depth Memory is finally live!
We propose a framework that unifies residual methods such as mHC and AttnRes and propose Linear Depth Memory, a novel depth mixer. Check it out!
Interactive blog post: phnazari.github.io/blog/2026…
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
Great paper, check ti out:
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
