@tk_rusch

Faculty @ELLISInst_tue and @MPI_IS leading the Computational Applied Mathematics & AI Lab (CAMAIL) + Scientific Advisor @liquidai || Prev. @MIT_CSAIL + @ETH_en

 Cambridge, MA
camail.org
Joined May 2020
  • Tweets277
  • Following304
  • Followers1.1K
33Media
"Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream."
alphaXiv
@askalphaxivOct 20
"Depth Memory" Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth. Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream. On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels. alphaxiv.org/abs/2610.18809
1
9
609
Konstantin Rusch retweeted
Arber Zela@ZelaArberOct 20
A looped transformer has to know when to stop, and its state has to stay bounded while it loops. We introduce Spherical Fixed-Point Reasoners (SFPRM) 🌐: loop on the unit sphere, where each step is an angle, and stop once the angle is small enough. 🧵 (1/9)
14
56
6
395
21,751
Konstantin Rusch retweeted
alphaXiv
@askalphaxivOct 20
"Depth Memory" Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth. Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream. On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels. alphaxiv.org/abs/2610.18809
10
51
5
403
16,371
Konstantin Rusch retweeted
Philipp Nazari@philna00Oct 20
Great summary of our paper, Depth Memory! We 1. introduce a framework for existing depth mixing methods and 2. propose a new depth mixer that outperforms AttnRes and mHC. Check it out! 🚀 @_davidgonmar @tk_rusch @Shiwei_Liu66
alphaXiv
@askalphaxivOct 20
"Depth Memory" Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth. Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream. On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels. alphaxiv.org/abs/2610.18809
5
15
837
🚨🚀 Once again great summary of our work by @askalphaxiv 👇
alphaXiv
@askalphaxivOct 20
"Depth Memory" Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth. Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream. On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels. alphaxiv.org/abs/2610.18809
1
10
407
Konstantin Rusch retweeted
Vera Milovanović@nevera_mOct 19
I learned a lot from this paper, thanks to the authors! 😍Building on FPRM’s fixed-point approach, QRR introduces rotational symmetry to recognize convergence even while the hidden state keeps moving ---> it stops in fewer iterations!
Sumin Park@psmizizOct 15
Can reasoning finish while latent states keep moving? Happy to share Quotient Recurrent Reasoner (QRR)! We let symmetry-generated rotations persist while task-relevant dynamics converge faster in quotient space, reaching 97.9% on Sudoku-Extreme with 78% fewer loops than FPRM.
1
15
1,100
This is by far the coolest visualization I’ve seen on here in a while!! 🔥👇
Sumin Park@psmizizOct 15
Can reasoning finish while latent states keep moving? Happy to share Quotient Recurrent Reasoner (QRR)! We let symmetry-generated rotations persist while task-relevant dynamics converge faster in quotient space, reaching 97.9% on Sudoku-Extreme with 78% fewer loops than FPRM.
1
7
702
Konstantin Rusch retweeted
Liquid AI
@liquidaiOct 15
Today we release 𝚍𝟷-𝚞𝚕𝚝𝚛𝚊, our most powerful decision model. > built for hard decision problems. > d1-ultra is multimodal > supports 44 languages > #1 on the @ArtificialAnlys Decision Index (65) > leads all eight categories of our 112-evaluation suite 1/
65
186
86
1,616
300,967
Konstantin Rusch retweeted
Sumin Park@psmizizOct 15
Can reasoning finish while latent states keep moving? Happy to share Quotient Recurrent Reasoner (QRR)! We let symmetry-generated rotations persist while task-relevant dynamics converge faster in quotient space, reaching 97.9% on Sudoku-Extreme with 78% fewer loops than FPRM.
3
38
3
300
18,139
Konstantin Rusch retweeted
Sumin Park@psmizizOct 15
Our paper, Looped Reasoning Finishes Earlier Than You Think, is now on arXiv, with code available below! Paper: arxiv.org/abs/2610.18174 Code: github.com/camail-official/Q… With my advisor T. Konstantin Rusch (@tk_rusch). We'd love to hear your thoughts! 6/🧵

Looped Reasoning Finishes Earlier Than You Think

Looped reasoning with dynamic depth can be viewed as relaxation toward solution attractors in a learned dynamical landscape. In practice, however, latent trajectories often oscillate, cycle, or...

arxiv.org
3
19
773
Konstantin Rusch retweeted
Sumin Park@psmizizOct 15
QRR combines faster convergence with stable reasoning performance. With 7M parameters, QRR reaches: 97.9% on Sudoku-Extreme 85.8% on Maze-Hard / 97.0% on Maze-Unique 62.4% Pass@1 on Mini-ARC 5/🧵
1
1
1
6
742
Konstantin Rusch retweeted
Sumin Park@psmizizOct 15
On Sudoku-Extreme, QRR reaches solution roots much sooner than FPRM (median 122 vs 234 lfwd). It also halts the reasoning loop earlier (27 vs 122 recurrent steps), with only 0.1% false halts, compared with 8.9% for FPRM. 4/🧵
1
1
7
571
Konstantin Rusch retweeted
Sumin Park@psmizizOct 15
FPRM and HRM often develop a dominant slow oscillatory mode that delays convergence. QRR's learned quotient dynamics, in contrast, settle earlier without a leading slow-mode bottleneck. Removing the leading mode provides only a small further gain in QRR unlike others. 3/🧵
1
1
5
625
Konstantin Rusch retweeted
Sumin Park@psmizizOct 15
A dominant oscillatory mode can impose its slow decay rate on coupled latent dynamics. We predict its coupling effect from local Jacobian analysis, then attenuate that mode once in FPRM and HRM. The measured responses and settling-time gain match our predictions. 2/🧵
1
1
8
698
Konstantin Rusch retweeted
Sumin Park@psmizizOct 15
FPRM shows alternating flips while HRM follows spirals near solution roots. QRR's raw state can also rotate while its quotient trajectory contracts. The key distinction is whether these oscillations affect the remaining computation, even as raw motion continues. 1/🧵
1
1
17
1,315
Introducing Quotient Recurrent Reasoners (QRR) 🚀 Looped reasoning models (e.g., TRM, FPRM) spend a large amount of loops on oscillatory dynamics that don't contribute to reasoning, wasting a lot of computation before converging. Turns out gauge equivariance helps tremendously by defining convergence in a quotient space instead of state-space, where these oscillations are part of the same equivalence class. Super strong performance on reasoning benchmarks: Sudoku-extreme: 97.9% 🤯 Incredibly work done by my student Sumin Park @psmiziz Check out the fascinating thread below. Also such a cool visualization! 🧵👇 @ELLISInst_Tue @MPI_IS @liquidai
Sumin Park@psmizizOct 15
Can reasoning finish while latent states keep moving? Happy to share Quotient Recurrent Reasoner (QRR)! We let symmetry-generated rotations persist while task-relevant dynamics converge faster in quotient space, reaching 97.9% on Sudoku-Extreme with 78% fewer loops than FPRM.
2
24
4
182
7,750
Konstantin Rusch retweeted
Eric
@Ex0bytOct 15
fixed-size matrix-valued memory is all the rage! code: github.com/davidgonmar/depth… Paper: arxiv.org/pdf/2610.18809
David González-Martínez
@_davidgonmarOct 14
Replying to @_davidgonmar
Joint work with @philna00 (co-first author), @tk_rusch and @Shiwei_Liu66. Paper: arxiv.org/abs/2610.18809 Code: github.com/davidgonmar/depth… Checkpoints: huggingface.co/collections/d…

Depth Memory

Residual connections propagate information through a network by additively accumulating layer outputs. Recently proposed alternatives, such as Manifold-Constrained Hyper-Connections (mHC) and...

arxiv.org
2
33
1,993
Konstantin Rusch retweeted
Songlin Yang
@SonglinYang4Oct 14
MesaNet rotated 90 degree?
David González-Martínez
@_davidgonmarOct 14
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔 Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output. Linear Depth Memory (LDM) equips models with a matrix-valued memory.
1
14
116
7,732
Konstantin Rusch retweeted
Yifan Zhang
@yifanzhang_Oct 14
Very interesting work! Seems closely related to Deep Delta Learning. arxiv.org/abs/2601.00417

Deep Delta Learning

Transformer residual streams are updated by addition. A sufficiently expressive residual block can represent content replacement, but the residual update itself has no operation that reads,...

arxiv.org
David González-Martínez
@_davidgonmarOct 14
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔 Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output. Linear Depth Memory (LDM) equips models with a matrix-valued memory.
2
9
108
7,002
Konstantin Rusch retweeted
Neehal Tumma
@ntumm120Oct 14
Super interesting work applying test-time regression along the depth axis
David González-Martínez
@_davidgonmarOct 14
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔 Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output. Linear Depth Memory (LDM) equips models with a matrix-valued memory.
2
1
14
716
Load more