Tübingen, Deutschland
Joined November 2025
- Tweets36
- Following39
- Followers213
- Likes45
Pinned Tweet
🔥 Frontier labs are redesigning the residual stream to beat the curse of depth: Kimi K3 uses AttnRes, DeepSeek V4 uses mHC, ByteDance proposed HC, and there are more: LNS, KEEL, MoDA…
🙋 But each method was tested with its own training budget, model design and codebase, so the results can't be compared directly. Which one actually works best? More importantly, which one turns architectural depth into effective computational depth?
👇 We built DepthBench to find out.
📄 arxiv.org/abs/2609.32534
really love this illustration comparing HC with mHC w.r.t. computational depth! It confirms our findings about rank collapse in mHC in DepthBench.
It is well known that hyper-connections create richer residual streams but also make training unstable, and mHC performs better by fixing these stability issues.
but HC might be a better choice to improve a model’s depth utilization
my notes on this here: aryagxr.com/blogs/hc-mhc-dep…
Keyu Wang retweeted
This next paper is a bit quirky, but here goes:
Introspection is hard to study in both humans and LLMs. But, we do have internal senses that are verifiable, like proprioception (the sense of where your arm is right now).
What would proprioception look like in language models?
As a digital analogue to the sense of position, a model could verbalize which expert activated in a certain layer for any previous token.
It turns out that LLMs can learn a range of such proprioception skills where they understand and report on their internals. We can calibrate and score these reports, and evaluate whether training for proprioception changes model behaviors.
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!
Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.
🔗 github.com/deepseek-ai/DeepI…
1/4
Keyu Wang retweeted
Great summary of our paper, Depth Memory!
We
1. introduce a framework for existing depth mixing methods and
2. propose a new depth mixer that outperforms AttnRes and mHC.
Check it out! 🚀 @_davidgonmar @tk_rusch @Shiwei_Liu66
"Depth Memory"
Attention Residuals stores earlier Transformer layer outputs for retrieval but this memory grows with depth.
Linear Depth Memory (LDM) maintains a fixed-size memory across layers, using recursive least squares to store and retrieve earlier layer information alongside the standard residual stream.
On 400M and 1.5B models, LDM achieves lower perplexity than standard residuals, mHC, and AttnRes, while maintaining competitive zero-shot accuracy and comparable throughput with optimized kernels.
alphaxiv.org/abs/2610.18809
Keyu Wang retweeted
Cognitive debt may be more dangerous in the AI era.
Introducing TraceVista, a tool for understanding agent trajectories, built around how people naturally make sense of complex work.
Try & examples: tracevista.pages.dev/
Code & skill: github.com/Superone77/TraceV…
Keyu Wang retweeted
should expand it to a "zoo" of pretraining and then a "zoo" of post-training 😆
mHC, iHC, oHC, xHC, and a zoo of AttnRes variants... An LLM architect with FOMO from the daily explosion of new papers? 🫠
Fear not! You can't afford to miss ScalingDepth — our community hub for keeping up with the ever-evolving world of residual connections!
We'll keep tracking and sharing the latest research, methods, and papers from the community 🚀
And it's an OPEN community! Missed something? PRs and DMs are always welcome! Got a cool blog, post, idea, or unexpected finding? We'd love to hear it! ScalingDepth hosts a dedicated space to showcase community insights and discoveries.🌍
Built by the community. Driven by curiosity. Pushing the limits of depth! 🔥
🌐 keyu-wang-2002.github.io/Sca…
Keyu Wang retweeted
A great resource for keeping track of residual connection research. Love the space for community insights alongside the papers!
We’re also preparing the xHC code release and look forward to sharing it with everyone soon! 🚀
Feel free to star the repo for updates! github.com/aHapBean/xHC
And we’ve got more cooking 👀
mHC, iHC, oHC, xHC, and a zoo of AttnRes variants... An LLM architect with FOMO from the daily explosion of new papers? 🫠
Fear not! You can't afford to miss ScalingDepth — our community hub for keeping up with the ever-evolving world of residual connections!
We'll keep tracking and sharing the latest research, methods, and papers from the community 🚀
And it's an OPEN community! Missed something? PRs and DMs are always welcome! Got a cool blog, post, idea, or unexpected finding? We'd love to hear it! ScalingDepth hosts a dedicated space to showcase community insights and discoveries.🌍
Built by the community. Driven by curiosity. Pushing the limits of depth! 🔥
🌐 keyu-wang-2002.github.io/Sca…
Keyu Wang retweeted
Memory Attention paper update!
Added extensive ablation studies exploring MA integration, placement, normalization, and Attnres.
simple idea: V = K + M.
More parameters, yet a lower GPU memory footprint, better performance, and no V cache!
💻 joluck.github.io/memory-atte…
Could this be a direction for future model architectures?
Introducing Memory Attention (MA).
alphaxiv.org/abs/2609.28399
mHC, iHC, oHC, xHC, and a zoo of AttnRes variants... An LLM architect with FOMO from the daily explosion of new papers? 🫠
Fear not! You can't afford to miss ScalingDepth — our community hub for keeping up with the ever-evolving world of residual connections!
We'll keep tracking and sharing the latest research, methods, and papers from the community 🚀
And it's an OPEN community! Missed something? PRs and DMs are always welcome! Got a cool blog, post, idea, or unexpected finding? We'd love to hear it! ScalingDepth hosts a dedicated space to showcase community insights and discoveries.🌍
Built by the community. Driven by curiosity. Pushing the limits of depth! 🔥
🌐 keyu-wang-2002.github.io/Sca…
5/5
💡 Got a crazy idea? An unexpected result? An insight that didn't make it into your paper?
Don't let it die in your notes!
ScalingDepth welcomes blogs, research insights, and fresh perspectives from the community.
Your ideas deserve to be seen! 🔥
Website: keyu-wang-2002.github.io/Sca…
super cool to see that Linear Depth Memory achieves impressive performance on DepthBench @_davidgonmar.
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
Love this sweet illustration!
I wrote a blog post detailing the writing approach I took while working on my recent paper: sophielwang.com/blog/paper-w…
I describe the behind the scenes for how this paper came to be, why and how I storyboarded figures before running experiments, and some figure design tips! :)
[10/11]
So the bigger picture is:
🚀 Depth is an underexplored and promising scaling axis, but only with the correct residual connections, and system-level design are badly needs to fulfill it in practice.
📄 Paper: arxiv.org/pdf/2609.32534
💻 Code: github.com/keyu-wang-2002/De…
🤗 Models: huggingface.co/aspect-ratio-…
[11/11]
DepthBench wouldn't exist without this team ❤️
@yangyi_huang_cn , @Jo1uck , @_davidgonmar, @Besteuler , and @Shiwei_Liu66
