Postdoc at @Harvard with @du_yilun and @gershbrain. PhD with @SchmidhuberAI. Prev: Apple MLR, AWS AI Lab. Same handle on 🦋
Cambridge, MA
Joined January 2018
- Tweets174
- Following505
- Followers763
- Likes1K
Pinned Tweet
Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles "what" (content) and "where" (position) information.
Our fix (PoPE) is simple but powerful. Paper: arxiv.org/abs/2509.10534
Anand Gopalakrishnan retweeted
🚨new method day! 🚨
Trust[ing results] in ML conferences is utterly broken. Let's fix it with an algorithm!
We are excited to introduce 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗪𝗶𝘁𝗻𝗲𝘀𝘀𝗲𝘀, a system for “honesty” certificates of data and evaluations which a verifier can cheaply check and trust.
Anand Gopalakrishnan retweeted
What if a foundation model could tell us how to modify its architecture to boost inference and reasoning instantaneously—without retraining? What if that tweak incurred near zero latency cost during generation and supported indefinite state tracking? arxiv.org/abs/2608.17981
Presenting PoPE today at #ICML2026!
We revisit RoPE through the lens of content-position entanglement, and show how polar coordinates can better decouple content from position.
Come by poster #4012 if you’re curious positional embeddings, pretraining or length generalization.
Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles "what" (content) and "where" (position) information.
Our fix (PoPE) is simple but powerful. Paper: arxiv.org/abs/2509.10534
Interesting work! Their central result that RoPE struggles to distinguish positions and tokens in long contexts is very closely related to our PoPE paper, where we analysed RoPE entangles “what” and “where” in attention and proposed a decoupled alternative.
PoPE: arXiv 2509.10534
Excited to share our new paper: RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
LLMs often fail on inputs well within their advertised context lengths. We show that these failures are not merely engineering issues, but from intrinsic limitations of RoPE in long contexts.
Main finding: In long contexts, RoPE-based attention frequently assigns the same attention weight to a token even when it is moved to different positions. Similarly, it can assign the same attention weight to different tokens at the same position.
In this sense, RoPE attention fails to distinguish both where a token appears and what token appears there — hence the title.
We prove these results theoretically and verify them empirically. While the theoretical analysis focuses on a single attention head, we complement it with experiments on real multi-layer, multi-head LLMs.
The experiments confirm failures predicted by our theory: LLMs optimized for needle-in-a-haystack-style retrieval will inevitably struggle on a very simple task that asks for the k-th item in a list.
My personal takeaway: advertised context lengths should be interpreted with care. Future long-context LMs may require rethinking how position and token order are represented. With current architectures, agentic frameworks that break long contexts into shorter ones may be a more effective way to work around the intrinsic limitations of RoPE.
Paper: arxiv.org/abs/2605.15514
Huge congrats to my student Yufeng Du and others!
Anand Gopalakrishnan retweeted
By the same token, some still don't seem to know that DeltaNet (shorthand for Delta Network) was introduced 5 years ago as a direct descendant of ULTRA (ICML 2021, @ImanolSchlag @SchmidhuberAI ).
Numerous recent papers refer to DeltaNet without citing the original 2021 paper.
Some still don't know: Google's 2017 normalized quadratic Transformer [TR1] is based on the principles of the 1991 unnormalized linear Transformer [ULTRA]. In 1991, KEY/VALUE was called FROM/TO. ULTRA’s computational costs scale linearly in input size, rather than quadratically! The 1993 paper on a recurrent ULTRA extension [FWP2] introduced the attention terminology: learning "internal spotlights of attention" by gradient descent. See the T in ChatGPT! Details and references: people.idsia.ch/~juergen/199…
Anand Gopalakrishnan retweeted
1/?) As promised to Sander Dieleman (@sedielem), we’re finally excited to share:
Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion
We show that continuous diffusion can achieve very strong language modeling performance when operating directly on bitstreams, outperforming masked and uniform diffusion baselines, and essentially matching autoregressive models under our evaluation settings.
Anand Gopalakrishnan retweeted
[1/5] Intelligent behavior relies on maintaining an evolving, dynamic model of the environment. But even frontier models sometimes fail at tracking state in dialogs, e.g., actual transcript from 4/20/2026:
Anand Gopalakrishnan retweeted
Back in 2019, I reduced transformer LM KV-cache size by: (1) setting K=V (storing only K), (2) deeper FF blocks & fewer self-attn layers overall.
Published at ICASSP 2020. To my knowledge, the first publication on KV-cache reduction--lmk if you know anything older!
Anand Gopalakrishnan retweeted
Can we replace RoPE with PoPE (Polar Coordinate Positional Embeddings) in pretrained language models?
Turns out we can! Using small pythia models, after a small recalibration (~2% of pretrain), we get significantly better length generalization.
1/3
xitter.cf/agopal42/status/200390…
Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles "what" (content) and "where" (position) information.
Our fix (PoPE) is simple but powerful. Paper: arxiv.org/abs/2509.10534
Anand Gopalakrishnan retweeted
State of the art World Models still lack a unified world memory for representing and predicting dynamics out of their field of view.
Why is that, and how can we fix it?
Introducing Flow Equivariant World Models: models with memory capable of predicting out of view dynamics!🧵⬇️
Anand Gopalakrishnan retweeted
When you're crossing the street and turn your head, you typically remember whether or not a car is coming from the other direction - so why can't today's world models?
Introducing Flow Equivariant World Models
flowequivariantworldmodels.g…
Led by @hansenlillemark & @huskydogewoof🧵👇
Anand Gopalakrishnan retweeted
Humans can't write programs that classify cats vs dogs. Deep learning lets GD write that program.
Continual learning is the same: it's too hard for us to design good CL algorithms. Let GD write that algorithm too. That’s the idea of metalearning CL algos: arxiv.org/abs/2312.00276
Posted this a day early and the pun practically writes itself. Noooooo!
Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles "what" (content) and "where" (position) information.
Our fix (PoPE) is simple but powerful. Paper: arxiv.org/abs/2509.10534
11/ Key takeaway: The what-where entanglement in RoPE hurts sequence modelling performance and length generalization.
PoPE's disentanglement provides a powerful inductive bias that solves these issues.
Huge thanks to my co-authors @robert_csordas , @SchmidhuberAI, @mc_mozer !
9/ Zero-shot downstream tasks (LAMBADA, BLiMP, CBT, HellaSwag, PIQA, ARC):
PoPE consistently outperforms RoPE across model sizes. At 774M params, we see improvements on every single task.
