Research Scientist, Google Brain now DeepMind. Training neural nets since 1979.
San Francisco, CA
Joined January 2022
- Tweets42
- Following211
- Followers1.7K
- Likes56
Michael C. Mozer retweeted
What if an agent could decide how to structure its own computation?
Agentic Meta-Reasoning: Let the model decide what to explore, what to build on, what to verify, and where to spend its next unit of compute: effectively constructing its own computational graph as it reasons. 🧵
Paras Dahal, @anton_bakhtin , @TacoCohen , Zhengxing Chen, Carole-Jean Wu, Rob Fergus, Scott Yih, @syhw , @rsalakhu , @prfsanjeevarora , @jaseweston
Michael C. Mozer retweeted
💥I have been able to independently verify @mc_mozer and Google team's work on ♾ Recirculation👀 using Llama 3.2 1B model on a M4 MacBook.
😱 +16.95% improvement on GSM8K-Platnium with zero weight changes.
Reproduction code with optimized mlx prefill kernel. 👇
Michael C. Mozer retweeted
Interesting work! Their central result that RoPE struggles to distinguish positions and tokens in long contexts is very closely related to our PoPE paper, where we analysed RoPE entangles “what” and “where” in attention and proposed a decoupled alternative.
PoPE: arXiv 2509.10534
Excited to share our new paper: RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
LLMs often fail on inputs well within their advertised context lengths. We show that these failures are not merely engineering issues, but from intrinsic limitations of RoPE in long contexts.
Main finding: In long contexts, RoPE-based attention frequently assigns the same attention weight to a token even when it is moved to different positions. Similarly, it can assign the same attention weight to different tokens at the same position.
In this sense, RoPE attention fails to distinguish both where a token appears and what token appears there — hence the title.
We prove these results theoretically and verify them empirically. While the theoretical analysis focuses on a single attention head, we complement it with experiments on real multi-layer, multi-head LLMs.
The experiments confirm failures predicted by our theory: LLMs optimized for needle-in-a-haystack-style retrieval will inevitably struggle on a very simple task that asks for the k-th item in a list.
My personal takeaway: advertised context lengths should be interpreted with care. Future long-context LMs may require rethinking how position and token order are represented. With current architectures, agentic frameworks that break long contexts into shorter ones may be a more effective way to work around the intrinsic limitations of RoPE.
Paper: arxiv.org/abs/2605.15514
Huge congrats to my student Yufeng Du and others!
Michael C. Mozer retweeted
New preprint: The Self Requires Learning. Self-consciousness requires continual learning + world-modeling. I introduce "bounded integration" to connect perspective, identity, and self-representation — and diagnose what current AI systems have and lack.
Michael C. Mozer retweeted
Posted this a day early and the pun practically writes itself. Noooooo!
Our new paper shows that RoPE—the positional encoding used in most modern LLMs like Qwen, Gemma, DeepSeek—has a fundamental flaw: it entangles "what" (content) and "where" (position) information.
Our fix (PoPE) is simple but powerful. Paper: arxiv.org/abs/2509.10534
Michael C. Mozer retweeted
Happy to announce that our work has been accepted to workshops on Multi-turn Interactions and Embodied World Models at #NeurIPS2025! Frontier foundation models are incredible, but how well can they explore in interactive environments?
Paper👇
arxiv.org/abs/2412.06438
🧵1/13
Michael C. Mozer retweeted
🌟To appear in the MechInterp Workshop @ #NeurIPS2025 🌟
Paper: arxiv.org/abs/2509.04466
How do language models (LMs) form representation of new tasks, during in-context learning? We study different types of task representations, and find that they evolve in distinct ways.
🧵1/7
Michael C. Mozer retweeted
[📜9/9] Check out our paper for more details.
Paper: arxiv.org/abs/2505.22310
Code: github.com/shoaibahmed/visio…
Michael C. Mozer retweeted
[📜1/9] Does machine unlearning truly erase data influence? Our new paper reveals a critical insight: 'forgotten' information often isn't gone—it's merely dormant, and easily recovered by fine-tuning on just the retain set.
Michael C. Mozer retweeted
We are announcing the launch of Airial Travel’s open-to-all beta version for desktop today. Airial is your personal travel agent with AI superpowers which makes planning and booking trips as easy as dreaming them up. airial.travel
Me and Sanjeev co-founded Airial Travel a year ago to solve a problem we faced repeatedly. Being avid travelers living in the US with our families in India, we were traveling for several months a year and spending multiple days planning and booking each trip. Hours and hours of research, browsing, watching videos, form-filling, spreadsheets, refinements etc. At the end of the process in most cases, we just booked because we were exhausted and wanted to get it over with. As we talked to people and read up about this, we realized that the scale and the intensity of this problem is stunning - hundreds of millions of trips are booked online every year and planning each of them takes over five hours on average.
Our vision “Just imagine your trip, and Airial it!” stems from our ultimate wish as travelers - AI that can figure out all the intricacies of trip planning for you - hotels, activities, flights, trains, transits, deals, date options, restaurants, interests, research, travel videos and everything else.
Our defining features originate from the core beliefs that Airial is built on:
📅 Detailed intricately crafted plans: Attention to detail makes trip plans incredible. Airial plans trips in a level of detail that is simply unmatched, taking care of hundreds of common sense constraints across thousands of variables in seconds.
🚀 From Reels to Itineraries: TikTok / IG reels to Trips is work that millions do manually. We now enable all this in a click. This is the intersection of the two big trends in travel - AI and Socials.
🏖️ Personalized Planning: Travel portals today are one-size-fits-all. We plan trips tailored to your specific interests - architecture tours, scenic hikes or samurai sword fighting lessons!
👆 Actionable Itineraries: Just having a chatbot pick out one combination for you isn’t practically useful. Which was the last trip you planned that didn’t need any refinement? Every decision Airial makes for you is changeable via chat or UI controls.
🌎 Discovery in context: As you plan your trip, Airial gives you the tools to discover incredible ideas and expert advice specific to your itinerary and interests which can be instantly imported into your trip.
Now you can “Just imagine your trip and Airial it!”.
Try it out now on your laptop at airial.travel!
Michael C. Mozer retweeted
Excited to present "Recurrent Complex-Weighted Autoencoders for Unsupervised Object Discovery" at #NeurIPS2024!
TL;DR: Our model, SynCx, greatly simplifies the inductive biases and training procedures of current state-of-the-art synchrony models. Thread 👇 1/x.
Michael C. Mozer retweeted
🔍 New LLM Research 🔍
Conventional wisdom says that deep neural networks suffer from catastrophic forgetting as we train them on a sequence of data points with distribution shifts. But conventions are meant to be challenged!
In our recent paper led by @YanlaiYang, we discovered a curious behavior in overparameterized networks, especially LLMs—as we train the network on a cyclic sequence of documents, it starts to anticipate the next document and reverses the forgetting trend! ⤴️
▶️ After 3-4 cycles, the network reverses over 90% of the forgetting right before seeing the original document again.
▶️ The amount of anticipation emerges with the size of the network. LLMs <= 160M show no anticipation.
▶️ We showed that you can reproduce such an effect in a toy network!
Check out more details in our arXiv preprint on anticipatory recovery:
Reawakening knowledge: Anticipatory recovery from catastrophic interference via structured training.
🚀 arxiv.org/abs/2403.09613 🚀
#LLM #AI #Research
Michael C. Mozer retweeted
Nature Comms paper: Subtle adversarial image manipulations influence both human and machine perception! We show that adversarial attacks against computer vision models also transfer (weakly) to humans, even when the attack magnitude is small. nature.com/articles/s41467-0…
Michael C. Mozer retweeted
1/ Today we are excited to introduce Phenaki: phenaki.github.io, short-link-to-paper, a model for generating videos from text, with prompts that can change over time, and that is able to generate videos that can be as long as multiple minutes!
Michael C. Mozer retweeted
Two important breakthroughs from @GoogleAI this week - Imagen Video, a new text-conditioned video diffusion model that generates 1280x768 24fps HD video. And Phenaki, a model which generates long coherent videos for a sequence of text prompts. imagen.research.google/video…
Michael C. Mozer retweeted
We are excited to make the jump to complex real-world data with this class of models — and about the potential that slot-based models have for reducing the need for detailed human supervision when learning about the physical world.
6/7
Michael C. Mozer retweeted
Excited to share our work on self-supervised video object representation learning:
We introduce SAVi++, a slot-based video model that — for the first time — scales to Waymo Open driving scenes w/o direct supervision.
🖥️ slot-attention-video.github.…
📜 arxiv.org/abs/2206.07764
1/7