Context Window (@ContextWindow_) header
Context Window (@ContextWindow_) profile picture

Learn the latest AI daily and prepare for interviews! ⬇️

San Francisco, CA
Joined August 2026
Photo by Context Window (@ContextWindow_)Photo by Context Window (@ContextWindow_)Photo by Context Window (@ContextWindow_)Photo by Context Window (@ContextWindow_)Photo by Context Window (@ContextWindow_)Photo by Context Window (@ContextWindow_)Photo by Context Window (@ContextWindow_)Photo by Context Window (@ContextWindow_)Photo by Context Window (@ContextWindow_)Photo by Context Window (@ContextWindow_)
How far can you push a frontier-scale LLM on Huawei Ascend alone? Xingchen AGI Lab pretrained a 285B MoE on 8,192 Ascend 910C cards, and its DSpark speculative decoder gets 2.67x to 3.77x the base decoding throughput, against 1.80x to 1.93x for MTP3. The Xing 4 report (the TeleChat line) is mostly systems work. For long context they add sparse attention on top of MLA: a small indexer picks the top-K past tokens for each query, and IndexShare reuses those picks across layers instead of recomputing them. Dropping pipeline parallelism for hybrid sharding adds 16.1% training throughput. They open-sourced a 29B MoE with 4B active params and native 256K context. The 4.68x prefill gain over plain MLA comes from a simulator, and the authors say real serving may look less rosy. Does it hold on a production cluster with mixed prompt lengths? arxiv.org/abs/2611.22115
35
A 36B MoE that activates about 4.6B parameters per token lands within 1.2 to 3.7 points of a dense 32B trained on the same data, on MMLU, GSM8K, HellaSwag and HumanEval. It is one of six fully open K2-Horizon models, from 0.9B dense to a 375B-A23B MoE. The trick is MoVA, from the K2 team at MBZUAI's Institute of Foundation Models. Normal MoE only makes the feed-forward layers sparse. MoVA also routes each token's attention value projection to 4 of 64 value experts and mixes them before caching, so the KV cache stays the same size and FlashAttention still works. Uno, a diffusion adapter, drafts a block of tokens that the same model then verifies: up to 2.2x faster decoding at batch size 1 on the 7B, same output distribution, no separate draft model. Everything ships: weights, 380 checkpoints, training logs, 21.6 TB of data, and the pipelines behind about 10T synthetic pre-training tokens. Caveat: that small gap is measured on base checkpoints, and at batch 64 Uno drops to 1.5x. arxiv.org/abs/2611.21003
1
46
Can an open-weight model learn to run a real robot the way frontier models learned computer use? A 27B model trained with RL only in simulation went from 1.6% to 74.4% on LIBERO, and succeeded in 60% of trials on a real dual-arm robot where the base model got 0%. Song Han's group at MIT, with UC Berkeley, NVIDIA and UCSD, treats the robot like a computer-use agent. Each round, the model looks at the camera views and arm state, reasons in a chain of thought, and outputs the next end-effector pose. Inverse kinematics moves the arm. No demonstrations, no harness. – LIBERO-Pro, new task in a familiar scene: 57.4% vs 0.75% for pi0.5 fine-tuned on demos – no measurable drop on nine reasoning, coding and vision benchmarks – code is released Caveat: the real-robot number comes from their own trials, and LIBERO is still a fairly narrow set of tabletop tasks. arxiv.org/abs/2611.18562
4
13
428
Ten robot demos on a 552B backbone beat fifty demos on a 2B one: 87.8% vs 85.6% on LIBERO Goal. ScaleVLA asks whether frontier-scale VLMs make better robot policies, not just better planners. The setup is simple. A team from Tsinghua, Peking University, SJTU and Galbot freezes DeepSeek-V4.1-Flash (a 552B MoE), reads its features once per action chunk, and trains only an adapter plus a diffusion transformer that outputs continuous actions. Backbones from 2B to 552B get the same recipe. The big backbone helps most where small VLAs struggle: LIBERO-PRO generalization reaches 45.1% vs at most 31.3% for smaller backbones, and a real charger insertion goes from 4/10 to 9/10. Median latency is 216 ms per action chunk. My question: with the backbone frozen, is the gain from scale itself or from what this one model saw in pretraining? A same-size model from another family would separate the two. And 10 real trials is a small sample. arxiv.org/abs/2611.21607
51
Latent video models make every camera control wait for a whole block of frames. PixVid moves a pretrained model into pixel space and cuts median control-to-screen latency from 495 ms to 34 ms, with each control landing on the next frame. The VAE compresses several frames into one latent, so conditions and controls apply to groups of frames. Researchers at Roblox (senior author Xun Huang) keep the Wan2.1-14B diffusion transformer, give each frame its own patch tokens, and swap the VAE decoder for a small causal convolutional head that paints pixels. Two tricks make it train: predict the clean video instead of velocity, and use a larger noise shift. Quality holds: VBench total 84.78 vs 84.05 for the Wan2.1 base, after about 4K B200 GPU-hours. Doubling the patch size gives 960x1664 video at the same per-frame token count as 480x832, and depth-conditioned edits keep 98% of fast motion vs 83% for a latent model. Caveat: fine-tuning used 64K videos generated by Wan2.1 itself, and FVD on editing trails the latent model (118 vs 91). arxiv.org/abs/2611.17336
9
2
74
3,204
Agent memory that keeps every record and still answers fast: MIRIX-2.0 reaches 72.6% on LongMemEval-S, 14.3 points above the strongest baseline, and 91.4% on LoCoMo, with no fine-tuning. The MIRIX AI team splits memory across six typed agents that write complete records, so nothing gets summarized away. A provenance hypergraph indexes those records with frames that cite their source rows, and retrieval runs under fixed budgets, so the evidence a question reads doesn't grow with the history. A background step called AutoDream merges duplicates and distills finished task sessions into skills. – On LongMemEval-S, the hypergraph alone adds 11.2 points – As procedural memory: 76.4% on ALFWorld with GPT-6 Luna, vs 73.6% for Agent Workflow Memory and 69.4% with no memory – With AutoDream off, it answers faster than HyperMem on all five timed benchmarks Caveat: the QA runs use gpt-4.1-mini with an LLM judge, and the authors are building this as a product, so I'd want independent runs. Code is public. arxiv.org/abs/2611.12068
1
2
21
667
Why does speculative decoding get worse when a model is looking at a picture? On Qwen3.5-4B, draft tokens that lean hardest on the image get accepted 22% of the time, vs 46% for tokens that barely use it. MMFlash targets that gap and reaches up to 3.58x lossless speedup. A diffusion drafter guesses a block of 16 tokens in one pass and the big model verifies them in parallel. Visual dependence is scored as the KL gap between the target's predictions with and without the image, and those tokens get upweighted when training the drafter. For long video, sparse attention shrinks a 40K+ token context to about 1.6K. Greedy decoding on Qwen3.5-4B and 9B averages 3.00-3.05x, about 10% above DFlash and well above EAGLE-3 (1.58-1.68x). From Sicheng Feng, Xinchao Wang and colleagues at NUS. Does the reweighting actually close the 46% vs 22% gap, or mostly shift where the drafter spends capacity? arxiv.org/abs/2611.18482
1
1
8
440
A 1980s idea solves 99.996% of Sudoku-Extreme and 100% of Maze-Unique after 42 ms and 132 ms of training on one GPU, at least 1,000x faster to train than the looped models used on these puzzles. Looped models reapply one block many times and train by backprop through the loop, which is slow and can be unstable. CRAM learns an explicit energy instead, a trainable version of Hopfield and Tank's puzzle energy. Training lowers the energy of real solutions and raises it for wrong boards, with no backprop through a solver. To reason, it fixes the clues, runs a few chains from random fills and lowers the energy one cell at a time, stopping once any chain gets below a learned threshold. It also matches the best looped baselines on Sudoku-Extreme with at least 282x fewer inference FLOPs. The authors' own caveat: if a tiny energy solves these this easily, Sudoku and mazes say little about whether looped reasoning scales. arxiv.org/abs/2611.17309
3
1
5
497
Nine arXiv papers from Nov 9-15 worth your time. The one I keep coming back to: an energy-based recurrent reasoner hits 99.9% on Sudoku-Extreme with up to 345x fewer inference FLOPs than EqR. 1. Energy-Based Reasoners: 99.9% on Sudoku-Extreme, up to 345x fewer inference FLOPs than EqR at comparable accuracy, with 1.8M params arxiv.org/abs/2611.08977 2. Embodied First Exam: GPT-6 Astra in Codex scores 77% with privileged access vs 30% in the standard setting on the same 391 tasks arxiv.org/abs/2611.12177 3. Unbundling Multimodal Vision Encoding: at 1.7B, encoder-free trails a scratch-trained encoder 39.0 vs 44.4 avg VQA arxiv.org/abs/2611.17334 4. Sage-Taiyi: Overall 54.0 vs 47.9 for AI Scientist-v2 on the 265-task Taiyi-Bench; one chemistry collab took a yield from trace to 84% arxiv.org/abs/2611.12762 5. Evolving Infra: up to 4.1x lower mean end-to-end latency and 14x lower P95 TTFT than Dynamo, up to 2.7x throughput arxiv.org/abs/2611.12304 6. SwarmEvolve: matches the baselines' 60-minute best in a median 3-8 minutes across 21 problems; beats AlphaEvolve constructions on 3 arxiv.org/abs/2611.08483 7. System Optimizations for Cost-Efficient Agent Harnesses: harness-side changes cut cost to 0.77x of baseline at 0.97x accuracy arxiv.org/abs/2611.12122 8. UGG: one model for video, depth and pose; ScanNet++ camera trajectory error drops from 0.190 to 0.087 vs Geo4D arxiv.org/abs/2611.14440 9. SO-Arena: 21 models across 11 domains; the most accurate System One model ranks only seventh on decision reliability arxiv.org/abs/2611.15777 If you only read one for work on Monday, I'd pick the harness cost paper.
2
2
12
531
GPT-6-Astra controlling a robot directly goes from 46.7% to 81.7% success on EBench, with no weight updates and no trained robot policy. The only change is a text memory it writes for itself. From one demonstration, the model writes a task memory. On training episodes it keeps a short working memory and rewrites it after each episode with lessons on procedure, finishing and recovering from mistakes. At test time both are frozen. The best harness that hands off to a trained policy gets 56.0%, the best post-trained policy 61.4%. Tokens per success also drop: 2.76M vs 8.12M for direct control. Yifei Yao, Chunhua Shen and colleagues at Shanghai AI Lab. My question: are these lessons general, or mostly fixes memorized for EBench's 26 tasks? They do report transfer to unseen layouts and to weaker models as is. arxiv.org/abs/2611.11509
3
27
1,235
Video generators and depth or pose estimators are usually separate models. UGG trains one model for all three, and on ScanNet++ its camera trajectory error (ATE) drops from 0.190 to 0.087 against Geo4D. How it works: RGB frames, depth maps and camera poses (as per-pixel ray maps) become aligned tokens in one transformer, and each modality and frame gets its own noise level. What you observe stays clean, what is missing gets denoised. Change the mask and the same model does camera-controlled video, depth or pose, and it can train on datasets that label only some modalities. The result I liked: drop RGB generation from training and KITTI depth error more than doubles (Abs Rel 0.152 to 0.322). Caveat: it is not uniformly better, and Geo4D still wins dense depth on Sintel and 7-Scenes. From Amani Kiruga, Yilun Du and colleagues at Harvard, MIT, UC San Diego and Cambridge. arxiv.org/abs/2611.14440
3
144
“An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale” (2020) Vision used to run on convolutions. ViT dropped them and set records anyway: 88.55% on ImageNet and 77.63% on VTAB, with 2.5k TPUv3-core-days of pretraining vs 9.9k for the BiT-L ResNet. The idea: cut the image into 16x16 patches, embed each one like a word token, add position embeddings, and run a plain Transformer encoder. The catch: trained on ImageNet alone it lands a few points below comparable ResNets, and only pulls ahead with 14M to 300M images. This week's Unbundling Multimodal Vision Encoding paper from Meta FAIR and NYU still builds every encoder it tests as a 0.4B ViT, and Lucas Beyer is an author on both. By Alexey Dosovitskiy, Lucas Beyer, Neil Houlsby and colleagues at Google Research, Brain Team. arxiv.org/abs/2010.11929
5
244
Does a multimodal LLM need a pretrained vision encoder? In this controlled study, an encoder trained from scratch with the backbone is on par. Dropping the encoder is what hurts: 39.0 vs 44.4 avg VQA at 1.7B, and giving the backbone the encoder's 0.4B params instead only gets 38.6 to 38.8. Ellis Brown and colleagues at Meta FAIR and NYU train every model from scratch on shared data and compare at matched lifetime compute, so encoder pretraining counts against the budget. They split the encoder into three choices: a visual prior, dedicated vision capacity, and placement before fusion. Only the capacity matters. Vision-only modules in the backbone's early layers match a pre-fusion encoder and cut time to first token. One loose end: a hierarchical patch MLP, still encoder-free, matches the encoder at 640x640 input. Is that dedicated capacity sneaking back in through the patch embedding? arxiv.org/abs/2611.17334
1
5
248
Teaching a model new facts by self-distillation works, but fully on-policy training burns a lot of supervision. SSD nearly matches on-policy distillation with 38-69% fewer supervised tokens on three closed-book QA benchmarks. The setup: the same model reads the source document and acts as teacher, while a closed-book copy learns to reproduce its answers. In SSD the student writes each rollout itself, and the teacher only steps in at tokens where its next-token distribution diverges a lot from the student's. So supervision lands exactly where the document changes the prediction. It also beats off-policy distillation on held-out tests and keeps the most general capability on a forgetting stress test. Caveat: the baselines are other distillation recipes, not just putting the document in context. From Shayan Talaei, Azalia Mirhoseini and colleagues at Stanford, with Jonas Hübotter at ETH Zürich. arxiv.org/abs/2611.12016
1
1
70
Zeroing fewer than 0.05% of an LLM's weights, and changing nothing else, matches full fine-tuning across RLVR, SFT and on-policy distillation. That's the claim from Mengye Ren's group at NYU with Yani Ioannou at Calgary. How it works: Deletion Tuning freezes the pretrained weights and learns one logit per weight in the gate projections. A sigmoid mask sharpens over training into a hard threshold, so each weight either keeps its pretrained value or becomes zero. The whole adapter is one bit per weight. – the mask is hundreds to thousands of times smaller than the FFT checkpoint – under RL, prior-task accuracy stays near base while FFT, LoRA and random sparse tuning lose up to 5.4 points – tested on eight backbones from 0.5B to 30B (including a MoE) Caveat: the match with FFT is claimed at the largest scale in each setting, so I'd check how it holds up on smaller models before swapping out LoRA. arxiv.org/abs/2611.10866
1
49
Where does the money in a coding agent run actually go? Mostly to re-reading context. Park, Stoica, Cheung and colleagues at Berkeley find cache reads grow to 53% and 73% of total cost as context gets long, and their harness-side fixes cut cost to 0.77 of baseline with an accuracy ratio of 0.97. The reason: every tool call is a new LLM call that pays for the whole cached prefix again. Agents also use only 27% of tool-output tokens later. So the fixes sit between the harness and the model (Claude Code, Codex): near-lossless compaction of tool outputs, garbage collection of stale context, SQLite metadata tables for file search, and a prompt nudging the model to batch tool calls. Savings are biggest on messy file trees and smaller on SWE-bench (0.80 to 0.92). Does context GC still pay off once providers make cache invalidation cheaper? arxiv.org/abs/2611.12122
6
2
22
616
A 1.8M-parameter looped model hits 99.9% on Sudoku-Extreme, and at comparable accuracy it needs up to 345x fewer inference FLOPs than EqR, the next-best recurrent reasoner. The problem with looped models like HRM and TRM: they iterate a block to reason in latent space, but nothing tells them whether a step made progress or when to stop. Energy-Based Reasoners learn an energy score over the latent state, and each loop is just a gradient step downhill on it. Low energy means a good answer, a near-zero gradient means stop, and with several samples you keep the lowest-energy one. Training adds exploration so the model finds more than one valid solution. With a single sample it gets 99.4% on Sudoku-Extreme (TRM 87.4%) and 87.4% on Maze-Hard. Caveat: these are algorithmic puzzles, and the baselines are numbers from their own papers. Authors include Yilun Du and Heng Ji. arxiv.org/abs/2611.08977
1
8
1
96
3,359
If you serve agents on prefill/decode disaggregated clusters, Evolving Infra cuts mean end-to-end request latency by up to 4.1x and P95 time to first token by up to 14x vs NVIDIA's Dynamo, with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs. The problem: agent traffic shifts between prefill-heavy and decode-heavy phases, and generic cache eviction throws away KV that the agent is about to reuse. Song Han's group at MIT, with Ion Stoica and Tushar Krishna, decouples compute from KV management so any node can flip between prefill and decode. An online planner retunes the split, and the KV manager evicts using the agent's own context signals (finished sub-agents, compacted history) first. – up to 2.7x Dynamo's throughput, 38% more than the best static split – 81 to 95% less lost KV reuse Caveat: the gains are big on coding and computer-use traces but only 1.15 to 1.23x on Putnam math, so it pays off most when contexts branch and get reused. arxiv.org/abs/2611.12304
1
2
1
10
590
Agent memory can break tasks an agent already solves. A KAIST team's fix, WikiTrace, lifts the average over eight agent suites to 90.2 vs 87.0 for the same lessons without their evidence (79.5 with no memory), and cuts solved-to-failed regressions from 6.5% to 4.5%. The diagnosis: in 52% of memory-induced failures, the agent was following a stored lesson, most often one that was simply wrong. Lessons are distilled from runs labeled only pass or fail, so they have to guess which actions mattered. WikiTrace keeps each lesson linked to a summary of why its source run succeeded or failed, and to the raw trajectory. When a lesson is vague, conflicts with another, or came from a failed run, the agent opens those files with terminal tools and checks before acting. Suites span ALFWorld, ScienceWorld, PlanCraft, τ3-bench Retail and AppWorld. My question: how much context and cost does each lookup add, and does it hold up as memory grows to thousands of lessons? arxiv.org/abs/2611.14320
2
14
315
Give GPT-6 Astra in Codex full simulator state and unlimited resets, and it solves 77% of 391 hard robot tasks. Give it only its own sensors and no resets, like the real world, and it solves 30% of the same tasks. Embodied First Exam, from Manling Li's lab at Northwestern with CMU, Berkeley, Stanford and others, asks whether coding agents can actually control robots: 1,156 tasks in 35 suites on 47 robots, from bimanual arms to quadrupeds and humanoids. The agent picks its own approach: drive the robot step by step, write a control program, or train a policy. A Realtime variant keeps the world moving while the model thinks, and 63% of those tests go unsolved by every agent. What I liked: in a soccer task, agents given a GPU trained a policy, and only trained policies won. Caveat: it is all simulation, so real robots will likely be harder still. arxiv.org/abs/2611.12177
86