Isha (@slowdownisha) header
Isha (@slowdownisha) profile picture

researcher (almost)

Joined February 2022
Photo by Isha (@slowdownisha)Photo by Isha (@slowdownisha)Photo by Isha (@slowdownisha)Photo by Isha (@slowdownisha)Photo by Isha (@slowdownisha)Photo by Isha (@slowdownisha)Photo by Isha (@slowdownisha)Photo by Isha (@slowdownisha)Photo by Isha (@slowdownisha)Photo by Isha (@slowdownisha)
Isha retweeted
currently witnessing @tszzl explain neon genesis evangelion to two blonde women at the bar
20
7
2
598
24,086
maybe gpt 6.2 astra was oai's way to pace the frontier.
1
4
229
New blogpost: The Deadly Triad of Off-policy Policy Gradient nanjiang.cs.illinois.edu/202…
1
22
209
13,792
Isha retweeted
We might’ve killed retraining for harness changes. Our model broke on a new harness. We didn’t retrain it. We edited 5,064 weights. Out of 4 billion. 20.1% -> 88.2% No gradient descent. Old-harness accuracy didn’t move. DAWE(Data-Attributed Weight-Editing) is fully open 🔥
22
35
11
333
25,981
Could be the start of a trend in new pre-training recipes: no LR decay phase anymore. (Also seen in Kolibri.) Instead: constant LR + averaging the last N checkpoints (or a running weight average). I guess that makes it easier in practice because you don't have to commit to a total training length upfront. A little meta analysis based on those who share training details: - Classic cosine: Llama 3, OLMo 2, OLMo 3, GLM-4.5, GLM-5, Kimi K3, Phi-4 - WSD ("Warmup-Stable-Decay"): DeepSeek V3, DeepSeek V4, DeepSeek V4.1-Flash, Kimi K2, Kimi Linear, Nemotron 3 Nano, Nemotron 3 Super, SmolLM3, MiniMax M2 - No decay (constant LR plus checkpoint averaging): Kolibri-1, DatologyAI's models
Replying to @datologyai
Trick #3: Merge, merge, merge. In pretraining, we skipped LR decay for a running weight average. Then we merged pre-context-extension models into post-extension checkpoints (see fig. for ablation study gains). Merging was also an expedient stand-in for MOPD in our RLVR recipe.
64
64
14
788
60,161
LLM pretraining has run on single-epoch text, bits per byte and iso-FLOP curves for years. visual generation still memorizes ImageNet for tens to hundreds of epochs and pushes FID down with CFG, with no strong signal those tricks transfer to real text-to-image. new blog: 🎨 scaling generative models on GPIC, 100M permissive text-image pairs. training pixel-space JiT from 3.6e18 to 2.3e20 FLOPs: 1) held-out loss is our main metric, and compute-optimal TPP sits at about 50 image tokens per image parameter 2) EMA is not necessary: a proper learning-rate schedule beats EMA on held-out loss at every budget by 0.4% to 1.0% 3) we use CFG = 1 (only the conditional branch) for every result, and call on everyone to drop CFG: it saves compute and leaves no room to hack the metric all code, configs and sweep data are open 🧵 blog: wenhaochai.com/blogs/scaling…
10
49
11
392
44,545
this one's going to be an interesting read.
Scaling matrix optimizers to wide networks is expensive. Our new research post introduces Echelon, a rank-anchored formulation of Muon that orthogonalizes only the top subspace in O(mnr) time while rescaling the tail: up to 10.5× faster optimizer steps, with Muon's loss at 1.3B. Blog: evolvingmachines.ai/blog/ech… We pair the algorithm with the first Lean-checked convergence proof for spectral steepest descent. Lean proofs: github.com/evolving-machines…
1
11
510
my take is, the grads through the norm may help with stability. the forward pass caps the logits but the projection keeps the norms from drifting.
anyone tried detaching the rms denominator in qk norm?
1
2
317
anyone tried detaching the rms denominator in qk norm?
1
3
478
deepseek flash is a good model.
1
9
402
more reasons to keep coming back to this. worth a reread.
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. thinkingmachines.ai/blog/end…
41
1
602
20,236
i will never get the hype of foldables and touch screen laptops. identity crisis in disguise.
1
6
566
deep layers are lazy
2
177
Isha retweeted
Jensen Huang is the Steve Jobs of Nvidia. He is irreplaceable .
1,902
5,915
729
85,442
4,067,693
today's read.
13
423
i cant express my love for opus 5.5 enuf. been using it for research and it stays coherent even during long sessions.
6
319
very cool post from marin on how they picked the 535B architecture. most interesting part to me is they went with token dropping instead of dropless, just to fit a bigger model on one rack, then switched to dropless mid-run once it was ready. not something you see often. good read!
Sharing the thought process behind the architecture for Marin 535B-A23B, which finished pretraining on 18T tokens last week: openathena.ai/blog/marin-535… Earlier this year the Marin project had 6 full-time employees at Open Athena, a mix of AI veterans and people newer to the space, like me. I learned a lot from the team as an IC. Since then we got our B200 GPU stack running in a month, scaled up 300x on GPU to a 535 billion parameter model, publicly forecasted the run, live-streamed the entire thing, and made it through pretraining slightly better than predicted with zero instabilities. In the blog I call out specific contributions on architecture- but the broader gratitude is to the ecosystem. 2 years ago this would have been inconceivable on a first large scale attempt. Now it’s possible, in large part due to published work from Llama, DeepSeek, Qwen, Gemma, Olmo, Cohere, Nemotron, Arcee, Inkling, Mistral, Microsoft AI, Kimi, Muse, academia, and many others. The recipes are out there. A small team with big dreams (and some funding) can train a big model. (And if you have doubts on that, you can copy our fully open recipe from data to code). Of course, training a model does not produce a binary outcome. I am proud of how far we’ve come and optimistic about how much more there is to explore! The team has grown recently and will be taking the model through SFT and RL soon, sharing the findings along the way.
8
421
something about deep learning still feels a little absurd to me - when a useful operation is hard to specify, we make it learnable. early attention learns which words to focus on. word embeddings learn relationships between words by predicting context. tuned lens learns to turn a model’s hidden states into readable predictions. all of them all share the same move. we choose the structure and the objective, and optimization fills in part of the procedure. its crazyy how far 'make the missing piece learnable' can take us.
1
9
392
positional embedding, lstm gates, moe routers, autoencoders.... all of them emerged from the same idea.
1
87
tpot is a teenage girl.
6
181