🔥 TinyAya v0.3 follow-up: the run early-stopped — so we launched the anneal leg it skipped. It's descending right now, and it's already beating the plateau. For everyone who asked for the nitty-gritty after the last post — here's what's actually happening on the TPU pod as I write this. 🧵👇 Why an anneal leg exists at all. The 110,463-step run trained on a WSD (warmup–stable–decay) schedule: 1,100 warmup steps, then a long plateau at lr 1.716e-4, with an 11,000-step linear decay to zero scheduled for the end. But we also ran stop-anytime early stopping — 10 validation cycles (2,500 steps) without improvement ends the run. At step 65,250 of 110,463, the plateau flattened out (best val composite 2.9048 @ step 62,750) and the early stop fired… which meant the scheduled decay never ran. Our round-3 ablation had shown the decay phase is where WSD harvests its loss drop — it beat the plateau on every metric at probe scale. So the model was parked one phase short of done. ⚙️ The anneal leg, mechanically. We resumed the exact step-65,250 checkpoint — weights, full Adam optimizer state, and RNG restored from GCS — and extended max_steps to 76,250. With the WSD scheduler's total set to start+11,000, the resume lands directly inside the decay: no warmup replay, no discontinuity. Verified from the first logged points: lr = 1.7121e-4 at step 65,275, stepping down by exactly 3.9e-7 per 25 steps — the predicted linear slope of 1.716e-4 ÷ 11,000. Every other hyperparameter is byte-identical to the plateau leg (global batch 32 = 2 rows/chip × 16 chips, grad-clip 1.0, same seed, same 3,200-sample val gate every 250 steps), and the data order continues deterministically mid-epoch-2. Early stopping is OFF for the leg — it's a fixed-length descent by design. 📉 Is it working? Yes — measurably, within the first 750 steps. • plateau best: 2.9048 (step 62,750) • step 66,000: 2.8922 • step 67,250: 2.8827 — text token accuracy 95.2%, text loss 0.458, semantic-codebook accuracy 40.6% Every validation cycle since the descent began has set a new best. ~9,000 steps (~6 h) still to go as the lr grinds toward zero. 🔢 Scale, in tokens: each step processes 32 sequences × 300 frames × 9 tokens/frame (1 text + 8 audio codebooks) ≈ 86k supervised tokens → the anneal alone covers ~950M tokens, bringing the full run to ~6.6B by step 76,250. Hardware: the same multi-host TPU v6e-16 (4 hosts × 4 chips, SPMD data-parallel), ~1.40 s/step, zero spot preemptions across the entire 2.2-day plateau leg. 🧯 One honest ops note for the monitoring nerds: W&B briefly showed the run as "crashed" mid-anneal — it was a heartbeat lapse during the boot's one-time XLA compile of the checkpoint-save graph, not a failure. The process was at 200% CPU the whole time and recovered on its own. Multi-day TPU runs are 50% training and 50% learning which alarms are real. 📦 Every checkpoint is kept (log-spaced early + every 1,000 steps) — the full trajectory ships as a Pythia-style suite for anyone who wants to study how a speech-to-speech model learns, including how the anneal reshapes it. 📈 Watch the descent live: wandb.ai/cataluna84/tinyaya-… 🔧 The full engineering trail — plan, hyperparameter rationale, resume drills (PR #10): github.com/tiny-aya-simultan… 📝 Project blog: labscommunity.cohere.com/blo… Release evals (ASR-chrF++, MOS, BLASER) and the Hugging Face suite land after the descent finishes. The next post has final numbers. 🎯 TPUs via the TRC @Google grant and project in the @Cohere_Labs Expedition Tiny Aya 2026 #SpeechAI #MachineTranslation #TPU #LearningRateSchedules #OpenSource #MLOps