Yuvanesh S

Let's goo 😊

Thanks a lot for getting back really appreciate you offering. Thinking it through, one day probably isn't enough for this one. The fitness evaluation alone would eat a big chunk of it, and that's before the mixture sweep or the distillation run. Going to keep building it out on our smaller setup for now. But thanks again for the offer, and if you have compute lying around again down the line, I'd love to take you up on it...

Two things we would want to see, both from q0 ("Primitives for Hyper-Epoch Pretraining"): 1. Inference-time mixture of experts-by-seed. Instead of one model trained N epochs on a scarce corpus, train several from different inits on the same data, keep cyclic snapshots, score them on held-out data, and mix the token probs of the top-k. Different inits genuinely end up understanding the same data differently. 2. Collapsing that mixture back into a single model but with GRPO rather than the SFT the paper uses. If the teachers disagree, on-policy RL against the mixture may transfer more than logit matching does, and you stop paying k× inference cost. We have a signal already: Qwen2.5-1.5B, trained only on GSM8K, evaluated on MATH-500 top-2 mixture of 4 snapshots beats the single best snapshot. What we can't do on our compute is scale K, or run the GRPO distillation at all. 16×H100 fits a 1.5B–3B sweep comfortably. We were working on this and would like to run it end to end and publish results + code. Credit to the q0 authors: @industriaalist @akshayvegesna @ShmuelBerman Bishwas Mandal

Yeah you're right, hashing is cheaper. Best things is to probably ship it as the baseline honestly but i think they're solving different things? Hashing gives a new brand a slot, not a useful vector...the bucket already has an embedding learned from whatever random other brands landed there, so it's basically noise till the model updates again.The content tower is for that specific window. Compute wise it's not really in the request path though it runs nearline, few thousand new brands a day, you just write the vectors into the same embedding store and serving is a normal lookup and a small MLP is like 100k params vs a 2M * 64 table, so it's kinda rounding error. where you're right: if we retrain hourly anyway the cold window is short enough that hashing alone might be fine that's an A/B thing on the cold brand slice, not overall metrics. Two things I think is good- multi hash with like k=2 to 4 so each brand is a unique combo of buckets, collisions hurt way less. and ID dropout in training so the model actually learns to run on just the content features, otherwise the tower output is off distribution at serving. So can have a system of real embeddings for head brands, multi hash for the tail, content tower for cold start, and some gate that shifts over to the ID embedding once the brand has enough impressions.

Instead of handing a new brand some random or shared UNK vector, we can just train a small feed forward net... basically an embedding generator, that maps the brand's side features (name subwords, category distribution, price band, seller region/tenure and stuff like that) into the same space as the learned ID embeddings. So at serving time, a brand that's like ten minutes old still has all of those features sitting there... which means it gets a plausible warm start vector or something instead of pure noise.

Been implementing GKD on GSM8K. wrote the whole pipeline of data loading, JSD divergence in log-space, on-policy generation + teacher-forced scoring, all of it. Results are mixed and I think that's worth posting honestly. setup: - student: Qwen2.5-0.5B (base, SFT'd myself on GSM8K) - teacher: Qwen2.5-1.5B-Instruct, used as-is, NOT fine-tuned on GSM8K (budget call) - on A10G via Modal, ~$30 total budget - eval: greedy decode, 500-example fixed subset of GSM8K test, exact-match on extracted final answer results: - SFT baseline: 32.4% (162/500) - reproduced independently across two separate random subsets (32.2% and 32.4%), so this number is stable, not noise - on-policy GKD (λ=0.85, JSD divergence, 4000 steps): 21.6% (108/500) so GKD made it worse. not the result I expected, but a real one, and worth understanding rather than hiding. Leading hypothesis: the teacher was never adapted to GSM8K's answer format (the "#### <number>" convention). the paper's own setup explicitly SFTs the teacher on the target task before distilling ... I skipped that step to save compute. Qwen2.5-1.5B-Instruct is genuinely strong at math (73.2% GSM8K per Qwen's own technical report), but its natural answer style likely doesn't match what my student was trained to produce. distilling against a teacher with a different output convention for thousands of steps probably pulled the student's format away from what my eval's answer-extraction expects, independent of whether the underlying reasoning got worse. Haven't confirmed this yet — next step is literally reading the raw generations side by side before assuming. what's next: - SFT the teacher on GSM8K too, matching the paper's actual setup, then rerun - if that closes the gap, it's a clean confirmation that teacher-task alignment matters as much as the on-policy mechanism itself - budget didn't stretch to a full λ/divergence ablation grid this pass - this was one config, run once, reported as exactly that paper: On-Policy Distillation of Language Models (Agarwal et al.) @agarwl_ @nino_vieillard @Yongchao_Zhou_ @OlivierBachem

Numbers Every Programmer Should Know, 2026 edition: L1 cache reference - 0.5 ns Branch mispredict - 5 ns Mutex lock/unlock - 25 ns Main memory reference - 100 ns @JeffDean leaving Google - 27 years Some latencies you just can't hedge a backup request against.

The good old days 🥹

Training a student on its own generations usually drags in the whole RL toolkit reward-to-go, importance weights, ratio clipping.("Policy grads") GKD gets it with a loss you backprop through exactly like cross-entropy. no policy gradients anywhere. The trick: never differentiate through sampling. sample a sequence from the student, freeze it, then compare full teacher/student distributions at every fixed prefix. sampling only decides where to look, not what gets learned. That one move is why fixing train/inference mismatch costs you nothing but a generate() call. Wrote up the mathematical view , temperature, why KL blows up on mismatched supports, JSD as the bounded fix, and where the λ / divergence knobs actually bite. Blog : yuvanesh.vercel.app/blogs/GK… Thanks for the paper : ON-POLICY DISTILLATION OF LANGUAGE MODELS: LEARNING FROM SELF-GENERATED MISTAKES @agarwl_ @nino_vieillard @Yongchao_Zhou_

There's a problem with KL divergence, and it's the same one from the last paper I read ... Normal knowledge distillation minimizes forward KL, D_KL(p‖q_θ), with p the teacher and q_θ the student. Two issues: it's asymmetric, and it's unbounded. If there's any token the teacher gives mass to and the student gives zero, that term blows up to infinity. So the student is pushed to put probability everywhere the teacher does, including the tail it doesn't have the capacity to model. In open-ended generation that shows up as the student assigning real probability to text the teacher would basically never produce. Here in GKD paper the approach is different. Instead of picking forward or reverse, it uses a generalized JSD: M = βP + (1−β)Q JSD^(β)(P‖Q) = β·D_KL(P‖M) + (1−β)·D_KL(Q‖M) β → 0 gives you back forward KL. β → 1 gives you reverse KL. β = 0.5 is standard JSD, symmetric and bounded by log 2. The useful part: for any β strictly between 0 and 1, M is zero only where both P and Q are zero. So the infinity can't happen the objective stays finite no matter how the supports line up, and so do the gradients. The takeaway is that forward vs reverse KL was never a binary choice. It's a continuum, and β is just a hyperparameter you tune to the student's capacity and the task. GKD's other contribution is separate but stacks with this: train on sequences the student generates itself, so you're not distilling on a distribution the student never sees at inference time. Divergence choice and sampling distribution are two different knobs.

Starting my deep dive into on-policy distillation. MiniLLM is where it begins. One direction change in the KL divergence and the entire distillation algorithm becomes something completely different. Standard KD optimises forward KLD. Forces the student to cover every mode the teacher has, including the ones it has no capacity for. Void regions get overestimated, exposure bias gets baked in from day one, calibration breaks. The bigger the teacher-student gap, the worse it gets. Reverse KLD flips this. The student seeks the teacher's major modes, ignores the long tail, trains on its own samples. On-policy, policy gradient, importance weights, PPO clipping, the whole RL machinery enters distillation through this one change. Built the complete mathematical picture from scratch. Every derivation step by step, the policy gradient from first principles, single step decomposition, teacher mixed sampling, length normalisation, the full algorithm with a worked numerical example. Mini implementation to be added soon.... Check it out : yuvanesh.vercel.app/blogs/Mi… Paper : arxiv.org/pdf/2306.08543 Thanks for the work by @donglixp @gu_yuxian

Was going through the MiniLLM paper and found a simple but fascinating optimization issue that can lead to a form of reward hacking where the model learns to exploit the objective instead of actually generating better responses. If the per-token reward is rₜ = log(p_teacher(yₜ) / p_student(yₜ)) then, E[rₜ] = -KL(q_student || p_teacher) ≤ 0 Every generated token has a negative expected reward. Since the return is the cumulative future reward, Rₜ = Σ(i=t→T) rᵢ longer generations accumulate more negative rewards, while an immediate EOS gives Rₜ = 0. From the optimizer's perspective: 0 > -0.3 > -5.0 The optimization objective can therefore unintentionally favor short or even empty responses, not because they're better, but because they avoid accumulating negative rewards. A simple and elegant solution is to optimize the average per-token reward instead of the cumulative reward: Rₜ^(norm) = Rₜ / (remaining sequence length) This removes the incentive to exploit response length and makes the objective focus on per-token quality rather than sequence length. A neat solution adopted in the MiniLLM paper. Working on the blog and a tiny implementation , will post it soon

The performance gap is clear in every panel. This made me want to understand exactly "why" on-policy distillation works so much better. So I looked at the method comparison and it’s beautifully simple: Left: Traditional Sequence-Level KD (Forward KLD) → Teacher generates responses (y ~ p) → Student just imitates those fixed outputs with standard NLL loss Right: MiniLLM (Reverse KLD + On-Policy) → Student generates its own responses (y ~ qθ) → Teacher scores those student-generated tokens via reverse KL (log p/q) → Student learns on-policy from its own generations + teacher feedback Instead of pure imitation, the student improves by exploring with its current policy and getting direct teacher signals on what it actually produced. That shift explains why MiniLLM closes the gap to the teacher so effectively in the graphs. Planning to write a full blog post soon with a deeper mathematical & practical breakdown of MiniLLM.

Started diving into knowledge distillation recently and came across SeqKD approaches a lot. Then I came across the MiniLLM paper (On-Policy Distillation of Large Language Models) and these graphs immediately caught my eye MiniLLM vs SeqKD across 3 different teachers: - GPT-2 1.5B teacher - GPT-J 6B teacher - OPT 13B teacher MiniLLM (solid lines) consistently outperforms traditional sequence-level KD (dashed lines) on GPT-4 feedback scores, and gets much closer to the teacher’s performance as student size grows. On-policy distillation is looking really promising. Paper: arxiv.org/abs/2306.08543

That 2011 DAGGER paper by Ross, Gordon & Bagnell remains foundational. It reduces imitation learning under distribution shift to no-regret online learning through iterative on-policy data aggregation , a direct conceptual precursor to today’s on-policy distillation and data flywheel approaches in post-training. arxiv.org/abs/1011.0686