Yuvanesh S

Let's goo 😊

Been implementing GKD on GSM8K. wrote the whole pipeline of data loading, JSD divergence in log-space, on-policy generation + teacher-forced scoring, all of it. Results are mixed and I think that's worth posting honestly. setup: - student: Qwen2.5-0.5B (base, SFT'd myself on GSM8K) - teacher: Qwen2.5-1.5B-Instruct, used as-is, NOT fine-tuned on GSM8K (budget call) - on A10G via Modal, ~$30 total budget - eval: greedy decode, 500-example fixed subset of GSM8K test, exact-match on extracted final answer results: - SFT baseline: 32.4% (162/500) - reproduced independently across two separate random subsets (32.2% and 32.4%), so this number is stable, not noise - on-policy GKD (λ=0.85, JSD divergence, 4000 steps): 21.6% (108/500) so GKD made it worse. not the result I expected, but a real one, and worth understanding rather than hiding. Leading hypothesis: the teacher was never adapted to GSM8K's answer format (the "#### <number>" convention). the paper's own setup explicitly SFTs the teacher on the target task before distilling ... I skipped that step to save compute. Qwen2.5-1.5B-Instruct is genuinely strong at math (73.2% GSM8K per Qwen's own technical report), but its natural answer style likely doesn't match what my student was trained to produce. distilling against a teacher with a different output convention for thousands of steps probably pulled the student's format away from what my eval's answer-extraction expects, independent of whether the underlying reasoning got worse. Haven't confirmed this yet — next step is literally reading the raw generations side by side before assuming. what's next: - SFT the teacher on GSM8K too, matching the paper's actual setup, then rerun - if that closes the gap, it's a clean confirmation that teacher-task alignment matters as much as the on-policy mechanism itself - budget didn't stretch to a full λ/divergence ablation grid this pass - this was one config, run once, reported as exactly that paper: On-Policy Distillation of Language Models (Agarwal et al.) @agarwl_ @nino_vieillard @Yongchao_Zhou_ @OlivierBachem

Numbers Every Programmer Should Know, 2026 edition: L1 cache reference - 0.5 ns Branch mispredict - 5 ns Mutex lock/unlock - 25 ns Main memory reference - 100 ns @JeffDean leaving Google - 27 years Some latencies you just can't hedge a backup request against.

The good old days 🥹

Training a student on its own generations usually drags in the whole RL toolkit reward-to-go, importance weights, ratio clipping.("Policy grads") GKD gets it with a loss you backprop through exactly like cross-entropy. no policy gradients anywhere. The trick: never differentiate through sampling. sample a sequence from the student, freeze it, then compare full teacher/student distributions at every fixed prefix. sampling only decides where to look, not what gets learned. That one move is why fixing train/inference mismatch costs you nothing but a generate() call. Wrote up the mathematical view , temperature, why KL blows up on mismatched supports, JSD as the bounded fix, and where the λ / divergence knobs actually bite. Blog : yuvanesh.vercel.app/blogs/GK… Thanks for the paper : ON-POLICY DISTILLATION OF LANGUAGE MODELS: LEARNING FROM SELF-GENERATED MISTAKES @agarwl_ @nino_vieillard @Yongchao_Zhou_

There's a problem with KL divergence, and it's the same one from the last paper I read ... Normal knowledge distillation minimizes forward KL, D_KL(p‖q_θ), with p the teacher and q_θ the student. Two issues: it's asymmetric, and it's unbounded. If there's any token the teacher gives mass to and the student gives zero, that term blows up to infinity. So the student is pushed to put probability everywhere the teacher does, including the tail it doesn't have the capacity to model. In open-ended generation that shows up as the student assigning real probability to text the teacher would basically never produce. Here in GKD paper the approach is different. Instead of picking forward or reverse, it uses a generalized JSD: M = βP + (1−β)Q JSD^(β)(P‖Q) = β·D_KL(P‖M) + (1−β)·D_KL(Q‖M) β → 0 gives you back forward KL. β → 1 gives you reverse KL. β = 0.5 is standard JSD, symmetric and bounded by log 2. The useful part: for any β strictly between 0 and 1, M is zero only where both P and Q are zero. So the infinity can't happen the objective stays finite no matter how the supports line up, and so do the gradients. The takeaway is that forward vs reverse KL was never a binary choice. It's a continuum, and β is just a hyperparameter you tune to the student's capacity and the task. GKD's other contribution is separate but stacks with this: train on sequences the student generates itself, so you're not distilling on a distribution the student never sees at inference time. Divergence choice and sampling distribution are two different knobs.

Starting my deep dive into on-policy distillation. MiniLLM is where it begins. One direction change in the KL divergence and the entire distillation algorithm becomes something completely different. Standard KD optimises forward KLD. Forces the student to cover every mode the teacher has, including the ones it has no capacity for. Void regions get overestimated, exposure bias gets baked in from day one, calibration breaks. The bigger the teacher-student gap, the worse it gets. Reverse KLD flips this. The student seeks the teacher's major modes, ignores the long tail, trains on its own samples. On-policy, policy gradient, importance weights, PPO clipping, the whole RL machinery enters distillation through this one change. Built the complete mathematical picture from scratch. Every derivation step by step, the policy gradient from first principles, single step decomposition, teacher mixed sampling, length normalisation, the full algorithm with a worked numerical example. Mini implementation to be added soon.... Check it out : yuvanesh.vercel.app/blogs/Mi… Paper : arxiv.org/pdf/2306.08543 Thanks for the work by @donglixp @gu_yuxian

Was going through the MiniLLM paper and found a simple but fascinating optimization issue that can lead to a form of reward hacking where the model learns to exploit the objective instead of actually generating better responses. If the per-token reward is rₜ = log(p_teacher(yₜ) / p_student(yₜ)) then, E[rₜ] = -KL(q_student || p_teacher) ≤ 0 Every generated token has a negative expected reward. Since the return is the cumulative future reward, Rₜ = Σ(i=t→T) rᵢ longer generations accumulate more negative rewards, while an immediate EOS gives Rₜ = 0. From the optimizer's perspective: 0 > -0.3 > -5.0 The optimization objective can therefore unintentionally favor short or even empty responses, not because they're better, but because they avoid accumulating negative rewards. A simple and elegant solution is to optimize the average per-token reward instead of the cumulative reward: Rₜ^(norm) = Rₜ / (remaining sequence length) This removes the incentive to exploit response length and makes the objective focus on per-token quality rather than sequence length. A neat solution adopted in the MiniLLM paper. Working on the blog and a tiny implementation , will post it soon

Independently verified Theorem 2 from the EGGROLL paper by @bidiptas13 @j_foerst @shimon8282 @AaronCourville (arXiv:2511.16652) on a 4GB RTX 3050. slope -0.898 (toy), -0.755 (MNIST MLP). R² = 0.998 and 0.9995. both firmly in [-0.8, -1.2]. but the methodology is the actual contribution , naive replications will get slope ≈ -0.004 and walk away thinking the theorem fails. Three things have to be right: 1. nonlinear fitness. the obvious quadratic (-‖W − W*‖²) gives EGGROLL and Gaussian ES identical bias by construction zero 3rd+ derivatives, theorem is invisible. 2. σ-tuning. bias ∝ σ², SEM is roughly σ-independent → SNR ∝ σ². at σ=0.05 slope is flat. at σ=0.3 it’s -0.755. same theorem, same code. 3. proper bias estimator. average gradients across trials first, then: bias(r) = √(max(err² − SEM_eg² − SEM_ref², 0)). per-trial error is just MC noise. Theorem 2 holds. the math is clean. the setup just has to be. think @yacinelearning would find the bias-isolation angle interesting. full writeup → yuvanesh.vercel.app/blogs/EG… code → github.com/YuvaneshSankar/EG…