Yuvanesh S

There's a problem with KL divergence, and it's the same one from the last paper I read ... Normal knowledge distillation minimizes forward KL, D_KL(p‖q_θ), with p the teacher and q_θ the student. Two issues: it's asymmetric, and it's unbounded. If there's any token the teacher gives mass to and the student gives zero, that term blows up to infinity. So the student is pushed to put probability everywhere the teacher does, including the tail it doesn't have the capacity to model. In open-ended generation that shows up as the student assigning real probability to text the teacher would basically never produce. Here in GKD paper the approach is different. Instead of picking forward or reverse, it uses a generalized JSD: M = βP + (1−β)Q JSD^(β)(P‖Q) = β·D_KL(P‖M) + (1−β)·D_KL(Q‖M) β → 0 gives you back forward KL. β → 1 gives you reverse KL. β = 0.5 is standard JSD, symmetric and bounded by log 2. The useful part: for any β strictly between 0 and 1, M is zero only where both P and Q are zero. So the infinity can't happen the objective stays finite no matter how the supports line up, and so do the gradients. The takeaway is that forward vs reverse KL was never a binary choice. It's a continuum, and β is just a hyperparameter you tune to the student's capacity and the task. GKD's other contribution is separate but stacks with this: train on sequences the student generates itself, so you're not distilling on a distribution the student never sees at inference time. Divergence choice and sampling distribution are two different knobs.

The performance gap is clear in every panel. This made me want to understand exactly "why" on-policy distillation works so much better. So I looked at the method comparison and it’s beautifully simple: Left: Traditional Sequence-Level KD (Forward KLD) → Teacher generates responses (y ~ p) → Student just imitates those fixed outputs with standard NLL loss Right: MiniLLM (Reverse KLD + On-Policy) → Student generates its own responses (y ~ qθ) → Teacher scores those student-generated tokens via reverse KL (log p/q) → Student learns on-policy from its own generations + teacher feedback Instead of pure imitation, the student improves by exploring with its current policy and getting direct teacher signals on what it actually produced. That shift explains why MiniLLM closes the gap to the teacher so effectively in the graphs. Planning to write a full blog post soon with a deeper mathematical & practical breakdown of MiniLLM.

Started diving into knowledge distillation recently and came across SeqKD approaches a lot. Then I came across the MiniLLM paper (On-Policy Distillation of Large Language Models) and these graphs immediately caught my eye MiniLLM vs SeqKD across 3 different teachers: - GPT-2 1.5B teacher - GPT-J 6B teacher - OPT 13B teacher MiniLLM (solid lines) consistently outperforms traditional sequence-level KD (dashed lines) on GPT-4 feedback scores, and gets much closer to the teacher’s performance as student size grows. On-policy distillation is looking really promising. Paper: arxiv.org/abs/2306.08543

That paper is the direct conceptual ancestor of modern On-Policy Distillation (OPD) approaches. It first showed how to reduce imitation learning and structured prediction to no-regret online learning by performing dataset aggregation on the learner’s own induced distribution (rather than the expert’s). I came across this while digging into the deeper theoretical foundations of OPD. arxiv.org/pdf/1011.0686

Independently verified Theorem 2 from the EGGROLL paper by @bidiptas13 @j_foerst @shimon8282 @AaronCourville (arXiv:2511.16652) on a 4GB RTX 3050. slope -0.898 (toy), -0.755 (MNIST MLP). R² = 0.998 and 0.9995. both firmly in [-0.8, -1.2]. but the methodology is the actual contribution , naive replications will get slope ≈ -0.004 and walk away thinking the theorem fails. Three things have to be right: 1. nonlinear fitness. the obvious quadratic (-‖W − W*‖²) gives EGGROLL and Gaussian ES identical bias by construction zero 3rd+ derivatives, theorem is invisible. 2. σ-tuning. bias ∝ σ², SEM is roughly σ-independent → SNR ∝ σ². at σ=0.05 slope is flat. at σ=0.3 it’s -0.755. same theorem, same code. 3. proper bias estimator. average gradients across trials first, then: bias(r) = √(max(err² − SEM_eg² − SEM_ref², 0)). per-trial error is just MC noise. Theorem 2 holds. the math is clean. the setup just has to be. think @yacinelearning would find the bias-isolation angle interesting. full writeup → yuvanesh.vercel.app/blogs/EG… code → github.com/YuvaneshSankar/EG…

Few weeks ago @anirudhbv_ce shipped TurboQuant in cuTile on a Blackwell B200. Beautiful work - genuinely. I wanted to see what the same algorithm looks like at the opposite end of the stack: raw CUDA + hand-written PTX on a 4GB laptop GPU. No cuTile. No B200. Just nvcc, shared memory, and a lot of nsight-compute. Quick context — TurboQuant paper proposed compression in KV-cache and vector-search embedding way harder than anything before it, which I have discussed earlier in my blog , please go through it for clearer understanding of the algorithm : yuvanesh.vercel.app/blogs/Tu… I implemented it three ways and bench marked them against each other: 1. Vanilla CUDA — two kernels. Rotate (FWHT) in kernel 1, quantize + bit-pack in kernel 2. Clean, but the intermediate rotated tensor gets written out to HBM and read back , 64 MB of wasted memory traffic per call at N=65k, d=128. Two launches too. 2. Fused — one kernel. Rotated vector stays in shared memory, quantize reads it from there directly. HBM round-trip gone. 1.10–1.25× faster on quantize, 1.17–1.41× on dequant. Dequant benefits more because its per-coord compute is smaller, so saving the HBM trip is a bigger fraction of runtime. Hit a fun bug here. The fused kernel agreed bit-for-bit with the unfused version at b=1 and b=2, but differed in exactly 1 coord out of ~4 million at b=4. Cause: --use_fast_math was fusing (smem * scale) - codebook[k] into a single FMA, which rounds once instead of twice. At midpoint ties, that's enough to flip which centroid wins. Fix: pin the scale multiply with __fmul_rn. Bit-exact parity restored. (cc @tri_dao — the __fmul_rn / FMA rounding thing felt like exactly the kind of footgun you run into in FlashAttention territory. Curious whether you pin rounding explicitly at ops that matter or just test against a tolerance.) 3. Fused + inline PTX. Two experiments. One paid off massively, one did nothing: pack_signs with warp ballot (vote.sync.ballot.b32) — ~2.0× across every config. 32 threads each contribute one bit in unison via a warp-level primitive. No clean C++ form. bfi.b32 for bit-packing the quantized indices — zero speedup within noise. I checked the SASS and nvcc already emits BFI from the C++ pattern word |= (idx & MASK) << shift. The inline PTX was cosmetic. Takeaway: inline PTX only pays off when it exposes a hardware primitive C++ can't express. End-to-end on SIFT-1M (1M × 128 vectors, standard ANN benchmark): — 93% Recall@10 at 8× compression with fp32 rerank — Naive scalar quantization at same bits: 68% — At 16× compression, naive is essentially random (6%); TurboQuant still preserves 52% of true top-10 (@vikhyatk @StasBekman — tried to be careful separating the three "baselines" here: fp32 ceiling vs fp16 cast vs naive scalar at matched bits. If anyone spots methodology holes, I'd take the feedback.) Paper's predicted distortion for b={1,2,4}: {0.36, 0.117, 0.009}. I measured {0.361, 0.116, 0.00933} over 10 seeds. Sits right on the predicted curve. Full repo, all three implementations, reproducible benchmarks: → github.com/YuvaneshSankar/CU… @mirrokni

Core NVL72 performance benchmarks are undeniably imposing, but synthesizing 25x H100-level inference density inside a Rubin Space-1 orbital module is a quintessential architectural masterstroke. I channeled my extensive Rubin blog into demystifying the ground-state infrastructure—mapping out exactly how ConnectX-9 bypasses MoE bottlenecks, BlueField-4 optimizes packet flow, and the absolute interconnect superiority of NVLink 6. Even after releasing that meticulous teardown a month prior to GTC, its hardware forecasts remain profoundly unassailable. Still, contending with those zero-G thermal envelopes radically rewrites the deployment playbook. The energy-conservation architectures I spotlighted surpass trivial optimization; they function as absolute existential prerequisites in the cosmos. The vault from terrestrial server grids to microgravity operationalization signifies a tectonic inflection point in systems engineering. Which specific disclosure from the presentation genuinely left you speechless? yuvanesh.vercel.app/blogs/Ru…

Foundational NVL72 throughput statistics are awe-inspiring, yet integrating 25x H100-level inference density aboard a Rubin Space-1 orbital module stands as an unparalleled engineering triumph. I dedicated my comprehensive Rubin blog to decoding the foundational architecture—demonstrating precisely how ConnectX-9 neutralizes MoE bottlenecks, BlueField-4 streamlines data-path processing, and the sheer communicative supremacy of NVLink 6. Even after publishing that granular breakdown four weeks ahead of GTC, its architectural predictions remain remarkably unassailable. Regardless, navigating those aerospace-grade thermal limitations completely redefines the infrastructural blueprint. The power-efficiency protocols I outlined eclipse mere iterative upgrades; they operate as non-negotiable survival mandates in deep space. The leap from conventional data centers to orbital execution represents a seismic shift in computing evolution. Which particular revelation from the showcase unequivocally blew your mind? yuvanesh.vercel.app/blogs/Ru…

Base-level NVL72 bandwidth metrics are staggering, but embedding 25x H100-level inference density into a Rubin Space-1 orbital module is a supreme architectural feat. I devoted my exhaustive Rubin blog to parsing the underlying topology—illustrating how ConnectX-9 circumvents MoE bottlenecks, BlueField-4 mitigates computational overhead, and the absolute interconnect dominance of NVLink 6. Despite launching that intricate analysis a full month preceding GTC, its technical foresight remains flawlessly intact. Nevertheless, contending with those orbital thermal constraints irrevocably alters the hardware paradigm. The energy optimizations I documented transcend superficial enhancements; they function as absolute existential imperatives in the exosphere. The pivot from standard server farms to microgravity deployment signifies a monumental leap in systems engineering. Which specific disclosure from the keynote left you genuinely astounded? yuvanesh.vercel.app/blogs/Ru…

Fundamental NVL72 throughput figures are formidable, but shoehorning 25x H100-level inference capacity into a Rubin Space-1 orbital module is an absolute engineering marvel. I committed my definitive Rubin blog to deconstructing the foundational fabric—detailing the way ConnectX-9 obliterates MoE bottlenecks, BlueField-4 supercharges processing offloads, and the unadulterated computing supremacy of NVLink 6. Even after deploying that comprehensive breakdown four weeks prior to GTC, the forecast has aged immaculately. Regardless, facing those zero-gravity thermal limitations completely rewrites the infrastructural playbook. The power-saving maneuvers I spotlighted are vastly more than trivial design flexes; they are non-negotiable lifelines in orbit. The ascension from conventional data hubs to microgravity environments constitutes an earth-shattering technological triumph. Which exact segment of the presentation thoroughly stunned you? yuvanesh.vercel.app/blogs/Ru…"

Baseline NVL72 performance metrics are imposing, but packing 25x H100-level inference density inside a Rubin Space-1 orbital module is virtually unprecedented. I dedicated my flagship Rubin blog to dissecting the underlying architecture—specifically how ConnectX-9 dismantles MoE bottlenecks, BlueField-4 accelerates the offloads, and the sheer computational elegance of NVLink 6. Despite launching that technical deep-dive a month ahead of GTC, the analysis remains astonishingly prescient. Nevertheless, confronting those aerospace-grade power constraints entirely redefines the hardware landscape. The efficiency optimizations I highlighted transcend mere architectural perks; they are critical survival imperatives in the vacuum of space. The escalation from terrestrial data centers to low-Earth orbit represents an epochal engineering breakthrough. Which specific aspect of the keynote truly left you awestruck? yuvanesh.vercel.app/blogs/Ru…"

Standard-issue NVL72 benchmarks are authoritative, but compressing 25x H100-level inference into a Rubin Space-1 orbital module is borderline sci-fi. I centered my main Rubin blog on deconstructing the core topology—how ConnectX-9 annihilates MoE bottlenecks, BlueField-4 supercharges the offloads, and the raw genius of NVLink 6. Even though I published that teardown a full month before GTC, it has proven strikingly clairvoyant. However, reckoning with those space-grade power limitations radically alters the equation. The efficiency optimizations I emphasized are no longer optional luxuries; they are non-negotiable lifelines in orbit. The vault from surface-level server farms to the exosphere constitutes a monumental technical leap. Which exact detail from the keynote completely blew you away? yuvanesh.vercel.app/blogs/Ru…"

Conventional NVL72 metrics are commanding, but condensing 25x H100-level inference into a Rubin Space-1 orbital module is utterly mind-bending. I anchored my primary Rubin blog on unraveling the base architecture—how ConnectX-9 eradicates MoE bottlenecks, BlueField-4 expedites the offloads, and the sheer brilliance of NVLink 6. Even though I released that breakdown a month ahead of GTC, it has remained remarkably prophetic. Nevertheless, evaluating those space-grade power limitations fundamentally rewrites the rules. The efficiency optimizations I underscored are no longer trivial bonuses; they are absolute survival mechanisms in orbit. The leap from terrestrial data hubs to the exosphere constitutes a colossal engineering milestone. Which specific revelation from the keynote left you most astounded? yuvanesh.vercel.app/blogs/Ru…"

Standard NVL72 specifications are formidable, but fusing 25x H100-level inference within a Rubin Space-1 orbital module is profoundly ambitious. I focused my flagship Rubin blog on decoding the foundational infrastructure—how ConnectX-9 shatters MoE bottlenecks, BlueField-4 accelerates the offloads, and the undeniable sophistication of NVLink 6. Even though I published that deep-dive a month prior to GTC, it has proven exceptionally prescient. Still, grappling with those space-grade power constraints shifts the entire paradigm. The efficiency optimizations I highlighted are no longer mere conveniences; they are absolute necessities in orbit. The trajectory from conventional facilities to the exosphere is a groundbreaking structural triumph. Which exact announcement from the keynote blew your mind? yuvanesh.vercel.app/blogs/Ru…"