Tech Founder & Data Scientist. Interested in scientific innovation, computer science, machine learning, AI, statistics, history, astronomy, 🇮🇳 and green tech.
Gurgaon, India
Joined April 2009
- Tweets227K
- Following2.1K
- Followers1.5K
- Likes32K
Pinned Tweet
Thoughts on all the #AI doomsday scenario being discussed on the internet #existential #Crisis :
[TL,DR : AI is very far from being an existential threat, they just want regulatory capture ]
quora.com/What-are-some-pote…
Muktabh Mayank retweeted
Natural Hard-Negative Mining from Positives for Instruction-Following Retrieval
Builds instruction-following training data by promoting a retrieved negative via a new instruction, with the original positive as a hard negative.
📝 arxiv.org/abs/2610.36900
Muktabh Mayank retweeted
Cool! We derived an AdamW batch-size scaling rule in Lemma 3.13 of arxiv.org/abs/2411.15958
B → B × k
𝗟𝗥 → 𝗟𝗥 × sqrt(𝗸)
𝗪𝗗 → 𝗪𝗗 × sqrt(𝗸)
βᵢ → 1 − sqrt(k) × (1 − βᵢ)
This was subsequently validated empirically in an ICLR 2026 paper:
openreview.net/pdf?id=elB9k4…
I'm posting a series of prediction questions from Stanford CS312 (Deep Learning Alchemy) exams to test deep learning intuition.
First up: optimization. If we increase an LM's batch size 8× (same token budget), can rescaling learning rate or weight decay compensate? Try below ↓
This quiz is part of our optimizer hyperparameter assignment and unit designed by @wen_kaiyue
Setup: We train an 8-layer, ~35M-parameter Transformer on 614.4M tokens using AdamW. At batch size 64, the best reference run has peak LR 0.003, weight decay (WD) 0.2, and final validation loss 2.9178.
Now increase the batch size from 64 to 512, which gives 8× fewer optimizer steps. Everything else stays fixed except the changes below.
Which recipe gives the lowest final validation loss?
A. Keep LR and WD unchanged
B. LR ×√8
C. LR ×8
D. WD ×8
E. LR ×√8 and WD ×√8
What would you predict, and why?
Muktabh Mayank retweeted
We build a hierarchy in semantic space: text → semantic → pixel. AR learns what to draw; diffusion learns how to draw it.
Like LLMs: pretrain for knowledge, post-train for behavior.
Know, then Draw: A New Learning Paradigm for Visual Generation 🧵
arxiv.org/abs/2610.37626
Muktabh Mayank retweeted
Pagos.jl - play ice-sheet modelling like it’s Lego — Jan Swierczek-Jereczek
youtu.be/fBxHRr5CEh0
Muktabh Mayank retweeted
Every time I hear "10 sigma" event...
BREAKING: Brazil's -19.8% stock market crash on Monday was one of the rarest market events in the last two decades, according to our analysis.
Brazil's EWZ ETF plunged -19.8% on October 26th, marking an 8.4 standard deviation move.
For context, the ETF's average daily move over the last 20 years has been just +0.04%, with a standard deviation of 2.36%.
Under a normal distribution, the probability of a decline this extreme is approximately 1 in 58 quadrillion.
In other words, this event should be virtually impossible under conventional statistical models.
What just happened in Brazil was *almost* impossible.
Muktabh Mayank retweeted
When the Adept finds his/her path in life or True Will, there is a temptation to become lazy or to think that everything should be easy. There are times when you have to "do things you don't want to do" in order to make it possible to work on a project, or to gain knowledge. Often times these things can be mundane, repetitive tasks, or having to work some kind of job as a means to an end.
One must remember that there is something to learn from every task. As we become used to the struggle, we realize that these things weren't difficult to begin with, and after they are accomplished we have more time and energy to pursue our passions. The biggest thing holding people back from their life's passion or spiritual attainment is laziness and fear. Often these things we avoid doing are the things most necessary for our development.
True Will is a balance between work and reflection. Anything else is pointless. Never avoid the difficult task as those are the ones most likely to lead to illumination.
Muktabh Mayank retweeted
Computation-Friendly Compression of Matrices: Speeding up Linear and Boolean Algebra
Gonzalo Navarro
arxiv.org/abs/2610.35362 [𝚌𝚜.𝙳𝚂]
ALT Large sparse matrices—adjacency matrices of web and social graphs, feature matrices of machine-learning pipelines, the relations of graph databases—are routinely compressed to save space. Classical compressors, however, force the matrix to be decompressed before it can be used, so the space savings often yield a computational overhead. A recent line of work studies computation-friendly compression: formats that are not only small, but let us carry out matrix–vector products, and in some cases matrix–matrix products, sums and closures, directly on the compressed representation in time proportional to that compressed size. This survey organizes that literature around the kind of redundancy each format exploits: spatial clustering of the nonzeros (k²-trees and their variants), similarity between rows (Boldi–Vigna-style referencing and set-difference compression), repeated substrings (grammar compression), and dense bipartite substructures (bicliques). For each family we describe the repre
Muktabh Mayank retweeted
New preprint:
Cross-modal steering vectors transfer zero-shot in both directions 🖼️ ↔️ 📝
From text generation to image generation and vice-versa
Exciting evidence for a shared conceptual space!
Fantastic effort led by Donghao Zeng and awesome co-authors @nsubramani23 & @TamarRottShaham
Details 🧵👇
Muktabh Mayank retweeted
I'm doing a course based on my Cultures of Programming book! A perfect excuse to also record the material for anyone who prefers listening over reading.
Part 1: What can we learn from the 80 years of programming history & why Knight Capital lost $440m:
youtube.com/watch?v=Kjf6Min5…
Muktabh Mayank retweeted
I've been really fascinated by the DL Alchemy course (deep-learning-alchemy.github…).
The decision to put the burden of theory construction on the student (and their claude) feels transformative and surprisingly challenging. Maybe knowledge in ML was never that critical.
Muktabh Mayank retweeted
Memory Attention is now integrated into Flash Linear Attention (FLA)!
Huge thanks to @uniartisan and @yzhang_cs !
⚡ FLA: github.com/fla-org/flash-lin…
📄 Paper: arxiv.org/abs/2609.28399
Muktabh Mayank retweeted
Thermodynamic Space of Chemical Reaction Networks link.aps.org/doi/10.1103/g87…
Muktabh Mayank retweeted
India joins 130 countries calling US to remove sanctions over Cuba.
This is an annual vote, and this is the 34th year in which UN has voted to call on US to remove sanctions over Cuba.
Muktabh Mayank retweeted
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
Muktabh Mayank retweeted
📝Fresh from the press:
"Geometric Interpolation and Geometric Jensen-Shannon Divergence Between Gaussian Measures On Hilbert Space", joint work with Hà Quang Minh
IEEE Transactions on Information Theory.
Publisher:ieeexplore.ieee.org/document…
Arxiv:arxiv.org/abs/2506.10494
Muktabh Mayank retweeted
A zero gradient doesn’t always mean there’s nothing left to learn.
I’m happy to share PolyStep, now published in #TMLR. It explores a question I find fascinating: how do we train networks when hard spikes, quantization, or discrete routing leave us with little useful gradient information?
These operations can produce losses with flat regions and sudden jumps. A tiny parameter change may tell us almost nothing, while a larger one reveals a meaningful difference.
PolyStep works at that finite scale. It evaluates structured parameter probes along the vertices of a rotated polytope, then uses the observed losses to weight the probe directions and take a step. The weights come from an optimal transport formulation, with a simple softmax solution in the main method. Training uses only forward passes.
The intuition I find most useful is that the scale at which we look determines which learning signals become visible. Probe radius controls what the optimizer can see, temperature controls how strongly it responds, and step scale controls how far it moves.
The PolyStep repo includes a small synthetic classification example with hard LIF spikes and no surrogate gradients, alongside CartPole policy search that learns from episode returns without gradients through the actions or simulator.
There is a practical tradeoff: those forward evaluations cost compute, and backpropagation is usually faster when useful gradients are available. I’m especially interested in the cases where hard operations are part of the behavior we want to preserve during training.
Many thanks to Viet T. Nguyen for the lovely interactive visualization that makes the geometry much easier to see.
If you’re working with spiking models, quantization, discrete routing, or simulators, I’d love to hear what you try, what works, and where it gets stuck. Questions, feedback, and contributions are very welcome.
Code: github.com/anindex/polystep
Paper: openreview.net/forum?id=8mlc…
Interactive visualization: vietngth.github.io/polystep-…
Muktabh Mayank retweeted
Expository article: "Exploring Neural Dynamics with Time Delays: A Numerical and Bifurcation Analysis Approach" (by Martin R Nelson, Jonathan J Crofts): arxiv.org/abs/2610.36920
