Difan Zou

Some recent discoveries with my student! We try to perform some "simple scientific understandings" regarding the LLM training and will release a series of blog!

We open sourced VALG, an agentic system for automated ML theory research that turns the real "iterate and revise" rhythm of theoretical work into a structured workflow. A key feature is joint problem solving and problem (re)formulation. VALG iterates between adjusting assumptions and deriving proofs in a unified loop. When a theorem cannot be fully established, it produces a well scoped relaxation or special case while keeping the mathematical link to the original question explicit. We tested it on 5 COLT 2026 open problems, running 9 subproblems and producing 22 theorem candidates. Two were fully solved, and the remaining seven were partially resolved via relaxations or restricted settings. We are also building a benchmark for ML theory automation, with initial results suggesting around 80% of arXiv 2026 theory papers can be handled end to end. Future directions include human AI proof alignment and systematic data collection. That said, AI is not replacing researchers yet. Problem formulation, key constraints, and final delivery still require human judgment. VALG helps with rapid validation and identifying which directions are worth pursuing and can give positive outcome. Try it, file issues, and help us improve. GitHub: github.com/DechenZhang/ml-th… arXiv: arxiv.org/pdf/2608.13060

Excited to introduce our recent work on deriving the scaling laws for precision in high-dimensional linear regression!! Low-precision training is standard for efficient LLMs, yet the theoretical mechanisms governing its scaling have remained unclear. In our new work, we introduce a unified theoretical framework based on high-dimensional sketched linear regression (see arxiv.org/abs/2406.08466) to derive precise scaling laws . By modeling quantization as a spectral transformation, we follow our previous work on low-precision training for linear regression problems (see arxiv.org/pdf/2510.18259), derive the excess risk bound (both lower and upper) as a function of model size, dataset size, and quantization errors, which bears similarity with the empirically found scaling law for precision in arxiv.org/abs/2411.04330. Technically, our result reveals a fundamental dichotomy for two types of quantization: Multiplicative Quantization (e.g., FP8): Our theory shows this acts as signal-dependent noise but preserves the effective model capacity (size of the model). This explains the empirical robustness of floating-point formats during scaling. Additive Quantization (e.g., INT8): In contrast, this scheme acts as a signal-independent floor that strictly reduces model capacity. It effectively "flattens" the spectral tail, rendering those parameters useless for learning. Despite these differences, we prove that both schemes degrade the effective dataset size ($N_{\text{eff}}$) through noise amplification. Future Work: The low-precision trainingdepends on the optimizers; we are working on understanding and developing theory for various optimization algorithms (e.g., Adam, Muon, etc) under the low-precision quantization regime! Open to any discussion on the related topics! Paper link: arxiv.org/pdf/2602.19241 Joint work with Dechen Zhang, Xuan Tang, and Yingyu Liang

We are excited to share our latest work on the implicit bias of stochastic steepest descent! While optimizers like Adam and Muon, variants of steepest descent under different norms, are popular for large-scale pretraining, their theoretical behavior under mini-batch stochastic gradients has remained elusive. We provide a unified analysis of how batch size, momentum, and variance reduction shape the solutions these algorithms find. Key findings: 🔹 No Momentum: Convergence to the (approximated) max-margin solution requires large batches. Small batches fail to recover the full-batch implicit bias. 🔹 Momentum: Acts as a stabilizer! It enables convergence to the max-margin solution even with small batches, though at a slower rate. 🔹 Variance Reduction: A better fix, it recovers the exact full-batch implicit bias regardless of batch size or momentum. 🔹 Small Batch Size (B=1): In the extreme case (per-sample updates), the implicit bias fundamentally changes, which provably cannot be explained by standard max-margin solutions. We believe thoroughly understanding these algorithms in theory is crucial for developing the foundation of large model training, and our work provides the initial attempts along this line. Paper: arxiv.org/abs/2602.11557 Joint work with Jichu Li (fantastic undergrad student) and Xuan Tang (genius grad student).

The course will add more advanced topics next semester, including generative AI and some in-context learning and reasoning studies! All course materials will be released to the public.

Joint work with Miao Lu, Beining Wu, and Xiaodong Yang. We discuss the behavior of the oscillation of SGD and its effect on the generalization. Comments are welcomed!

Looking forward to presenting my ICML paper (joint work with @sfrei_ and @QuanquanGu) on the agnostic learning of robust half-spaces via adversarial training algorithms icml.cc/virtual/2021/poster/…. Welcome to Poster Session 3 (room 5) tomorrow to discuss and share your thoughts!

Always coffee + diet coke

Really excited to be part of this project!

Welcome to our website covid19.uclaml.org for more results!

I am presenting my paper "On the Global Convergence of Training Deep Linear ResNets" on zoom now. If you are interested, please join and share your thoughts. #ICLR2020 @QuanquanGu @philipmlong iclr.cc/virtual/poster_HJxEh…

We have made a prediction about the peak of active cases in LA, it shows that the peak will be at 5/24/2020 based on the data before 4/9/2020. More detail can be found in covid19.uclaml.org.

How to efficiently train deep and large graph convolutional networks? Come and check our #NeurIPS2019 poster tomorrow (Thu 10:45 AM @ East Exhibition Hall B + C #63) on the efficient training of GCN. Joint work with Ziniu Hu, Yewen Wang, Song Jiang, @YizhouSun, and @QuanquanGu.

Come and check our #NeurIPS2019 poster tomorrow (5:30 PM @ East Exhibition Hall B + C #160) on Hamiltonian Monte Carlo. We propose a new sampling algorithm that provably converges faster than all existing HMC-type algorithms. Joint work with @PanXu_CS and @QuanquanGu.

Come and check our #NeurIPS2019 poster tomorrow (10:45 AM, East Exhibition Hall B + C #135) on training over-parameterized DNNs. We significantly improve the previous results in terms of both over-parameterization conditions and convergence rates. Joint work with @QuanquanGu.

go wahoo go!