Adam Visokay

Machine Learning is not just (glorified) statistics! Sure, in its most basic form, ML doesn't seem that far removed from fancy statistics... But ML is its own thing! Steve Brunton, @eigensteve of the University of Washington, in his fabulous short lecture series on probability and statistics (look for it on YouTube), gives these clear-eyed definitions: Probability: Assuming you have a known probability distribution that describes your data (say, a Gaussian), probability theory allows you to say something about samples of data you might observe in the future. So, one can calculate the probability that a random variable, X, takes on some value, given the parameter theta that describes the probability distribution (theta for a gaussian would be the mean and variance). Statistics: It's the flip side of the above. Now, we have in hand the data or samples. We are trying to say something about the probability distribution that best models the data; or given the data we want to say something about the probability of the parameter theta of the distribution Machine Learning / Deep Learning: Sometimes, we have the data, but the underlying distribution is unknown, or impossible to characterize analytically, using parameters specified by some theta. ML is used to learn the (empirical) distribution by examining the data. This is the point at which someone will say, but, that is just statistics! Here's why it's not: Take the problem of learning the probability distribution over the millions of images of the natural world. And not just that: once you have learned the distribution, you have to sample from it, and generate new data that looks like an image from the original dataset. Try doing this with standard statistical methods in some tractable way. It's near impossible. But diffusion models, heavily influenced by the physics of non-equilibrium thermodynamics, do exactly that. That's deep learning at its best. Also, take large language models (LLMs). Of course, LLMs learn the statistics of human written language from an enormous corpus of training data, and generate new text by estimating and then sampling from conditional probability distributions, given some input text. Again, while it sounds like it's just statistics, standard statistical methods could not pull off what an LLM can. The chapter on probability and statistics in WHY MACHINES LEARN was one of the most difficult to write (it involved getting across the basics of two large fields of math and tying them to ML, all in the space of 30 pages). Nearly killed me. But it was also one of the most satisfying chapters to write, not least because I got to understand so much while trying to make sense of these issues. More here: US: penguinrandomhouse.com/books… UK: penguin.co.uk/books/446849/w…

Important topic, but this is more of a quick-start guide. For cutting-edge research on LLM evals, see these papers using Prediction-Powered Inference to incorporate synthetic data and model predictions for narrower CIs. 👇 Gemini already knows about them! arxiv.org/abs/2403.07008 arxiv.org/abs/2406.04291