A learning objective is written as an expectation over a distribution p: J = 𝔼_{x ∼ p(x)} f(x). A rule of thumb is to replace the expectation with the mean of a mini batch of N samples {xᵢ} from p: 1/N ∑ᵢ f(xᵢ). This works great if p is given; however problems occur if p itself is to be learned, because sampling is not naively differentiable. There are several existing workarounds: 1. The re-parameterization trick, best represented by VAEs and GANs. The idea is to enable gradients in samples via the transform: x ∼ p(x) -> z ∼ p(z), x = g(z), where p(z) is simple&known, and g(z) is differentiable. Further assuming f is differentiable, the empirical objective becomes 1/N ∑ᵢ f(g(zᵢ)) which can be trained with SGD. 2. Straight through gradient/continuous relaxation of discrete variables (via eg, Gumbel softmax), which also creates gradients for samples. A good example is the (two ways of) training a VQVAE. 3. The policy gradient trick. Instead of approximating the objective, one approximates its gradient: ∇J = ∫ₓ ∇p(x) f(x) dx = 𝔼_{x ∼ p(x)} ∇logp(x) f(x). And again, the expectation can be approximated by 1/N ∑ᵢ ∇logp(xᵢ) f(xᵢ). This works the best if f is given and known, and/or x is discrete. It's also useful to note that the list above follows an increasing order of learning difficulty. A funny example: you can in principle apply policy gradient to the learning of a regular VAE, but it would be much less efficient/effective than the standard re-parameterization based implementation (this was actually done in this old paper arxiv.org/abs/1402.0030).