Alessio Devoto

interesting paper tacking the following question: "how do I squeeze the most generalization out of a fixed dataset?" in their setup data is the bottleneck, and compute is infinite -- contrast this with Chinchilla, which operates under a fixed training compute budget, optimizing for the lowest possible loss; conclusion was that 20 TPP (token per param) is optimal (in their setup). if number of params / inference is the constraint instead optimal TPP can go much higher (e.g. 2000 TPP), i.e. the "overtrained" regime -- conclusions: 1. repeat data + stronger regularization, weight decay should be 30x larger than standard (~3 vs 0.1), this allows N (num params) -> inf and still monotonic loss 2. ensemble scaling: instead of spending all compute on one larger model, train multiple independent models on the same data and average their predictions. this achieves a lower loss asymptote than parameter scaling alone 3. distillation: generate synthetic data from the ensemble and train a smaller student from scratch, retaining ~83% of the ensemble's improvement at 8x fewer parameters combined, in their setup these techniques yield an estimated 5.17x improvement in data efficiency important caveats: the main experiments are relatively small-scale (200M unique tokens, models up to 1.4B params), and the 5.17x improvement is based on extrapolated scaling laws