Jakub Bartczuk

people keep saying AI is moving so fast. some days I agree, but some days I'm not sure – so many papers published, but I don't feel like we're making that many fundamental breakthroughs. to cap off 2023, here's a list of things we still don't know about language models: - how can we reliably answer questions from long texts, like books? - can we pretrain a world-class LM for less than a million dollars? - can we prevent a model from making up facts? - how can we "think more" for some inputs than others? - how can we train on both private and public data? - are our architectures (transformers) close to optimal? - are our parameters (found via SGD) close to optimal? (how far can a 7b model go?) - can we build systems that are resistant to jailbreaking and prompt inversion? - how can we tell when a model is unsure of what it's saying? - what's the most efficient way for two LMs to communicate with each other? - can we detect LM-generated text? - can we watermark LM-generated text for detection? - what do we do when we run out of internet text data to train on? (some folks think we're quite close) - how should we fairly evaluate trained models, if they've trained on all the text in the world? - what's the best way to input text into models? (tokenization is a hack) - how far can we shrink models down without losing performance? (lots of research in quantization but a long way to go) - how can we understand what models are doing under the hood? (what are they thinking about, how do they store information, etc.) - is next word prediction the best learning objective for AI? - is there a natural way to combine text inputs with other types of data (eg images, video, audio)? (...what did i miss?)