Pablo Patarca

The future of AI language models may lie in predicting beyond the next word: Multi-Token Prediction Studies suggest that the human brain predicts multiple words at once when understanding language, utilizing both semantic and syntactic information for broader predictions - now researchers from Meta are hoping to train their LLMs to do the same. The authors proposed a new training method for language models, called "multi-token prediction," which predicts multiple words simultaneously instead of just the next word. "Our 13B parameter models solves 12% more problems on HumanEval [benchmark test] and 17% more on MBPP than comparable next-token models." Predicting hierarchical representations of future input and generating a multi-token response enhances performance, coherence, and reasoning capabilities (particularly for larger models) as it attempts to mimic the human brain. Works particularly well with coding, and may become a key feature of advanced language models to be released later this year that include the number "5". Just speculation.