Dawid van Straaten

More thoughts on Jev. 1. It cannot be faster to decision than an LLM and be as good. The time an LLM takes until it prints the first token of the answer is equivalent to the compute spent on making a decision. So, if Jev takes less time to output its full output than it takes for an LLM of equivalent size to print the first token, Jev automatically loses in quality. 2. You can get the same classification speed from an LLM if you do the following: Let's say you have 3 options you want to evaluate. 1) Prepare 3 separate requests to an LLM, where each request has the same question and only a single option of the three. For example: - "What color is this dress? Output a single number between 0 and 9 to tell how likely it is that it's RED." - "What color is this dress? Output a single number between 0 and 9 to tell how likely it is that it's GREEN." - "What color is this dress? Output a single number between 0 and 9 to tell how likely it is that it's BLUE." The part right before the COLOR ("What color is this dress? Output a single number between 0 and 9 to tell how likely it is that it's") is the same and is in the KV cache. Then, in parallel, you send only the COLOR word to the LLM and only get from it the very first output token, which will be a number between 0 and 9. 2) Now, you have three numbers, say [2, 3, 9]. You use softmax to convert them to probabilities: [0.09%, 0.25%, 99.66%] If you call the LLM with thinking disabled, you get your prediction time as the time of the embedding of the common part of the payload (very fast) + the addition of a single answer option to the KV cache (very fast) + generation of a single-digit output (very fast). If you enable the thinking, it will take more time to generate the thinking tokens, but the quality will be way better than Jev.

We have a very poor understanding of why deep neural networks like transformer models learn the parameters they learn. For example, in the paper below from 2013, the authors demonstrated that 5% of the weights of a trained deep neural network can be used to predict the values of the remaining 95% of trained weights. In another result from 2019 (arxiv.org/abs/1803.03635), the authors show that they can delete ("prune") 80% of parameters of a trained deep neural network, and it will keep generating almost identical outputs. In this recent result, the authors train a better quality model by strategically removing 2/3 of the training dataset: arxiv.org/pdf/2309.04564 We don't know much about why the model learned what it learned. All the tricks that work in practice were invented by pure chance, and in very rare cases do they get a theoretical explanation and some sort of theory that can be used to optimize the trick's parameters for an optimal model training outcome. Furthermore, training and running the deep model is prohibitively expensive in terms of energy consumption. Our brain consumes a million times less energy per decision compared to running an LLM while being much more reliable. And we have no idea how to reduce this difference. When scaling no longer works, answering these questions and addressing these blank spots is where the key to better AI might be.