Inference scaling

0:00 It could take several months, a whole lot of GPUs that are used for training these. And then there's a fine-tuning step. Usually, this is a less or historically has been a less kind of compute-intense process. If you're talking about trillions of data tokens in the pre-training side, on the fine-tuning side, this is orders of magnitude less data that we use for fine-tuning. And then there is the inference part, where we get to use the model and we're going to see different ways that at inference time, we can make the model be better and become more useful without changing the parameters of the model and without any kind of fine-tuning of the models.

0:40 So, let's first start with the Large Language Monkey's paper again. You are familiar with the Infinite Monkey Theorem by now because we discussed it on Monday. But to recap, the idea of this paper is that we have an LLM and we have a problem as input. Instead of generating the response for this problem only once, we're going to repeatedly ask the same input problem again and again. This could be a number of times, it could be 10 times, 100 times and so on.

1:00 And we'll see how that affects the performance. And then we're going to have a verifier that can tell us which one of these responses were the correct response. And then that is going to be the output of this system. The reason we like that is that by doing this repeated sampling, we can make models that are inferior. For example, a llama 3B or 7B that is not as capable at GPT40 with single attempt. But by doing this repeated sampling and selecting the correct one among the generated candidates, we can significantly improve the performance of these inferior models and make them better than these larger and proprietary ones.

1:40 And these are here, you can see examples across really hard math, coding, and different styles of math problem, question answering problems. Is that, is this clear to everyone? It just seems like the models and smaller models already know the answers to these hard problems and we are just like by doing this repeated sampling, we are kind of eliciting those and surfacing those answers. It just, they don't, they they just don't tell us that in the first trial.

2:00 Now, this kind of paradigm of repeated sampling works well across pretty much any domain that we have tried in so far, including agentic benchmarks such as Swebench, which kind of mimics like how a software engineer goes about editing code and creating new patches. So here we are seeing the coverage, meaning the the fraction of problems that are solved by at least one of the samples.