Building on LLMs

0:00 And the first lecture, what we've covered is that large language models are really good at solving natural language processing problems. And I think most of you are very familiar with using them in chatbot kind of scenarios. But to make them useful in real-world tasks, they do need to be able to interact with real-world environments, tools, and code and learn from what they interact with. So you've already saw some examples of test-time compute and robust verification that Azalia covered in the last two lectures.

0:30 Now we'll build a little bit more on top of that. Okay, so today's lecture we will focus on three different papers. The first paper will start with the notion of tool calling and how can we get the models to go beyond just reasoning to also take into account actions via tool calls. And the paper that we'll cover here is called React, which actually was one of the first that went around to combine reasoning and acting in language models.

0:56 The second paper that will look into is RLHF or grounding code LLMs in execution feedback. What this will focus on is how do you how do you actually do code generation by using execution feedback from like being able to run the code or run the tasks of the code. And the third thing that we'll look into is constitutional AI where we use the model critiquing itself as a feedback loop to improve the model. So all three of these techniques are ways for the model to improve itself, but what differs in each of these techniques is where is the feedback coming from.

1:30 It's some form of interaction with either the environment or some form of critique mechanism or grounding using what we know should be the right answer. So let's start with just let's start with tool calling as the first abstraction.