Course recap

0:00 Okay, so we will, uh, start with a, with an overview of what we have covered, um, in this quarter. So we started the class with, uh, just giving you an overview of LLMs and how, uh, over the past, uh, year, test time scaling and train time scaling using self-improvement techniques where the fundamental loop is driven by verifiers and feedback, uh, from, uh, running goes verifiers and getting rewards and then using reinforcement learning and even search algorithms to, to help the models to hill climb on math and coding, for example, have been a lot of the focus of initial part of the class.

0:29 Uh, from there, we started talking about evolutionary strategies. So there was a whole class on open-endedness, where we're not just, uh, uh, driving self-improvement loop with, um, rewards, we also allow the models to just go explore if the, exploration space can be defined. And here we discussed Alpha Evolf and, uh, similar techniques. Then we moved, uh, to, to a paradigm where we build end-to-end workflows for, uh, agents using tool use, and, uh, as the models interact with environments and tools, uh, you basically are able to drive the workflows end to end.

0:58 And, um, if you imagine that, uh, you have tasks that need to be completed over multiple steps, then you do need to have, perhaps the ability to look at knowledge bases, so that requires, uh, retrieval and memories. So basically, um, as we go towards, uh, real-world workflows, um, we need, uh, better planning and multi-step reasoning in these models. So, we, we discussed, uh, some works around that, and, um, how to really look at, uh, evaluations in the next generation of, uh, models and what, uh, capabilities would drive that.

1:19 And then we had several guest lectures on post-training and multiple model agents and robotics and reasoning and so on. Asari, is there anything you would like to add here? Yeah, I think, I think this sounds good. This was full coverage. I guess, so one of our, uh, guest speakers also talked about, um, the symbolic techniques, which goes under the umbrella of tool use and how that could add to the synthetic data, which is another trend we're seeing.

1:47 Amazing. Thank you. So, uh, so I think just to, uh, recap and end this, uh, the class is named as self-improving agents. So it's not just focusing on LLMs, it's focusing on the agent aspect of it. What, uh, roughly, if I were to, if you were to remind you of what we discussed in first class, uh, is that what how is an agent, a generalization of an LLM? It has a goal, and it will go and interact with the environment, collect feedback, and use that feedback to correct its steps.

1:58 So they're basically systems that can direct their own processes, use tools, and then accomplish a goal. And oftentimes, in the current paradigm, uh, the LLMs, uh, by themselves are not powerful enough to drive towards a full goal. Uh, they're getting there. So often times we would handwrite these, uh, systems as workflows where you are orchestrating LLMs and tools. Uh, however, in certain scenarios, uh, certain scenarios, like coding agents, you are seeing some of these, uh, agentic workflows being driven by themselves.

2:17 So, it's an exciting time where we are seeing a lot of progress in this area.