R1 overview and the R1-Zero path

0:00 learn about deep seek r1's Innovative AI architecture in this course taught by Yassin a published researcher and experienced data scientist this course explores how R1 achieves exceptional reasoning through reinforcement learning focusing on group relative policy optimization or grpo and how it improves upon traditional po methods you'll also understand kale divergences role in model stability with practical code examples and clear mathematical explanations you probably heard of deepy Goan a new open source reasoning model released in January 2025 that reveals the o1 series of reasoning models in this tutal we'll go through the theory and lying the methodology in depth the major significance of dcan is that it's effectively an open- Source implementation of oan from open AI this is a breakthrough because the methodology for reasoning model was closed Source until now the from Deep seek as you're going to see is surprisingly simple and elegant to understand the paper fully we will be going through three level of details first an overview of the whole paper

1:09 methodology which include the reinforced learning the data augmentation and distillation portion then a deep dive into grpo or group relative policy optimization and its implementation in an open- Source reproduction from Ugg phase finally I even deeper dive into the K Divergence penalty term that is used within grpo that has a very particular shape at the end of this joural you should have more than a good enough appreciation of how the researchers at Deep seek and open AI managed to create reasoning models we're going to go Fairly in depth here maybe maybe a bit too much so don't hesitate to pause the video and reread the paper to really absorb all of the information let's get to [Music] it let's start with an overview of the mety the used I found out a really nice map that shows visually how the me is structured because it can be a bit difficult to follow which model is doing what so if we just take a broad overview of what's going on we start with the base model here deep seek V3 base and the whole output is to generate these three things so deeps r10 DEC R1 distilled and decr1 these models are

2:20 reasoning models um like uh the 01 model from open AI so if we dive into the result here uh you can see here DEC car one is this big blue bar and you have like these six Benchmark and the the tldr here is that it's very close to open AI 01 so close to open1 that you could argue that it kind of reproduced it a bit except on this one um but it's very close in in general right and it's better than like just the deeps V3 uh which is was the model that is based upon so the reasoning that we're adding on top of deep secq tree is helping in all of these different complex benchmarks one thing to note here is that this is mostly just doing posttraining it's not doing pre-training U it's using the pre-trained model deeps V3 which uh was uh recently put out right so like when you look around and see like uh the different tech company kind of freaking out about uh deep seek it's not like the deeps R1 that they're freaking out it's it's literally this one the d v tree which is used as the base uh reproducing the r10 result is not that uh complicated that much um

3:33 it's this one that is a bit more complicated to reproduce um and where is um um a lot of the the hype is around so we're not going to go into deeps V3 but like know that this is a mixture of expert uh model um that is uh also open source let's start our um little um uh walk through this paper so we're going to start with a deep seek V3 base and then we're going to go over here right we're going to go into this little block and then we're going to do the reasoning oriented reinforcement learning right to learn uh reasoning not through like a supervised fine-tuning uh with human feedback and whatever just with the reinforcement learning so just with this and proper reword functions the researchers at Deep seek managed to get this model to uh becoming a reasoning model on par with o1 this is what they did right so let's take a look at it it there's two block that are important here there's U the reward function that they're using right the reward system and then there's this thing the grpo that we were're calling both of these things play together in order to make it

4:44 so that we don't need human in the loop or more data here to do what we need to do so the group relative uh policy optimization is not coming from this paper actually the idea came a bit earlier and this Auto paper so deep seek mat limit of mathematical reasoning and open language model so this was in April 2024 um and you see here deeps matth is uh better at that this particular math benchmark so I think that they see seen what they did with group relative uh policy optimization on the matte Benchmark and then they were like hey we could do so much more uh with it so DC car10 was uh built out um we're not going to go into this paper either but like you know that this is not a new idea it's coming from here and then uh it's very useful so what's going on is