Intro and setup

0:00 Welcome to Intro to Language Modeling. In this course, you're going to learn a lot of crazy stuff. Okay. I'm just going to give you a heads up. It's going to be a lot of crazy stuff we learn here. However, it will not be insanely hard. I don't expect you have any, any experience in calculus or linear algebra. Uh, a lot of courses out there do assume that, but I will not. We're going to build up from square one.

0:22 We're going to take baby steps when it comes to new, uh, fundamental concepts in math and machine learning, and we're going to take a larger steps once things are fairly clear and they're sort of easy to figure out, uh, that way we don't take forever just taking baby steps through every little concept. This course is inspired by Andre Karpathy's, uh, building a GPT from scratch lecture, so I'll shout out to him and yeah, we don't assume you have any experienced, maybe three months of Python experience, uh, just so the syntax is sort of familiar and you can, you're able to follow along that way, but, uh, no matter how smart you are, how quick you learn the willingness to put in the hours is the most important because this is material that you won't normally come across.

1:16 Um, so as long as you're able to put in that constant effort, uh, push through these lectures, even if it's hard, take a quick break, grab a snack, whatever you need to do, grab some water, water is very important. And yeah, hopefully you can make it to the end of this. You can do it, uh, since it's free code camp, everything will be local computation, nothing in the realm of paid data sets or cloud computing, uh, uh, we'll be scaling the data to 45 gigabytes for the entire, uh, training data set, so have 90 reserved so we can download the initial 45 and then convert it to an easier to work with 45.

2:05 So, uh, yeah, if you don't actually have 90 gigabytes reserve, that's totally fine, you can just download a different, uh, data sets and sort of follow the same, uh, data pipeline that I do in this video through the course, you may see me switch between macOS and windows, the code still works, all the same, both operating systems, and I'll be using a tool called SSH. It's a server that I can connect from my Mac book to my windows PC that I'm recording on right now, and that will allow me to execute, run, build, whatever, do anything coding related, uh, command prompt related on my, uh, Mac book, so I'll be able to do everything on there that I can, my windows computer, it'll just look a little bit different for the recording.

2:32 So, uh, why am I creating this course? Well, like I said before, a lot of beginners, they don't have the fundamental knowledge like calculus, linear algebra to help them get started or accelerate their learning in this space. So I intend to build up from baby steps and then larger steps when things are fairly simple to work with, and I'll use logic analogies and step-by-step examples to help consent conceptualize rather than just throw tons of formula at you.

2:58 So with that being said, let's go ahead and jump in to the good stuff. So in order to develop this project step-by-step, we're going to use something called Jupiter notebooks. And you can sort of play with these in the Anaconda prompt or at least launch them from here. So Anaconda prompt is just great for anything machine learning related. So make sure to have this installed. I will link a video in the description so that you can sort of set this up and install it step-by-step guide in there.

3:48 Um, so we can do from this point is sort of just set up our project and initialize everything. So I'm going to do is just, uh, head over into my directory that I want. Just going to be a Python testing. We're going to make a directory free code camp GPT course, and then from this point, uh, we're going to go and make a virtual environment. So virtual environment, it will, and initially in your desktop, you will have, uh, just all of your Python libraries, all your dependencies there, just floating around.

4:14 And what the virtual environment does is it sort of separates that. So you have this isolated environment over here, and you can just play around with this however you want, and it's completely separate. So that won't really, uh, cross with, uh, all of the global libraries that you have, all the ones that just affect the system when you're not in a virtual environment, if that makes sense. So we're going to go ahead and set that up right now by using Python, uh, dash M, and then we're going to go V and V for virtual V and V and then CUDA.

4:40 So the reason why we say CUDA here is because, uh, later when we, uh, try to accelerate our learning, uh, or the model's learning, uh, we're going to need to use GPUs, GPUs are going to accelerate this a ton and basically CUDA is just that little feature in the GPU that lets us do that. So we're going to make our environment called CUDA. I go and press enter. It's going to do that for, it's going to take a few seconds.

5:05 So now that that's done, we can go ahead and do CUDA and we're just going to basically activate this environment so we can start developing in it. We're going to go backslash and we're going to go scripts and then activate. So now you can see it says CUDA base. So we're in CUDA and then secondary base. So it's going to prioritize CUDA. So from this point, we can actually start installing some stuff, uh, some libraries here, so we can go pip three, install, uh, mat plot lib, numpy.

5:32 Uh, we're going to use P Y L M Z a L Z M a, and then what are some other ones? We're going to do IPY kernel. This is for the actual Jupiter notebooks and, uh, being able to bring the CUDA, uh, virtual environment into those notebooks. So that's why that's important. And then just the actual, uh, Jupiter notebook feature. So go ahead and press enter. Those are going to install. That's going to take a few seconds to do.

6:07 So it might actually happen is you'll get a build error with, uh, P Y L Z M a, which is a compression algorithm. And don't quote me on this, but I'm pretty sure it's based in C plus plus. So you actually need some build tools for this. And you can get that with, uh, visual studio build tools. So what you're, you might see, you might see a little error and basically go to that website and you're going to get this right here.

6:33 So just go ahead and download build tools. What's going to download here, you're going to click on that. It's going to, it's going to set up and then you're going to go ahead and click continue. And then at this point, uh, you can go ahead and click modify if you see this here, and then you might get to a little, uh, workloads section here. So once you're at workloads, that's good. What you're going to make sure is that you have, uh, these two checked off right here, just make sure that you have these two.

7:22 Um, I'm not sure what desktop particularly does. It might help, uh, but it's just kind of good to have, uh, some of these build tools on your PC anyways, even for future projects. So, uh, just get these two for now. That'll be good. And then you can click modify over here if you want it to modify, just like that. And then you should be good to, uh, rerun that command. Command. So from this point, what we can actually do is we're going to install torch and we're actually going to do it by using pip install, uh, three, install torch.

7:46 We're not going to do it like this. What we're actually going to do is we're going to use a separate command and this is going to install CUDA with our, uh, torch. So it's going to install the CUDA extension, which will allow us to utilize the GPU. So it's just this command right here. And if you want to find like, uh, a good command to use, what you can do is go to the, uh, pie torch docs, uh, just go to, go to get started and then, uh, you'll be able to see this right here.

8:10 So we have stable, uh, windows, pip, Python, then CUDA 11.7 or 11.8. So I just clicked on this and since we aren't going to be using, uh, torch vision or torch audio, I basically just did hip three, install torch, and then with this index URL for the, uh, CUDA 11.8. So that's pretty much all we're doing there to install CUDA. That's a part of our torch. So we can go ahead and click, uh, enter on this.

8:39 So great. We've installed a lot of things, uh, libraries, a lot of setup has been done already, uh, what I want to check now is just to make sure that our Python version is what we want. So Python version 3.10.9, that's great. Uh, if you're between 3.9, 3.10, 3.11, uh, that's perfect. So if you're in between those, it should be fine. Uh, at this point we can just jump right into our Jupyter Notebook. So the command for that is just Jupyter Notebook spelled like that, click enter.

9:06 It's going to send us into here and I've created this little bi-gram.ipynb here, uh, in my VS code. So, uh, pretty much you need to actually type some stuff in it and you need to make sure that it has the IPYNB, uh, extension or else it won't work. So if it's just IPYNB and doesn't have anything in it, uh, I can't really read that file for some reason. And yeah, so just, just make sure you type some stuff in it, open that in VS code.

10:01 Type like, I don't know, A equals three or STR equals banana. I don't care, uh, at this point, let's go ahead and pop into here. So this is what our Notebook's going to look like and we're going to be working with this quite a bit, uh, throughout this course. So what we're going to need to do next here is make sure that our virtual environment is actually inside of our Notebook and make sure that we can interact with it from this, uh, kernel rather than just through the command prompt.

10:28 So we're going to go ahead and check here and I have a virtual environment here. Uh, you may not, but all we're going to do is basically go into here. We're going to end this and all we're going to do is we're going to go ahead and do Python, uh, dash M and then IPY, uh, kernel install user. And you'll see why we're doing this in the second user name equals CUDA. This is from the virtual environment we initialized before.

11:04 So that's the name of the virtual environment. And then the display name, how it's actually going to look in the terminal is going to be, uh, display name. Uh, we'll just call it, um, CUDA GPT. I don't know. That sounds like a cool name and then we'll go and press enter. It's going to make this environment for us. Great installed. Good. So we can go and run our Notebook again and we'll see if this changes. So we can go ahead and pop into our BiGram again, kernel change, kernel, boom, CUDA GPT, let's click that sweet.

11:29 So now we can actually start, um, doing more and just sort of experimenting with, uh, how the notebooks work and actually how we can build up this BiGram model and sort of learning how, uh, language models work from scratch. So let's go ahead and do that. So before we jump into this actual code, we're going to go ahead and into this actual code here. What I want to do is, uh, delete all of these. Good.