Intro

0:00 welcome to this Cuda programming course where you will learn to leverage gpus for high performance Computing the course starts with an overview of the deep learning ecosystem and guides you through setting up Cuda and reviewing essential C and C++ Concepts you'll explore GPU architecture and write your first Cuda kernels Advanced topics include optimizing matrix multiplication and extending pie torch with practical applications like implementing a multi-layer perception for the mest data set Elliot Alid created this course so what is Cuda or compute unified device architecture by Nvidia uh my name is Elliot and I'm an instructor on free code Camp as well as a student studying for my computer science degree so in this course I bring to you Cuda for deep learning but don't let that repel you if you're not in deep learning because there's still a lot that we're going to be able to cover uh many other fields of parallel programming so this is more oriented for deep learning but not specifically aimed at it um there's going to be a lot covered here so uh

1:10 I'll show what the final prodject what the final project is first um so that you can get a feel forward and see kind of what we're going to end up building by the end um and then we'll just kind of go from there so before we get started with anything crazy I should include a disclaimer um this course may not be fully up to date by the time you're watching this if you're watching this 10 years years down the line from when I've released it it might not all be the same there might be things that are updated the new uh compute capabilities might be you know way better there might be a bunch of different stuff happening so I'm not too sure where the ecosystem will be at in 10 years but as of 2024 this is pretty much the best you're going to get so just trying to include that and I thought I'd try to make everything uh not entirely centered around time so you can go back into this version uh or or

1:57 certain Cuda versions and reproduce all the same stuff it just might be a little bit different down the line if you're watching this later on so why did I create this course exactly well a lot of these performance and kernel engineering jobs require a lot of knowledge they require a lot of experience in the industry uh and it's just really hard to get up to that point where you're able to compete with the top and the best of the best performance Engineers so these are the people that are writing the training runs for like gbt 4 gbt 5 all of this um you need a lot of skill to optimize a massive neural network training run and inference on a larg data center or compute cluster so this aims to prevent some of that manual weaving on your part still encouraging you to do so U on your own but prevent some of that hardcore labor of going through and really figuring things out on your own from scratch uh that's one of the reasons why I created this another one is like generally speaking the point of writing GPU kernels or playing with code at all on the GPU is to run something faster so if you have a nested Loop um you know it's like 4 I in range 4J in range four 4K in range

3:08 whatever however many you want to put uh essentially what parallel programming and Cuda allow us to do is unroll those so if you take like for example four ion range you could take each little thing in that and run that instruction on a different CA cor so if you have 10,000 cacor and you have 10,000 different iterations in your Loop then you can affect L do each iteration in a single instruction or a single thread on on the GPU so this is some of the things that allows us to do you're going to use your job of uh you're going to use your your knowledge of GPU architecture kernel launch configurations and a bunch of other cool stuff we end up learning in this course to make that code run as fast as possible uh and then the last one is really there's so much data nowadays they say we have way too much data but very little cleaned data I've taken everything from all the other video courses everything on the internet and YouTube uh and I put them in a single course so I filtered out a bunch of the nonsense a lot of you know the old stuff a lot of the new stuff that maybe isn't covered as well and kind of just projected into this one Masterpiece so this includes topics covered by paid

4:19 courses as well I haven't actually paid for them but I kind of just looked at you know what are the chapters that they cover and then include some of those important Concepts in this course um I do have links for YouTube videos and all of these resources which I've gone through only the high quality ones but I've gone through a lot of these videos and resources and these are all going to be uh put in links inside of the um GitHub Link in the description so everything you need is going to be there um and I put a lot of all of those links in that um in that link so what are some use cases for Cuda parallel GPU programming what are some of the use cases for this well you have graphics and rate tracing so the computer Graphics that you're seeing in video games um you know user interfaces all of this you have fluid simulation for like physics um and modeling you know engine

5:06 Dynamics you have video editing so the video that I'm editing for this right now is using uh parallel Computing to render uh crypto mining which a lot of you might be doing already that's going to be using uh you know your GPU hardware and some of the advantages of that to like mine through the the crypto mining problems and then you have 3D modeling and software like blender so when you have a bunch of different points going on and you have to render things it's essentially the same as video editing but just um 3D instead of 2D so the last one which you probably guessed it already is deep learning so the number one use case for Cuda right now is primarily what I'll be covering in this course which is deep learning so we're not going to go as as deep into like say convolutions but uh to kind of understand how to optimize an algorithm like matrix multiplication uh we're going to go quite in depth with that so now you might ask Elliot what are the requirements or the prerequisites for this course so there are some that are more intellectual and academic and there are some that aren't so this is strictly for NVIDIA gpus in case you didn't catch on to that earlier um if you don't have one you can always consider renting uh

6:17 the cheapest ones in the cloud um I advise you to look into the pricing before giving a definite no on the pricing for some of these Cloud gpus um at first I was actually surprised how low the cost was for some cloud instances um especially the non-comp compute demanding ones so if you have like only a CPU or like a ram intensive machine it might actually cost significantly less than one with gpus on it um the gpus one are still very cheap you can use things like vast AI which I'll cover a little bit more um you can use this for actually getting really cheap uh consumer grade Hardware that you can SSH into in the cloud um and then just do all of your experiments and go through the course on that you can continue uh you can continue running with any you know NVIDIA GTX RTX or data center level gpus so all of the Nvidia cards are pretty much supported for this uh maybe like the lower ones that are like 15 years old those might not work um but generally if you have like a GTX like 1660 or something like that it's like it's going to be fine um as for course prequisites Python Programming will help in understanding

7:26 while we're implementing in lower languages so um just understanding the whole programming uh Concepts is really what's going to be needed here again all these different languages is just like a change in syntax right so um you know we're going to use basic differentiation and Vector calculus uh that'll make learning easier if you know it already um it's really only required for intuition behind back propagation and some of the stuff we're going to use to build neural networks from scratch um linear algebra will definitely make your life easier by not having to learn fundamental algorithms from scratch so like if you're not really intuitively um you know into matrix multiplication yet if you haven't really uh you know gone into that extensively it might be a little hard for you to catch up uh but matrix multiplication is very easy it's quite trivial in retrospect it's very it's very easy to understand um but just the intuition there and optimizing it might be a little hard if you haven't worked with it a lot already um then if you really care I would recommend just reviewing you know Matrix uh transpose matrix multiplication chain rule from calculus and then difference between gradients and derivatives um

8:39 there's maybe a few more that I missed but those are like the general ideas that you're going to need for going into this um and then just a heads up uh if you are in a Windows machine this might be a little harder for you so I do have a little setup guide on Windows Hardware um but I I do everything here on on Ubuntu Linux so this is what I'm running uh just on my local machine here and this is what we're going to go through the course with um you can always use uh WSL on Windows to simulate a a Linux system or you can use uh Docker so Docker is an awesome tool that'll allow you to essentially fire up uh a little simulated Linux machine uh just in your terminal on Windows and you can just do everything through that uh I think it supports Nvidia gpus directly through Windows I'm not entirely sure yet I haven't tested that but um if you're on a Windows machine machine I would recommend uh WSL or Docker if you do run into errors or issues throughout this uh I do suggest you you check GitHub stack Overflow Nvidia developer forums pytorch docks uh if your issu is related to any of this course material so you know you have a

9:47 lot of resources at your disposal if you need to resolve an error that doesn't come up in the course material uh you also have really powerful language models to use there's a lot of language models that have been released recently that are really really good at solving and addressing coding problem s so I do suggest you try those out um if all if all goes wrong right um all the all the code and notes for this are kept in the GitHub repo in the description the ecosystem is going to change all the time so in case this video isn't up to date uh the GitHub repo will be because I'm able to push that and actually make changes so if something is a little off in here you might want to go check in the repo and see like what it actually looks like so that you can actually write it properly and maybe there's a more optimized version thing things will change but you get the point uh I do suggest following uh the repo for

10:34 maintaining a structured learning approach I include excal draw diagrams so this is going to help illustrate like high level ideas how we're going to approach things uh as well as how to do things on the level of Kernel optimization so all the way top down all of it excal draw is awesome for illustrating things and it's completely free so all the diagrams there will be included in the in the GitHub repo uh and in the course too um you know you can always uh reach out to me through my Discord server which will also be in the GitHub repo um and you can reach out to me through there and talk with the community there's going to be a lot of other students learning there's going to be a dedicated set of channels for this so in case you get stuck or wanted to discuss something or just have a cool chat in the server uh you can totally join that I do want

11:22 to note early on that this course isn't on Cuda only so there's a few things that I cover outside of it including pytorch referencing uh going into like Triton and c and C++ with like externally not including Cuda just to you know help illustrate things on how how that the naive version of an algorithm works but uh so there's there's the code side and then there's also um I'm going to provide some prerequisites or not even prerequisites but rather just uh a good understanding about the whole deep learning ecosystem so this is actually what one of the next chapters is going to be about is how does the whole ecosystem work and where can I apply Cuda it would be a little silly of me to say here's how you optimize a kernel and make it run really really fast on your Hardware but not actually give you some solid use cases for that so you might already know what the use case is but in case you're just trying to learn Cuda and you might look at some ways that you can apply it I provide that Resource as well so spoiler alert but some takeaways you might get from this course is that through experiment experimentation and research you'll learn that the main GPU performance bottleneck is memory

12:32 bandwidth so in deep learning we have these giant inscrutable matrices that cannot fit into the onchip memory at once so think about if you have like a giant cluster of gpus and each of them have really really fast tensor cores these are like super optimized for doing you know tensor operations in deep learning um but if you're doing these across many gpus you really have to exchange and and and mix and manage information between them so you end up sending electrons uh you know from this node to this node to this node to right and there's a lot of this communication that's going on so you really get a ton of speed from the compute inside of the chips but when it comes to communicating there's actually a a pretty big bottleneck there and that's you know one thing that you might take away from this um there's also on chip constraints too so you have like GPU vram which is going to be uh you know comparatively slow to what the on chip stuff is so vram is like off the actual you know cores and all this and then it has to communicate with the cores and all the the the shared memory on chip and all the

13:41 registers and that ends up being a ball neck too so it's not just the the massive um the massive matrices communicating across a lot of gpus it's actually a lot of the onchip communication too so there's multiple bottleneck that that's arise or that that arise um but these are just things that you'll end up coming across and and being able to address later on through optimizations another key takeaway is would be to take an existing implementation and make it faster so a lot of the times you'll see a new research paper come out and you'll see a really cool algorithm but you might not know exactly how it works and so or or you maybe know maybe you know how it works and you just want to make it fast and you want to integrate it into Pi torch for example so this is something we're actually going to do in this course is we're going to uh we're going to build up uh an algorithm and we're

14:29 going to optimize it and then we're GNA actually Port it into a pytorch extension so that you can call it in Python which is super cool um but just learning how to integrate your own research into things to make them faster to have it operate at production scale um these are some really important things that you'll have to do when you start working you know very deeply with Cuda um another thing is karpathy LL M.C a lot of you have probably heard of this um if you go search up LL M.C uh LL M.C on uh on you not not on YouTube on Google um you'll come across guy named Andre kpoy and he pretty much built up a giant gbt2 training run in C from scratch so it uses C and Cuda and all of it there's a ton of stuff in it and I really felt like it's hard to understand that at first um you know as someone who's not like super super enriched uh and have having done Cuda for like 20 years um it's kind of hard to understand that at first so having a really nice basis like this where you can actually understand how to use Cuda and where the where the real uh benefits are from it and how to

15:40 use it that will allow you to read and approach kpoe lm. see a little better so that was one of the reasons why I actually made this is to make it easier for people to go into llm Doc and understand what's going on so in the GitHub link and the notion document inside of my GitHub repo uh you will see this in the intro section so just a bunch of cool videos on how uh Cuda Works how Transformers work a bunch of just really cool fun videos to you know really get you motivated and upbeat on uh all of this so got some technical stuff we got some fun videos by fireship um but generally speaking these are just some cool resources you can check out uh Cuda programming kudam mode is a really good server actually I highly recommend you join this it's just a Discord community of a bunch of people who are really into Cuda so I believe Andre gpoy is in here A bunch of really cool uh you know coders a bunch of Engineers are in here just to discussing how to uh how to get certain kernels working and and generally just Cuda stuff um hence why it's called cuda mode right so uh really cool server I highly recommend you join that as well as my server which is also