AI research realities

0:00 The first thing that came as a surprise to me is I was absolutely grilled on theory. So there were seven professors. They asked me everything. So they asked me explain KL divergence, cross entropy loss, mixture of experts, how large language models work. They asked me everything. And this makes sense. We cannot do research if we don't understand in detail what we are researching. And of course, I was able to answer it and graduate because I did my own thesis.

0:47 However, a lot of you guys are using AI to code, to do AI research. Uh, now I don't know how good of an idea is to use AI to implement things that you don't understand. So my first advice is you can use AI to implement things but you need to understand 100% what AI wrote and how your system works in order to be able to research it or improve it. So maybe the first time you are learning about it.

1:11 Maybe it's good to implement it yourself by hand. From my thesis defense I realized how absolutely crucial how important it is that you understand your research deeply. Everything you do deeply. I didn't have this idea before that it's so important that you deeply understand everything about your research. Second realization I had previously I've been trying to help you guys write AI research papers but I think now I'm going to change to helping you become AI researchers over long term because let's take a look at this very famous AI researcher and check at the check at the look at these recent papers they have zero citations one two two 16 one two two citations And then papers from one year ago, even two years ago, they start to have couple up to 100, few hundred some of them citations.

1:58 And this is by one of the best AI researchers in the world. This guy could be the best AI researcher in the world. So if you were to write some of these papers, you would get zero citations and you would think, "Oh my god, this paper is a flop." So, but I'm not even trying to say that citations don't determine the quality of the paper. That's obvious. What I'm trying to say is usually you will write a lot of papers and then one of those papers will be absolutely viral, will be absolutely uh incredibly important and impactful.

2:47 So, a lot of papers and only a few have a lot of citations. So, only a few are very impactful. That's why I don't want to just focus on helping you write research paper. I want you I want to start focusing on making you AI researcher over the long term. So you write a bunch of papers and then one two three of those become extremely valuable. And of course it's very difficult to predict which ones would become very uh valuable.

3:09 So that's why we need to write a lot. I want you all to start writing blogs or even YouTube videos explaining things. For example, you're learning rope. If you write down, explain it, it's going to help you understand it so much better. I would say the main value of teaching this is for you to understand it better. This is how you will become AI researcher. There is no other way. You cannot even get into MIT, Harvard or AI PhD until you start writing papers.

3:32 And even if you cannot get into these top programs because it's difficult and it's also dependent on luck and finance and everything. So just by writing blogs you can become AI researcher. By the way if you can attend this top university program it's very good but the reason I'm doing this channel in the first place is to bring this top university top uh research program to everybody. So on the discord here in this uh channel we write blogs, we help each other, we give feedback, we learn AI research together.

4:24 Now you may also get my personal help and mentorship if you join the school link below the video. Whatever path you choose, it's just about you sitting down, writing blocks and deeply understanding uh everything you're doing. And it doesn't even matter. Don't be too excited about LLMs. uh I'm not 100% sure that LLMs are the next thing but LLMs contain so much fundamental things you are learning about cross entropy kale divergence those are used everywhere transformers so there are uh so many common things between LLMs Japa and all the other architectures so whatever you are interested in you can learn LLM Japa reinforcement learning just learn something you are interested in and by the way guys now that I graduated in like two weeks I'm going to move and live next to these two top AI universities in the world.

5:11 And I have my friends there who are AI PhD students at Pecking University in Sinua. And they're going to help me get in. I'm going to look to join some of the labs and then bring top university AI research on my channel for everybody in the world to learn. But the main thing for you to become AI researcher is sit down and start writing a blog explaining the topic, learning the topic, deeply understanding the topic and the knowledge you get in one area.

5:33 It's actually very transferable to other areas. You will over sometime become very competent in the whole thing. quickly checking few other famous researchers to show to you that you need to write a bunch of papers and then one of them will go viral. So Haiing very famous researcher working at MIT uh he his very famous paper is residual learning residual networks. So he has a massive amount of citations on those but recent ones not so much.

6:20 So don't get discouraged and look at this with the long-term lens. and just start writing blog posts. When you write blog post, you can publish it in our discord. Just link it, explain what you did. Uh whoever has time, we will check it. We will give some feedback, etc. Or join the school and I'm going to give you feedback there personally. Very good researcher here as well. Citation counts. But look how many papers she writes. It's 2025.

6:43 All of these are 2025. So these are papers from her students, I believe. and she's co-author maybe mentoring but so many papers it's so it's it's just about producing papers sitting down understanding producing papers and then usually discover a lot of discoveries is just randomly stumbled upon because it's not possible to know which of these papers is going to be like the best one recent PhD graduate from MIT a lot of papers not so many citations but it's just to show you that citations does do not measure quality So this paper gated delta nets could potentially prove to be very big paper but we have yet to see but this is a very good invention that's already used by Alibaba and I think Kiny Moonshot and other companies possibly even open AI but we don't know uh what they use.

7:31 So just 123 citations you would think a random paper but it's widely used right now. It's a very good invention. So she was writing all of these papers and then this one kind of turned out to be she didn't know beforehand that this one would be like the best one. So after just testing trying a bunch of these this one proves to be the best one. So that's why I want to stop focusing on helping you write a specific single paper and help you just start doing AI research in general over the long term producing a bunch of papers and getting this down.

8:20 Jan's research is very interesting to me because it's about the classic machine learning, neural networks, autoenccoders, representation space, semantic space. While the whole industry is going towards LLMs because LLMs are useful right now unlike Japa is not so useful right now like LLMs but the ideas here are actually a lot more I have a feeling they are a lot more similar to classic machine learning and the idea of learning and machine learning and neural networks and all of the new ideas that uh pop out of this research.

8:44 So this is like a very good new idea exploration. So that's going to be it for this video. Just start writing the blog post right now about anything whatever I mentioned in this video about any of these papers, any other papers, LLMs, whatever. And just start understanding deeply what you write about and that's it. And don't look at the views, citations, likes, don't look at any of that. Just start writing and understanding and then tell us about what you wrote.

9:08 And understanding and writing is interesting. Getting likes is interesting for like three seconds and then it gets boring. But writing a blog post, understanding, learning, it can be interesting as long as you are doing it. Hello guys, I just made a research plan for our $5 large language model where we have around $80 US to do this research on GPUs. The goal is to make the best $5 LLM. So, pre-train from scratch as best as possible.

9:57 I'm going to leave this link below, although it may be updated when you see it. So, first GitHub repository. This is our large language model. It's fully functional. It has three billion parameters. I'm looking for cool names still. So, it's a mixture of experts LLM, but we want to make this a lot better. As I said in my past video, we want to start competing with other LLMs, at least open source on LM Marina, RKGI, artificial analysis.

10:20 So, uh, those LLMs cost maybe half a million dollars to train at least. For now, we have $80, but when we prove ourselves here, we're going to be able to secure more compute. If you know somebody who can give us more compute, tell me below the video on Discord. So first uh we will start with model specs 3 billion parameters and then you can check the configs for now. Uh these are the configs for now the dimension of the model number of heads all of this good stuff but we need to experiment and test and improve this.

10:43 So we will do research on this more. For data set we will just use this small LM corpus which is really good for small language models. And I want to minimize the amount of research we do because we already have too many things to do. And this is really good. I tested this multiple times and small LM tokenizer that goes with this data set. Now we got around 96 hours of age 100. We could use cheaper GPU, but a lot of the code and infrastructure is made for age 100.

11:30 So I feel like we're going to use age 100s. So we need to split these 96 hours between research and training. So I'm going to also tell you some issues I'm thinking about right now. So you can comment below or give your suggestions. I'm not sure if we should train on two times age 100 or one times age 100. Benefits of training on two is we are doing research on scaling and parallel parallelization as well which will help us in the future but it's more complex.

11:55 Maybe we should we it's already very complex so we should go for simplicity here. Give your opinion below. And we will not use any of these like nanotone, megatron because they are for very complex clusters not with just two GPUs. Then we have how to contribute. So I imagine this has everybody contributing. So come to discord work with us. We will make sure to uh write who did what. So you can also uh put the research you do here in your CV when you're applying to uh job in AI research and stuff.

12:17 And you will learn how to do AI research with us. So this is actual experience like you would get at a job or at a university or previously we let anyone contribute whatever they want do whatever research they want on our large language model which resulted in just a mess and nothing else. So now we need to actually structure it more like a lab or a startup. So now we need to actually have research questions tasks that we first discuss.

13:02 we first need to discuss um what research needs to be done. So if you have some idea, you should first talk to us about it to make sure that this is not going to also result in some random people doing random things and that never gets used for anything. So um but I'm going to talk more about this later. Below you can find task research questions. So you would just clone this repo and then do the experiments and then tell us.

13:26 Sometimes we will merge back the results. Sometimes we will just uh link to your repo and implement like one or two lines of code without needing to merge all of your experiments etc. And this is good because uh you will have this repo you will have your own experiments at your GitHub profile. So it's literally the work you've done will show and you will contribute to open source. But first, talk to us on Discord before you start doing anything just to make sure what you do aligns with our goal.

13:48 I'm also thinking about this question because uh 590 is like seven times cheaper than age 100. So I can do we can do more experiments. The problem is will it transfer the results? So if we find something out here, will that same rule be applied to age 100s? We'll see about this still because I want to do the training the end training on H100s or maybe not by the way maybe we can just like put a few of these in parallel and then do everything on 590s 5090s also when you do experiments um your experiments your changes might change the optimal learning rate and even other hyperparameters but learning rate is most impactful.

14:37 So make sure to run the ablations for your learning rate again after your significant changes. This article contains all of the architectures of the latest LLMs. I think we can learn a lot from this implement a lot from this uh we want to first implement but we also want to create a research environment because we cannot just be implementing the current stuff. We need to be innovating if we want to stay competitive. Then uh I have some optimizers here.

15:02 For example, this one seems like a new optimizer. The problem with these is they're very new. They're not tested yet. We have also decision history where I decided not to use these. You may always talk about these things. So uh each of these papers would take some time to implement. So for me alone, this would take ages. Look how many papers I have. So um um I'll see how I want to structure this who does what.

15:49 But also we need to talk because we may not be actually implementing all of these. I'm going to listen to feedback and talk to people still. On the other hand, we do want to do our own research on optimizers on attention and all this stuff. So I got some papers here maybe NGPT. This one is one year old. This one was super cool when it came out, but I don't know if we're going to implement this one or maybe take some parts of this.

16:11 And below at the end, I have like re these are exact tasks we need to do immediately. And it's for finding hyperparameters right now. So this one will be done by the time you watch this maybe even but there will be more tasks here. So you can uh discuss this with us on the discord. Give some feedback and opinions and stuff. There is also this newly released LLM a few like two weeks ago and it contains all of the data, weights, training, setup, everything.

16:35 Oh, this is really interesting. I think I'm just going to take what they did and then add this to our experimentation and then we'll see uh what works best. And then some advice for from Grock 4.1 as well. We should only use 5090 for maybe some quick debugging, checking if code runs, but keep the main thing on age 100s. That's going to be more relevant. So, I'm think I'm going to do that. Follow me on X.

17:00 By the way, guys, it says Google's nested learning paper that I mentioned might be too difficult, too heavy for our case. I'm still thinking because I did mention a lot of papers. There is so much work to be done. On the other hand, these um companies are releasing LLMs like shortest period of time is six months or more to pre-train an LLM. So maybe we're going to also be spending six months on this one. But we have a very small LLM.

17:23 Um 6 months is for very maybe big complex where whole Google is building the LLM or I mean a lot of people. There is also this new paper maybe that's interesting. I'm going to add this as well. It's something for attention mechanism but it's just recommended by Gro so I don't know what it is looking at all of this there is on one hand there is a lot of work to be done on the other hand I always remember that Elon Musk says you need to delete things not do things if you're not adding things back in you're not deleting enough and all this stuff but I'm not sure how to approach that how to apply that here I guess we'll quickly skim over some things but also uh inventing something good takes time you cannot just force put a deadline on invention I don't know if you want to become AI researcher join our school

18:36 link below the video in the description there are courses live calls mentorships where you can engage with us and we will help you figure out your path and there is the community so check the link below we use this to help us pay for our research 7-day free trial, then just $9 per month. See you in the next video. Today, we made so much progress in our AI research. We merged many pull requests and we figured out a very good way to structure new research questions, research tasks, and this is exactly how they do it at OpenAI deep mind anthropic.

18:59 Currently, I'm running research on this large language model. Our goal is to make pre-training of large language model as fast as possible. We already made GPT1 level model. Our next goal is GPT2, then GPT3, 4,5, etc. And here I have all of these settings, hyperparameters for the large language model. And I want to find the best one that's going to train to GPT2 fastest. I'll talk more in detail a bit later for technical audience who want to know more.

19:23 But I want to show now how well I structured readme. Now it's a lot shorter, a lot simpler to understand. And here there is the first research. This is also a very good uh structure for the research. So replacing swigloo with squared relu. And I measured some improvement. But we need to make sure this is repeatable. This is going to happen again. So uh there is this improvement. There is this loss. But you can see that loss is actually very similar between them.

20:11 So we want to actually understand this even deeper. So any of you can come here and then do more research. But I don't want you just start pasting walls of AI generated text spending 5 10 minutes on this. This is AI research. AI research takes years. It's not something you will just ask Gemini or Claude to do. So uh don't waste your time if you don't want to spend two hours to understand this deeply. uh write carefully, code carefully.

20:34 This is not something that you just tell AI agent to do. It's not going to work. So for example, this pull request, I'm very suspicious. So here I see so much text. Now I see some images is showing showing some results. I also see a bunch of AI generated text. So I'm going to review this later. I don't know if this is good or not. If this proves to be okay, then okay. If it proves to be a bunch of slop, then we're going to have to put in some uh more rules against this AI generated slope.

20:58 It looks like my experiment here is done and it's drawing the image properly. So, make sure to debug. This was just a debugging run very uh very quick. So, it shows the image. And I see that for example, the pink one loses loss fastest. It's going to be this Blueberry B8 config, but this is actually the smallest model as well. It's two to three times two to two and a half times smaller than other models. So, it makes sense that it will uh learn faster.

21:45 It's going to be computed faster, but it should achieve a higher loss overall. These bigger models should eventually achieve lower loss. So, now I'm going to run this same experiment for a longer time. And even if the loss of these bigger models is not necessarily lower, they should just know more because