About the course

0:00 This course walks you through the complete process from data extraction and cleaning to tokenization and transformer architecture implementation giving you the skills to preserve someone's voice or create models for under reppresented languages. Immad sodic developed this course. Hi everyone and welcome to this new course on pre-code camp. My name is Ahmad and I am here to guide you step by step on how to pre-train and fine-tune a language model in your language. This course covers many topics.

0:48 So I have decided to divide it into two parts. In the first part, I'm going to start small. I'll show you how to load a small data set and later train a small language model. This part will allow us to understand the individual concepts clearly. In the second part which I called less scale is made to show you how to use large data sets and how to train models with large parameter sizes. In this course you will learn the full pipeline.

1:11 The first step consists of taking the data set. We are going to work with textual data. Later we are going to train a tokenizer with the bite pair encoding algorithm. A tokenizer is used to convert the text into a sequence of numbers. These are the individual tokens that the model will train on. Later I will show you the transformer architecture which is used to create the language model. And then after creating the model, after encoding the data, we are going to start the pre-training phase.

1:57 At the end of this phase, we are going to get a base model. Later, in order to create a conversational assistant, we are going to create a supervised fine-tuning data set that we will use in the fine-tuning step. And at the end of the pipeline, we are going to get an assistant that we can chat with since we have a lot to cover. I recommend taking your time. Learn each part slowly because this course is made for beginners.

2:23 So don't worry about any concepts. You can find all slides, notebooks, scripts, everything I have used in this course in the link that I will make sure to put in the video description. Also, if you want to connect with me, I will make sure to link my LinkedIn profile or if you want come to my YouTube channel or connect with me on Discord. I'll be happy to chat with you. Now, let's get started. Hello, welcome to this course.

2:46 Here you will learn how to train a language model from scratch. The model