Complete AI Training
Sign inGet my AI kit

Your job's AI kit

Get your AI kit

Tell us who you are and what you do. We show you your kit right away and email you the link: skills, prompts, AI agents, MCP servers and courses for your job.

500+ jobs ready, and we make a kit for any other job. No payment needed to look.

Share

Video course · 15 chapters · 344 min · certificate

Create a Large Language Model from Scratch with Python – Tutorial

Build a GPT-style language model from scratch in Python: setup, a Wizard of Oz bigram model, tokenizers and tensors, PyTorch basics and GPU, embeddings and matmul, training loops, activations, the transformer and self-attention, a GPT decoder with multi-head attention, scaling to a large corpus, scripts, and pre-training vs fine-tuning.

Become a memberAlready a member? Sign in

What you'll learn

  • Set up a PyTorch environment
  • Build character tokenizers and a bigram model
  • Implement embeddings, matmul and training loops
  • Explain transformers and self-attention
  • Build a GPT decoder
  • Scale training and run a chat prompt

Chapters

15 chapters · 343:41
  1. 12:11 01Setup Members Intro and setup Libraries and Jupyter.
  2. 21:17 02Tokens Members Data, tokenizers and tensors Wizard of Oz.
  3. 32:35 03PyTorch Members PyTorch overview and GPU CPU vs GPU.
  4. 19:39 04Embeddings Members Embeddings and dot products Vectors for tokens.
  5. 25:11 05Training basics Members Matmul, batches and gradient descent nn.Module.
  6. 27:18 06Loop Members Logits, generation and training loop Generate text.
  7. 28:44 07Activations Members Loss, normalisation and activations ReLU, sigmoid, tanh.
  8. 30:59 08Transformer Members Transformer architecture Position matters.
  9. 29:02 09GPT Members Building a GPT Decoder-only.
  10. 13:54 10Init Members Standard deviation for parameters Initialisation.
  11. 18:53 11Blocks Members Transformer blocks and attention Multi-head.
  12. 18:13 12Scaling Members Scaling by 1/√dk and hyperparameters Why scale.
  13. 27:36 13Data Members Scaling the data A large corpus.
  14. 16:32 14Scripts Members Scripts and GPU memory Port to scripts.
  15. 21:37 15Wrap-up Members Prompt completion and next steps Pre-training vs fine-tuning.