Complete AI Training
Sign inGet my AI kit

Your job's AI kit

Get your AI kit

Tell us who you are and what you do. We show you your kit right away and email you the link: skills, prompts, AI agents, MCP servers and courses for your job.

500+ jobs ready, and we make a kit for any other job. No payment needed to look.

Share

Video course · 15 chapters · 63 min · certificate

LLM Development Essentials: Build and Train Qwen 3 from Scratch

Build and train a small Qwen 3-style LLM from scratch in PyTorch: the architecture, configuration, grouped query attention, training hyperparameters, the Muon optimizer, data loading, RoPE, self-attention, SwiGLU feed-forward, assembling the model, evaluation, training and text generation.

Become a memberAlready a member? Sign in

What you'll learn

  • Describe Qwen 3's architecture
  • Implement grouped query attention
  • Explain the Muon optimizer
  • Implement RoPE and self-attention
  • Build SwiGLU feed-forward blocks
  • Train and generate text with the model

Chapters

15 chapters · 63:04
  1. 1:46 01Intro Members Intro and demo Qwen 3 from scratch.
  2. 3:40 02Architecture Members Architecture and setup Transformer plus Qwen specifics.
  3. 3:00 03Config Members Model configuration Dimensions and heads.
  4. 3:58 04Architecture Members Qwen 3 specifics Sharing key/value heads.
  5. 4:54 05Training Members Training hyperparameters Batch, accumulation, LR.
  6. 1:38 06GQA Members Grouped query attention logic Repeating KV heads.
  7. 10:06 07Optimizer Members Muon optimizer A newer optimizer.
  8. 3:35 08Data Members Data loading and tokenization Preparing text.
  9. 4:19 09Positions Members RoPE positional embeddings Rotary embeddings.
  10. 7:32 10Attention Members Self-attention code Implementing attention.
  11. 3:08 11FFN Members Feed-forward and SwiGLU The MLP block.
  12. 4:58 12Model Members Building the final model Stacking blocks.
  13. 3:09 13Training Members Evaluation and the training loop Loss, accuracy, perplexity.
  14. 2:55 14Training Members Running the training Data loaders and runs.
  15. 4:26 15Inference Members Inference and results Generating text.