Complete AI Training
Sign inGet my AI kit

Your job's AI kit

Get your AI kit

Tell us who you are and what you do. We show you your kit right away and email you the link: skills, prompts, AI agents, MCP servers and courses for your job.

500+ jobs ready, and we make a kit for any other job. No payment needed to look.

Share

Video course · 15 chapters · 57 min · certificate

Modern AI Evaluation Frameworks: A Fast-Paced Course

Learn how modern AI evals work: offline evals versus production, writing evals that are neither too easy nor too hard, moving from QA to task thinking, building agentic evals on real genetics problems, sampling, and finding the model ceiling.

Become a memberAlready a member? Sign in

What you'll learn

  • Explain what an eval is and why it matters
  • Write evals that avoid being too easy or too hard
  • Move from QA benchmarks to task-based evals
  • Build an agentic eval grounded in expertise
  • Sample multiple runs and find the model ceiling
  • Apply subject-matter expertise in product decisions

Chapters

15 chapters · 56:56
  1. 2:41 01Intro Members Why evals matter now Big tech is reinventing product work for AI, and evals are central.
  2. 4:22 02Basics Members What an eval is Evals communicate product quality; offline evals versus shipping, and why they got more complex.
  3. 3:02 03Method Members Writing an eval The mechanics: questions, expected answers and scoring; and why too-easy or too-hard evals fail.
  4. 5:27 04Shift Members From QA to task thinking Old QA benchmarks are saturated; evals must test tasks, not trivia.
  5. 3:17 05Build Members Building an agentic eval Building an agentic eval live, starting from deep understanding of the problem.
  6. 4:22 06Case Members The cystic fibrosis eval An eval on the canonical cystic fibrosis mutation using the ClinVar database.
  7. 4:21 07Run Members Running the variant file eval Running the eval on a variant file, deliberately without fully automating it.
  8. 5:35 08Test Members Testing the agent Testing the agent on the disease case and interpreting its answers.
  9. 3:00 09Case Members Congenital heart disease eval A harder eval based on a recent paper about digenic genetic disease.
  10. 4:32 10Difficulty Members A harder problem Pushing models on harder questions: one disease or two abnormalities?
  11. 3:31 11Method Members Sample multiple times Because outputs vary, run each eval several times.
  12. 2:54 12Ceiling Members Finding the model ceiling Comparing models to find the ceiling, and deciding acceptable performance as a PM.
  13. 2:57 13Expertise Members It's all expertise Evals depend on subject-matter expertise, not templates.
  14. 2:58 14Career Members PM at Meta versus Google Differences between product management cultures at Meta and Google.
  15. 3:57 15Wrap-up Members Building a lab and closing Starting an AI lab, and closing thoughts.

Jobs this course suits

Our AI checked this course against 500 jobs; these get the most out of it. Each job links to its learning path.

AI EngineersMachine Learning EngineersProduct ManagersQA ManagersData ScientistsProduct AnalystsQuality Assurance TestersSoftware ArchitectsChief Product Officers (CPOs)Software EngineersPrompt EngineersSoftware Developers