What You Will Learn

• Design executable evals as precise product requirements (prompts, keys, scores)
• Build and calibrate offline eval suites with Goldilocks difficulty (25-50%)
• Implement automated scoring for agentic tasks (LLM-judge, tool checks, structured outputs)
• Use offline and online evals to gate rollouts and monitor production
• Leverage domain expertise and harness engineering to create realistic, reliable evals

Study Guide

Introduction: The New Language of AI Product Building

If you're building products on top of modern AI systems, you've probably felt that uncomfortable moment when someone asks, "So... how do we know this works?" And you mumble something about "testing" and "prompts" and "vibes." That moment is exactly why this masterclass exists. The old ways of specifying product behavior,thousand-word requirement documents, detailed user stories, meticulous acceptance criteria,break down completely when your product is a probabilistic system that can do an infinite number of things in an infinite number of ways.

The solution? Evals. Structured, executable sets of prompts, expected responses, and scoring logic that tell your team exactly what "good" looks like. Think of them as tests that never lie, contracts that never need a lawyer, and product requirements that actually run on a computer. By the end of this session, you'll understand why evals have become the new product requirements document, how to design them from scratch for any domain, and how to use them to steer AI products from idea to production with confidence. This isn't theory,this is a practical toolkit used by the people at the frontier of AI product development. And it's accessible to you, right now.

Why AI Evals Just Became the Center of Gravity

Here's the brutal truth about product management in the age of AI: the traditional role of orchestrating people, motivating teams, and analyzing outcomes is getting automated. That's not a prediction,it's happening. The PMs who'll thrive are the ones who can specify, measure, and iterate on AI behavior itself. That skillset starts with evals.

Let me walk you through why. You're building a feature that uses a large language model. You could write a 50-page PRD that describes how the system should behave in specific situations. But generative AI systems can support an almost unlimited range of use cases. "Describe everything in words" becomes impossible. And if you say "the system should handle everything correctly," that's neither operational nor testable. It's a wish, not a spec.

Evals fix this by being executable specifications. An eval is essentially a structured collection of tasks for the model,call them trivia questions, call them test cases, call them scenarios,paired with answer keys and scoring logic. If your model passes the eval, it does what you want. If it doesn't, you know exactly what's missing. The eval becomes the best way to communicate what your product should be doing and to explain to the engineering team what success looks like. No ambiguity. No second-guessing. Just measurable reality.

Offline Evals: Your Pre-Flight Checklist

Let's talk about the two flavors of evals: offline and online. Offline evals run during development, before real users ever see your system. You bake a set of prompts and tasks, define the answer key, and run the model against it. It's repeatable, consistent, and gives you a reliable signal of progress.

Say you're building an ice cream recipe assistant. You'd create around a hundred prompts covering diverse styles,gelato, vegan ice cream, low-sugar options, weird flavor combinations like basil and black pepper. For each prompt, you define what a good response looks like: completeness of ingredients, sensible instructions, proper formatting. Then you score each response.

Why bother? Three reasons. First, offline evals give you a consistent way to assess model progress as you iterate. Second, they enable A/B testing of models, prompts, and your tool harness. Third,and this is the hidden gem,they help you detect impossible product requirements early. If no model or harness can score above 20% on your eval, you know your product idea might not be feasible with today's technology. It is far better to find that out in development than to ship a feature and frustrate real users. The product lifecycle should always flow through these gates: exploratory experimentation first, then offline eval success, then limited online rollout, then broad launch.

Online Evals: The Reality Check

Offline evals are necessary, but they're not sufficient. Online evals measure what happens when real users interact with your system,satisfaction, task success, retention, and other product metrics. The critical question online evals answer is whether your offline scores actually correlate with real user outcomes. Sometimes they do. Sometimes they don't.

A model that scores beautifully on your offline eval might fail in production because users ask things you didn't anticipate, or the interface changes how people phrase requests, or the stakes of their tasks create emotional reactions you can't simulate in a test. That's why online evals exist: to validate that your proxy was actually a good proxy. The discipline of moving through offline evals to limited rollout to broad launch, with evals gating each step, keeps you from betting the company on an eval that doesn't reflect reality.

The Saturation of Question-Answering Benchmarks

To understand where evals are going, you need to see where they've been. In the early days of generative AI, models were framed as search or chat replacements. So benchmarks focused on question-answering: MMLU, HellaSwag, ARC, Winograd, HumanEval, DROP. You'd give the model a question, it would produce an answer, and you'd check if the answer was right.

Here's a classic MMLU item: "If you know the actual brightness of an object and its apparent brightness from your location, then with no other information you can estimate its: (a) speed relative to you, (b) composition, (c) size, (d) distance from you." The answer is distance. That's a challenging high school physics question. And here's the problem,frontier models now crush these benchmarks. Scores are effectively at the ceiling. A model that can handle challenging high school physics questions is now routine, not impressive.

When we think of a good model now, we don't think of one that can answer a tough physics question; we think of models winning gold medals at international math Olympiads. The old benchmarks are saturated. They measure basic competence, not frontier capability. And that's fine,it means the evaluation landscape had to shift to something harder and more meaningful.

Agentic Tasks: The New Evaluation Frontier

The shift happened because the most valuable use cases for AI are no longer single-prompt, single-answer interactions. The core model task is no longer to take a prompt from a user and return an answer. It's to take a task from a user that requires many steps,reasoning, searching, calling tools, executing code, interacting with external systems. These are agentic tasks, and they demand a completely different evaluation approach.

Frontier AI labs now highlight benchmarks for agentic coding, terminal use, computer use, knowledge work, and financial analysis. Think about what that means for evaluation. A task description becomes the input, and the system must produce a multi-step sequence of reasoning and tool calls before reaching a final outcome. That's not a trivia question anymore. That's a performance.

The implication is enormous: evals must now capture tasks, not just question-answer pairs. And scoring must handle long sequences of actions and complex final outputs. You can't just check if the answer matches a string anymore. You have to evaluate whether the entire journey,the tool calls, the intermediate reasoning, the final output,accomplished the user's goal.

The General Framework for Building AI Evals

Despite the complexity, the high-level framework for building evals remains remarkably consistent. Three steps. That's it. First, define the problem precisely. Second, collect representative prompts that are calibrated to the Goldilocks zone,not too easy, not impossibly hard. Third, design a scoring strategy, preferably automated.

Let me dig into each step because the devil is in the execution.

Step One: Understand the Problem Deeply

You cannot build a meaningful eval for a domain you don't understand. This sounds obvious, but it's the most common failure mode I see. AI labs now actively hire investment bankers, lawyers, accountants, and biologists to design evals because they know that without substantive domain knowledge, you can't define realistic tasks or judge whether an answer is correct.

You are unlikely to be successful in creating an eval for a topic if you don't have some background in it or haven't really educated yourself on it. Let me give you an example. If you're building image generation for a visual discovery platform, you need to understand what users actually want. The phrase "beautiful kitchens" is not self-explanatory. What style? What lighting? What layout? What aesthetic preferences do your specific users have? You have to translate vague desires into concrete examples and scoring criteria.

The problem definition should always include several elements: the context and user persona, the input modalities and constraints, the desired output structure, and the failure modes you must avoid. Get these explicit before you write a single prompt.

Step Two: Collect Goldilocks Prompts

Now we get to the heart of eval design. Your prompts need to be in what I call the Goldilocks zone. Too easy means the model scores near 100%, and there's no room to optimize anything. Too hard means the model scores near 0%, and you can't tell if the task is even feasible with current technology.

A typical generative AI eval contains roughly a hundred prompts,more or less depending on your domain and needs. And here's the key calibration: a good eval should start at something like 25% to 50% success for the model and harness you're working with. That gives you room to improve. Over months, as your system improves and approaches 100%, you retire that eval and build a harder one. This is how the frontier keeps moving. A typical good eval will have a 25-50% success rate, then over months that goes to 100%, and then you throw it away and create a new one that is harder.

Construction is a systematic process. Brainstorm real-world tasks users would actually perform. Use AI models themselves to generate synthetic variants and edge cases. Then manually inspect both the easy and hard examples to ensure they're realistic. Don't just rely on your imagination,rely on data, even if that data is machine-generated and human-verified.

Step Three: Design Scoring Methods

Scoring is where evals live or die. For QA tasks, scoring is straightforward: exact match, F1 score, or human raters. But for agentic tasks, scoring gets genuinely hard. Outputs are multi-part and multi-step. Some steps can be correct while the overall outcome is wrong. Time horizons are long. Manual evaluation of every step in every trajectory is so labor-intensive that you simply can't do it at scale.

This means automatic scoring isn't a nice-to-have,it's mandatory. The most common approaches: LLM-as-judge, where another model compares the output to a rubric or reference answer; tool-based checks, where you run generated code or validate against external APIs and knowledge bases; and structured output formats that can be programmatically verified.

At the end of the day, evals are literally just prompts, responses, and ways of scoring whether the response is correct. That simplicity is the point. It's why evals can scale across teams, domains, and product lifecycles.

Case Study: Clinical Genomics in the NICU

Let me show you what all of this looks like in practice with a high-stakes domain: clinical genomics. This is a field where whole genome sequencing has become the absolute gold standard for diagnosing conditions in neonatal intensive care units. A baby is sick, doctors don't know why, and sequencing the baby's entire genome can provide answers.

But here's the bottleneck. Sequencing produces about a billion reads of roughly 150 base pairs for a single person. Processing turns those raw reads into a VCF file,a text file listing positions where the patient's genome differs from a reference. Each record includes the chromosome, position, reference bases, alternate bases, and annotations like quality and zygosity. The interpretation of that file,determining which variants matter for the patient's condition,is extremely labor-intensive. It requires experts who are scarce and expensive. And that labor intensity limits access to this life-saving technology. That's where AI agents come in, and that's where evals become essential.

Building the Easy Eval: Cystic Fibrosis

When you start building evals for genomic interpretation, you begin with something straightforward: cystic fibrosis. CF is one of the best-understood genetic diseases. The canonical cause is a variant in CFTR, the cystic fibrosis transmembrane conductance regulator gene. A well-known pathogenic variant is the deletion of a phenylalanine at position 508, written as F508del.

Here's how you structure the eval. The phenotype is cystic fibrosis. The input is a synthetic VCF file representing a patient genome with the canonical CFTR variant inserted. The task asks the agent: "Given this genome and the suspected phenotype, identify the genetic cause." The answer key is simple: the correct gene is CFTR, and the correct variant is the specific deletion at position 508.

Scoring? You can use LLM-as-judge or simple structured comparison. Does the agent name the right gene? Does it correctly describe the variant?

What happens when you run different models on this eval? A powerful frontier model with a good harness identifies the CFTR variant. But a smaller, cost-optimized model might notice that CFTR is relevant yet fail to pinpoint the exact variant,or worse, hallucinate deletions that don't exist in the data. Even a "solved" domain like CF genetics reveals meaningful differences between models when you use realistic, noisy data with hundreds of variants in a large VCF file. This is your floor. It's easy for top models but non-trivial for weaker ones, which makes it valuable.

Building the Hard Eval: Di-genic Congenital Heart Disease

Now let's push to the ceiling. Some diseases aren't caused by a single gene. Congenital heart disease can arise from di-genic architectures,where two heterozygous variants in two different genes jointly cause a single phenotype. Research has catalogued patients with CHD whose causal architecture involves such combinations.

Your eval design: the phenotype is congenital heart disease with an appropriate clinical description. The input is a synthetic VCF with two heterozygous variants in two specific genes, as described in the research. The task: identify the genetic cause. The answer key demands both gene names, both specific variants, and the recognition that the disease is di-genic,neither variant alone explains the phenotype.

Here's where it gets interesting. A strong but straightforward agentic harness fails this task in one configuration, surfacing unrelated variants instead. Smaller models perform poorly. But in a more powerful configuration, a frontier model succeeds by mapping the phenotype to relevant literature, finding the specific scientific paper describing di-genic CHD, and extracting the correct gene pair and variants.

Notice what happened. The model didn't just look at the VCF and compute an answer. It used reasoning, searched literature, identified relevant research, and connected clinical knowledge to genomic data. This eval establishes your ceiling,tasks that even top models find challenging. And it proves why vertical, harness-engineered systems are necessary to reliably solve cutting-edge scientific problems. A general model with no tools, no search, no structure just isn't enough.

Sampling, Shots, and Variance

Let me address a topic that causes a lot of confusion: sampling. Historically, you could boost performance on QA benchmarks by generating multiple candidate answers and selecting the best one. Benchmarks like MMLU reported few-shot performance,5-shot, 32-shot,and labs used sampling to their advantage in benchmark comparisons.

Here's the deal with modern agentic tasks: repeated sampling of the same prompt often produces similar results, especially with harnesses that have deterministic or semi-deterministic tool chains. Sampling can still matter, but its impact is much smaller than it was for early QA-only use cases. My safe approach: test sampling effects empirically for your specific domain. For sensitive or high-stakes evals, run each task multiple times if you observe variance.

The di-genic CHD example shows why this matters. A model may fail on one run and succeed on another simply because it followed a different reasoning trajectory,one that happened to surface the right paper. This non-determinism means you need transparent logging of reasoning traces and careful design of your eval harnesses. You want to know whether you're measuring true capability or luck of the draw.

Using Evals Across the Product Lifecycle

When should you start building evals? Day zero. The first thing you should do when you're starting to build a new AI feature is understand what the user wants and translate that into prompts, responses, and ways to score those responses. Not after you've picked a model. Not after you've built a prototype. Day zero.

Let's go back to the image generation example. If you're building for a visual discovery platform, the questions start immediately. What do users consider a beautiful kitchen? What elements matter,style, lighting, layout? How do you encode those preferences in examples and scoring rules? These questions drive everything downstream: model selection, prompt design, user research, feature scope.

Evals also shape your product scope in powerful ways. Let's say your eval shows the system handles a specific set of tasks with 90% reliability but another set at 40%. That's information you can act on. You restrict the initial feature set to the tasks where eval scores cross acceptable thresholds. You add guardrails that block or gracefully decline tasks where evals show weak performance. You route complex or ambiguous cases to human experts. And you hand the hardest failing tasks to your research and engineering teams as targets for improvement.

This is how evals act as cross-functional contracts. Product management defines the user problem and success criteria. Engineering and research optimize models and harnesses against the eval metrics. Operations and safety use the evals to ensure risky behaviors are tested and constrained. Because evals are concrete and executable, they reduce the ambiguity and misalignment that kills AI projects.

The Organizational and Cultural Dimensions

Here's something most technical guides don't talk about: your organization's culture determines whether eval-driven development will thrive or die. Look at the contrast between major tech companies. An engineering-led culture prizes technical excellence and consensus. Decision-making is deliberate, thorough, but sometimes slow. A product-led culture is more aggressive, founder-driven, and willing to make large bets quickly. The products managers carry strategic weight.

Both models are being disrupted by AI. In an engineering-led culture, you might have beautiful eval infrastructure but no product vision driving what to measure. In a product-led culture, you might have bold product bets that ignore the rigorous measurement evals provide. The winning path is a rebalancing: engineering excellence without eval-defined goals is directionless, and product strategy without evals degenerates into vague aspiration. Both sides have to meet in the middle.

And here's the uncomfortable part for product managers: the metas and the Googles and all the other large companies have to reinvent themselves right now in the age of AI. The old PM playbook,orchestrate, motivate, analyze,is being automated. The new playbook demands that you understand eval design, interpret results, and make trade-offs between model capability, harness sophistication, and product scope. This isn't optional. It's the new core competency.

Vertical AI and Harness Engineering

Let me tie this back to a bigger opportunity: vertical AI. General models alone are not sufficient in high-stakes domains. Whole genome sequencing interpretation for rare diseases in NICUs is a perfect example. Complex legal analysis. Specialized financial research. These require harness engineering,building domain-specific workflows, tools, and knowledge integrations around a frontier model.

The motivation is compelling. A new generation of startups wants to make WGS-based diagnosis accessible not just at elite academic centers but in resource-constrained hospitals across the country and around the world. They're reducing the cost and expertise required to interpret genomes for rare diseases. And the early evidence shows it's possible,AI-augmented systems have already identified previously unknown pathogenic variants, and in at least one documented case, genomic insights empowered a family to safely plan another child. That's the real-world impact these systems can have.

Evals are the backbone of this work. They encode known gene-disease relationships and newly discovered architectures like di-genic conditions. They let teams measure whether their harnesses are catching known cases and generalizing to novel ones. And they support regulatory and clinical validation by demonstrating consistent performance on well-characterized benchmarks. Without evals, you can't prove your vertical AI system is safe, effective, or ready for prime time.

Practical Implementation: A Checklist for Your First Eval

Let me give you something actionable. If you're a product manager or engineer who's never built an eval, here's your starting point. First, find someone who's done this before and learn from them. Eval construction looks deceptively simple,it's just spreadsheets,but there's a taste component that's hard to codify. Learn from experience.

Second, start with the fundamental question: what does success look like? Define the problem statement clearly. Then identify roughly 100 representative prompts in your target distribution. Establish the answer key with expected correct responses. Calibrate the difficulty so you're targeting 25% to 50% initial success. Select your scoring methodology,automated, LLM-judge, or human,and be honest about which one your task actually requires.

Test sampling effects. Run the same cases multiple times to assess variance. Archive and version your evals as models improve and become saturated. And regularly refresh your eval suite with harder tasks so your measurement stays meaningful. This is a living process, not a one-time deliverable.

Using Eval Results to Make Decisions

An eval is only as useful as the decisions it drives. Let me walk through the stages. During development, eval results guide engineering improvements,they tell you what to optimize next. Before shipping, they validate readiness,whether you go or no-go. After shipping, they verify online performance and real user impact. In continuous operation, they track regressions and alert you to quality degradation. And for product scoping, they set feature boundaries based on minimum performance thresholds.

Here's a concrete example. Say you score 50% on your eval. A smart product manager might decide to ship anyway, but with guardrails that limit the feature to the subset of tasks where the system achieves 80% or higher. Hard cases get routed to research teams with instructions to improve model or harness capabilities. That's how you launch responsibly without waiting for perfection that never comes.

Action Items for Product Managers

Let me be direct about what you should do next. First, embed evals into your product process from day one. As soon as you define an AI feature, start drafting an eval table with columns for context, input, expected behavior, and scoring notes. Second, partner with domain experts. In healthcare, finance, or legal, co-create your evals with practitioners who know the edge cases and real failure modes. Third, define "good enough" explicitly. Decide your offline performance thresholds for launch,maybe 80% exact match on core tasks,and document which tasks remain off-limits for your system. Fourth, use eval results to narrow scope. Restrict your AI feature to tasks where evals show strong performance. Don't overpromise generality your system can't deliver.

Action Items for Engineering and AI Teams

For the technical folks: build lightweight internal tooling for evals. Start with spreadsheets and scripts before investing in complex platforms. Standardize how prompts, outputs, and scores are stored and versioned. Automate scoring wherever possible,use LLM-as-judge, unit tests for generated code, and programmatic checks for structured outputs. And instrument online behavior to complement your offline evals. Capture anonymized task outcomes and user feedback for continuous improvement. Your eval suite should never be static; it should evolve with your product and your users.

Action Items for Organizations and Leaders

Leaders, this one's for you. Make eval literacy a core competency across your organization. Train PMs, engineers, and executives to read and question eval results the way they read financial metrics. Incentivize improvements on meaningful product-specific evals, not just public leaderboard benchmarks. The leaderboard-chasing game is a distraction from what actually matters: delivering products that serve users reliably.

And invest in vertical AI initiatives where the stakes and complexity are high. Support teams that combine frontier models, specialized harnesses, and strong evals in domains like healthcare, education, and industry-specific workflows. That combination is the clearest path to solving problems that general-purpose AI can't touch.

Critical Mistakes to Avoid

Before you go build your first eval, let me flag the mistakes that will waste your time. The first is treating evals as a one-time activity. They're not. They saturate. They get easier as models improve. You must be willing to throw away evals that everyone passes and build harder ones. The second mistake is skipping domain expertise. An eval written by someone who doesn't understand the domain produces neither realistic tasks nor trustworthy scoring. The third is relying on manual scoring for agentic tasks. It doesn't scale, and you'll drown in evaluation labor. Automated scoring is not optional. The fourth is ignoring variance. If you don't test whether your results are stable across runs, you'll make decisions based on noise. Run your evals multiple times and understand your confidence intervals.

Evals in Creative and Subjective Domains

What if your product generates something subjective,images, writing, music, design? Scoring gets harder because "good" is a matter of opinion. This is where human evaluation matters. For creative tasks, you may need panels of human raters to assess outputs, or you need well-calibrated LLM judges that have been aligned with your specific users' aesthetic preferences.

But don't give up on structure just because the domain is subjective. Break "a beautiful kitchen" into components: style consistency, lighting quality, layout realism, color harmony. Define each component explicitly. Use examples to anchor your scoring. Accept that some evals will require human judgment, but design those evals to make the human's job as easy as possible,clear rubrics, focused comparisons, minimal ambiguity.

The responsibility here belongs to product managers. You must understand that different user segments have different aesthetic preferences. An eval that assumes one universal standard of beauty will mislead you. Build evals that respect the diversity of your user base, and test your assumptions about what "quality" means to different people.

Why Evals Will Only Matter More

Here's the trajectory I see. Models will keep improving. They'll keep conquering benchmarks we thought were impossible. And every time they do, evals will need to get harder, more nuanced, and more domain-specific. This is not a problem to solve once,it's a capability to build permanently.

The teams that master evals will ship better products. They'll make better decisions about what to build, when to launch, and where to restrict. They'll communicate more clearly across teams and avoid the expensive failure of shipping an AI product that doesn't do what users need. And they'll be at the center of the next wave of AI innovation.

Start small. Pick a domain you understand. Write a hundred prompts. Score them. Calibrate. Iterate. You don't need a fancy platform,you need a spreadsheet and the discipline to actually measure what matters.

Conclusion: Make Evals Your Superpower

Evals have become the central mechanism for defining, building, and governing AI products. Traditional product documentation can't capture the open-ended, probabilistic nature of modern AI systems. Executable evaluation suites now carry the burden of specifying what AI systems should do, how well they must do it, and where they must not be trusted.

The principles we've covered,deep domain understanding, Goldilocks task design, automatic scoring, and tight integration of evals into the product lifecycle,apply across sectors. We saw them in action in clinical genomics, where they're helping diagnose sick babies and expand access to life-saving technology. The same patterns hold in finance, law, education, and beyond.

For PMs, engineers, and leaders, the path forward is clear. Become fluent in designing and interpreting evals. Treat them as living contracts between user needs and AI behavior. Use them to steer your organization through the reinvention that AI demands. Those who master evals won't just build better products. They'll help define the standards by which future AI systems are judged. The skills you've learned here are the foundation of that mastery,now go apply them.

Frequently Asked Questions

Introduction

This FAQ compiles the essential questions and answers about AI evaluation methodology as covered in the masterclass. It serves as a practical reference for product managers, engineers, and technical leaders who need to build, interpret, and communicate AI evaluation systems. The questions progress from foundational concepts through advanced implementation techniques, addressing both the "why" and the "how" of modern AI evals. Each answer provides actionable guidance that can be applied directly to real-world product development.

Foundations of AI Evaluations

What is an AI evaluation (eval) in the context of generative AI?

An evaluation, or eval, is a structured method for measuring how well an AI model performs a specific task or set of tasks. It typically consists of a set of prompts (inputs), an answer key or expected outcomes, and a scoring mechanism to determine whether the model's output is correct or acceptable.

In practice, an eval acts like a set of "trivia questions" for the model. It defines the shape of what the model needs to do well. If the model performs well on a carefully designed eval set, that is a strong signal that real users will be satisfied. Evals are therefore the core tool for communicating product requirements, guiding engineering improvements, and deciding when something is ready to ship.

What does the claim "evals replace the PRD" mean?

Traditional Product Requirements Documents (PRDs) spent most of their length describing the specific behaviors of a feature: how it would work, how it would behave in edge cases, and what the user experience should look like. Generative AI products, however, are open-ended. They can handle a nearly infinite range of inputs and situations, so describing every behavior in prose becomes impractical.

Evals replace the descriptive part of the PRD by using concrete examples. Instead of saying "the product should generate pleasing images of kitchens," you provide a set of example prompts paired with expected outputs, plus a scoring rubric. That eval set becomes an executable specification: if the model passes the eval, it effectively does what the PRD was trying to describe. This shift makes "success" measurable rather than interpretive.

Keep in mind that evals do not eliminate the need for product strategy or customer knowledge. You still need to know who the user is and what they value. But the actual definition of "good" becomes a collection of examples, not a written description.

What is the difference between an offline eval and an online eval?

An offline eval is run during development, before users ever see the product. You prepare a pre-baked set of prompts and expected answers, then feed those prompts through the model and score the outputs. For example, a recipe website could create 100 prompts asking for different ice cream recipes, with a checklist for what constitutes a good recipe. Offline evals are fast, repeatable, and allow you to catch serious failures before releasing anything.

An online eval happens after the product is live. You measure how real users interact with the product, using metrics like satisfaction, retention, and task completion. The assumption is that if your offline eval is well designed and the model scores well on it, then online user satisfaction will also be high. This assumption is not always correct, which is why monitoring real-world performance remains essential.

What are the basic steps for creating an offline eval?

The process is straightforward in structure, though it requires judgment and domain knowledge:

1. Define the problem. What exactly are you asking the model to do? What types of inputs will users provide?
2. Determine how to measure success. What does a correct or good output look like? Identify the key components that must be present.
3. Collect a representative set of prompts. Usually around 100 prompts is a solid starting point. They should reflect the diversity of real user inputs.
4. Choose a scoring method. Options include automated checks, human reviewers, or another LLM acting as a judge.
5. Run the eval. Send the prompts through the model, collect outputs, and score them.
6. Analyze and iterate. See which prompts are too easy or too hard, identify failure patterns, and share the results with the team.

The goal is to create an eval set with enough headroom to measure improvement. If the model scores 100% immediately, the eval is too easy. If it scores 0%, the task may be impossibly hard with current technology.

What is a "Goldilocks" set of prompts, and why does it matter?

A Goldilocks set of prompts is neither too easy nor too hard. It is calibrated to the model's current abilities so that success rates start around 25-50%, creating room for the engineering team to optimize and improve.

If every prompt is trivially easy and the model gets 100%, there is nothing to improve and the eval no longer provides useful signal. Conversely, if every prompt is impossibly hard and the model gets 0%, you do not know whether the task is simply beyond current technology, and the team cannot make meaningful incremental progress.

As the model improves over time, a good eval set will gradually saturate. Once the model achieves near-perfect scores, it is time to retire that eval set and create a new, harder one. This continuous cycle keeps evals aligned with evolving model capabilities.

Scoring Methods and Measurement

How should outputs be scored in an AI eval?

Scoring depends heavily on the domain and task type:

Deterministic domains (math, coding, data extraction) often have clearly correct answers that can be checked automatically.
Open-ended domains (creative writing, image generation, complex analysis) may require human raters or rubric-based evaluation.
LLM-as-a-judge is increasingly common: a separate model compares the output to the ground truth and makes a pass/fail decision.

For agentic tasks, scoring is more challenging because the final output is the result of many intermediate steps, and some steps could be wrong yet still lead to the right answer, or vice versa. Automated scoring is especially valuable in these cases because it allows many rollouts and repeated experiments, whereas human review is too slow and expensive at scale.

What are the three primary scoring methods and when should each be used?

The three scoring approaches are automated scoring, LLM-judge scoring, and human scoring.

Automated scoring uses deterministic rules to evaluate outputs. It works best for tasks with clear right answers, such as code, mathematical problems, or data extraction. If the expected answer can be expressed as a specific value or pattern, automated checks are fast and reliable.

LLM-judge scoring uses another language model to compare the output against the expected answer. This works well when the expected answer is clearly defined and the model's output either contains or clearly references the correct answer. The judging model needs unambiguous criteria to make reliable pass/fail decisions.

Human scoring is reserved for worst-case scenarios where automated approaches fail. Creative domains,writing, image generation, subjective analysis,often need human evaluation because preferences vary between users and quality is difficult to define formally. The trade-off is speed: human review is slow and expensive, so it should be used sparingly.

How do you build a scorecard for an agentic eval?

A scorecard maps each eval prompt to its expected answer and defines the criteria for a pass or fail. For agentic evals, this requires more thought than simple QA tasks because the path to the answer may vary.

Start by listing every prompt in your eval set. Next to each prompt, write the ground truth,the specific outcome that constitutes success. For a genomics example, the ground truth might be "identify the p.Phe508del variant in the CFTR gene." For a coding task, it might be "produce a function that passes these test cases."

Then define the comparison logic. A secondary LLM reads the model's output, compares it to the scorecard entry, and determines whether the output matches. The judging instructions must be specific enough to avoid false positives. For instance, if a model identifies the correct gene but the wrong variant, that should be marked as a failure, not a partial credit.

The scorecard also needs to account for multiple valid paths to the same answer. If the task allows different approaches, the judge must recognize any valid approach that reaches the correct outcome.

Evolution of AI Benchmarks

Why have traditional QA benchmarks like MMLU become less relevant?

Traditional question-and-answer benchmarks such as MMLU, HellaSwag, and ARC served an important role in the early days of large language models. They measured knowledge retention and reasoning ability. However, frontier models have now saturated these benchmarks, scoring near 100% on many of them. That means they no longer differentiate between models.

More importantly, QA is no longer the primary use case for generative AI. The headline applications today are agentic: systems that write code, browse the web, use tools, analyze financial data, and perform multi-step knowledge work. Consequently, modern frontier model announcements use entirely different benchmarks that measure agentic coding, agentic terminal use, and agentic reasoning. QA benchmarks were a foundational step, but the frontier has moved on.

What were the major QA-era benchmarks and what made them useful?

The QA era was defined by several key benchmarks:

MMLU (Massive Multitask Language Understanding) tested knowledge across 57 subjects using multiple-choice questions. HellaSwag measured commonsense reasoning by asking models to complete sentences in plausible ways. ARC (AI2 Reasoning Challenge) used science exam questions to test reasoning. Winograd tested pronoun resolution and commonsense reasoning. HumanEval measured Python code generation. DROP tested discrete reasoning over paragraphs.

These benchmarks were useful because they provided standardized, repeatable measurements of model capability. A model that scored well on MMLU demonstrated broad knowledge. A model that scored well on HumanEval could generate functional code. Teams could compare models, track improvements, and make decisions based on quantitative scores.

Their limitation was that they measured single-step tasks. A model received a prompt and returned an answer. That format does not capture the multi-step, tool-using behavior that defines modern agentic systems.

How have frontier AI labs changed their benchmark approach for agentic models?

Frontier labs have shifted their published benchmarks from QA-style tests to agentic evaluations. When Anthropic announced Opus 4.5, the benchmark suite looked completely different from what was used two years prior.

The old suite,MMLU, HellaSwag, ARC, Winograd, HumanEval, DROP,measured single-step question answering. The new suite includes agentic coding, agentic terminal coding, multidisciplinary reasoning, agentic reasoning, agentic computer use, agentic knowledge work, and agentic financial analysis.

This change reflects a fundamental shift in what models are being asked to do. The core task has transformed from "get a prompt and return an answer" to "get a task that requires many steps." Those steps may involve reasoning, tool calling, code execution, or specialized software interaction. The benchmarks now measure whether a system can complete an entire workflow, not just answer a question.

Agentic Evals: Framework and Construction

What distinguishes an agentic eval from a traditional QA eval?

A traditional QA eval sends a prompt and expects an answer. The task is essentially one step. An agentic eval, by contrast, asks the system to complete a task that requires many actions. The system may need to plan a sequence of steps, reason internally, call tools such as search, read and interpret files, and iterate based on intermediate results.

In an agentic eval, you are evaluating not only the final answer but also the system's ability to work through a long-horizon workflow. This makes the problem definition and scoring more complex. The overall framework remains the same,define the problem, collect prompts, establish difficulty, score outputs,but the prompts are tasks rather than simple questions, and the answer key describes successful completion of an entire workflow.

The time horizon also differs. A QA eval takes seconds. An agentic eval can take minutes or even hours. Scoring difficulty increases because the final output is the product of many intermediate steps, some correct and some not.

Can you walk through a practical example of constructing an agentic eval?

As a case study, consider the goal of building an AI system that helps interpret whole-genome sequencing data for medical diagnosis.

First, you choose an intentionally easy condition, such as cystic fibrosis. This is a well-understood single-gene disease with a canonical genetic cause in the CFTR gene. You create a synthetic genome file that contains the exact known mutation. The eval prompt is something like: "This file contains the genome of a patient with symptoms consistent with cystic fibrosis. Please find the genetic cause."

You then pass this task through the system,the model combined with its tools and framework. If it finds the correct CFTR variant, you have confirmed that the task is possible.

Next, you attempt a much harder case: a digenic disease where two heterozygous variants in two different genes together cause congenital heart disease. This is recent, complex research, and most systems will fail without strong external resources or specialized guidance. Running this establishes the ceiling.

Finally, between these two endpoints, you assemble a set of roughly 100 phenotypes and genomes of varying difficulty. Each prompt is paired with the correct answer in a simple scorecard. You run all prompts, compare outputs to the scorecard (often using an LLM judge), and generate a success rate. This gives you a quantitative picture of what the system can do and where it needs improvement.

Why is domain expertise crucial for designing good evals?

Evals in specialized fields are only as good as the knowledge behind them. If you do not know which genetic variants cause a particular disease, or what variant notation looks like, you cannot create meaningful test cases or judge the model's answers correctly.

This is why AI labs are actively hiring domain experts,investment bankers, accountants, lawyers, doctors, and scientists,to build vertical-specific evals. The eval itself is often just a spreadsheet of prompts and expected answers. The hard intellectual work is defining the problem and knowing what "correct" means. Without that expertise, evals become superficial and fail to capture real user needs or technical nuance.

The pattern applies across industries. A bank building a fraud-detection AI needs someone who knows fraud patterns. A law firm building a contract-analysis AI needs someone who knows legal language. The eval builder must be able to say with confidence: "This output is correct because it identifies the specific clause that creates the liability."