Domain Experts, Not Models, Drive AI Product Success
The choice of AI model matters far less than how you organize your team around domain expertise. That's the core argument Chris Lovejoy, founder of Notius Labs, made at a recent AI Engineer Europe event. His framework for building domain-native AI organizations offers product teams a practical structure for scaling AI products without getting lost in model selection.
Lovejoy presented three distinct roles that domain experts fill: Oracle, Evaluator, and Architect. Each serves a different function as products grow.
The Three Roles
The Oracle embeds domain expertise directly into the AI application, typically through prompt engineering or curated training data. This role works well for early-stage products where a single expert can maintain quality.
The Evaluator defines what "good" looks like. This person builds metrics, establishes quality thresholds, and creates systems to measure AI performance against those standards.
The Architect designs systems that improve automatically. They build feedback loops and processes that let the product learn and refine itself without constant manual intervention.
The role you need depends on your specific use case and scale. A startup might start with one domain expert as an Oracle. As the product scales and customer needs diversify, you may need to hire multiple Oracles or shift toward an Evaluator or Architect model.
Common Mistakes Companies Make
Organizations often hire domain experts too late, pick the wrong type of expert, or fail to integrate them into the product development process. The real problem isn't finding the right model-it's operationalizing expert judgment around whatever model you're using.
Lovejoy outlined a decision tree to guide this choice. If AI quality can be measured objectively, focus on defining and measuring performance. If a single expert can iterate fast enough, the Oracle role suffices. If manual iteration becomes a bottleneck, you need an Architect who can automate improvements.
How Three Companies Scaled Their Domain Expertise
Granola generates meeting notes using AI. The company hired Jo Barrow, a writer and journalist with strong research skills, to act as the primary quality gatekeeper. Her Oracle role worked because meeting notes are subjective and the core output is singular-one expert could maintain quality standards.
Tandem builds AI medical scribes. When the company hired a doctor early, it worked well initially. But as the product expanded to support different medical specialties and geographies, Tandem shifted to a decentralized Oracle model, hiring doctors from various regions and specialties to handle local variations in medical practice.
Anterior developed AI for prior authorization decisions. The company started with an Oracle (building prompts and code), moved to an Evaluator role (defining metrics and hiring clinicians to review outputs), and eventually needed an Architect role because prior authorization rules vary too much to manage manually across all cases.
Each case shows the same pattern: start with what works at small scale, then evolve your structure as complexity and customer diversity increase.
What Product Teams Should Do Now
Assign clear accountability to a single principal domain expert. This person becomes the decision-maker and accelerates development by eliminating design-by-committee.
Give that expert real ownership in product development, not just a consulting role. The best products come from domain experts who shape the product, not ones who are asked for input after decisions are made.
Hire for breadth. Look for people with relevant domain expertise and adjacent skills-someone who can grow from Oracle into an Evaluator or Architect as the organization scales. Don't hire narrowly.
The underlying principle: your organizational structure and how you integrate domain knowledge matter more than which AI model sits underneath. Get that right, and you'll build products that actually work in the real world.
For product managers building AI products, understanding how to structure domain expertise across your team is as critical as any technical decision. AI Learning Path for Product Managers covers the strategic decisions that shape how AI teams operate.
Your membership also unlocks:
