# We Built a Brilliant AI Search-And a Bad Business

> A dazzling AI search demo hit 0.92 nDCG, then flopped in the real world. Why it failed: workflow friction, costly queries, and vitamin value-plus a simple Product P&L test.

- Type: AI news
- Page: https://completeaitraining.com/news/we-built-a-brilliant-ai-search-and-a-bad-business/
- Published: 2026-01-06
- Updated: 2026-02-19
- Author: Jeroen Erne
- Jobs: Product development
- Reading time: 4 min

### Why Technically Great AI Products Fail: A Case Study and the Product P&L Test

Teams feel the squeeze to "ship AI" in 2025. Boards demand it, competitors hype it, and product roadmaps bend around it. But shipping a feature that looks brilliant in a demo means nothing if it breaks the business.

Case in point: a technically impressive AI search tool with near-perfect relevance launched and fell flat. It cleared the accuracy bar, then failed on adoption, margin, and retention-the only bars that matter.

#### The Setup: Magic in the Demo, Silence in the Market

The team built a modern RAG pipeline with a vector database and a top-tier LLM. The relevance score jumped from 0.65 to 0.92 nDCG-near textbook performance. If you care about metrics, that's a serious leap ([nDCG](https://en.wikipedia.org/wiki/Discounted_cumulative_gain?ref=completeaitraining.com), for reference).

On launch, usage flatlined. Not because the answers were wrong-but because the product created friction, burned cash on every query, and wasn't critical enough to keep users coming back.

### Three Reasons It Failed (That You Can Prevent)

#### 1) The Wrapper Fallacy: Users Don't Want Homework

The "chat with your data" pattern looked slick. In practice, it asked users to stop their flow, open a side panel, type a prompt, wait, copy, and paste. That's a workflow tax, not a workflow boost.

Users don't want to search. They want to be done. The feature was shipped as a destination instead of a utility-when it should have delivered answers inline, without prompting, and without cognitive overhead.

#### 2) The COGS Nightmare: Great Accuracy, Broken Unit Economics

Chasing the highest accuracy led to expensive models and aggressive retrieval. Each complex query averaged ~$0.08 in compute. Pricing was a flat $29 per seat per month.

Do the math: 15 queries per day from a single power user flipped a profitable customer into a loss. The more they loved it, the more it hurt margins. That's not a feature-it's a subsidy.

#### 3) The Vitamin Problem: Nice-to-Have Doesn't Retain

When the feature went down for a few hours, no one escalated. That silence said everything. It wasn't a painkiller; it didn't block the job if removed. It demoed well and changed nothing.

### The Product P&L Test (Use This Before You Build)

Before investing six months into an AI feature, force it through these three tests.

- **The Value Test: Did We Remove Labor?**  Trap: The AI produces a draft that still needs heavy editing and copy/paste.  Win: The task is automated end-to-end or reduced to a one-click confirm.
- **The Margin Test: Can We Afford to Win?**  Trap: Unlimited AI inside a flat subscription with no guardrails.  Win: Usage-based pricing or firm caps, plus cheap defaults and model fallbacks.
- **The Retention Test: Is It a Painkiller?**  Trap: "We'll just do it the old way."  Win: "We can't do the job without this."

### How to Build AI That Works in a P&L

#### Design for Flow, Not Demos

Prioritize [Design](https://completeaitraining.com/tag/design/) that integrates answers into the user's flow rather than forcing a separate destination.

- Inline answers where work happens: suggest, auto-complete, or fill fields quietly.
- Zero-prompt defaults: trigger on events (form open, field focus, save) to help without asking.
- Kill copy/paste: write back to the right place automatically with a one-tap confirm.

#### Engineer for Cost First, Accuracy Second (In That Order)

- Model cascade: small/fast models by default; escalate only on low confidence.
- Aggressive caching and deduplication: don't pay for the same question twice.
- Cheaper retrieval: filter and chunk smartly; keep vectors lean; precompute frequent joins.
- Guardrails: per-user budgets, per-tenant rate limits, nightly audits of cost per action.

#### Price So You Don't Pay to Lose

- Usage-based pricing or credits for AI calls; reserve flat fees for non-AI value.
- Fair-use caps and overage tiers that align heavy usage with higher revenue.
- Separate SKUs for "assist" vs. "automation" features; automation should command premium pricing.

#### Make It a Painkiller

- Attach the feature to a critical job-to-be-done with clear before/after time savings.
- Own the final artifact (submission, report, approval), not just the draft.
- Offer SLAs where it counts: if it breaks, work stops-and customers feel it.

#### Measure What Matters

- Time-to-complete vs. baseline (not clicks or prompts).
- Percent tasks fully automated (zero edits).
- Cost per successful action and margin per user segment.
- Retention impact: feature-off churn risk, expansion tied to usage, NRR lift.

### A Simple Rule for Product Leaders

Don't ship AI for the spectacle. Ship AI that kills labor, defends margin, and locks in retention. If it can't pass the Product P&L Test, it's a whiteboard idea-not a roadmap item.

If your team insists on an AI search UI, make it invisible: answer in context, automate the handoff, and price usage so winning doesn't sink you. And if you're starting from scratch, consider whether RAG even needs a chat box-many use cases don't ([RAG overview](https://en.wikipedia.org/wiki/Retrieval-augmented_generation?ref=completeaitraining.com)).

#### Next Steps

- Run your AI feature through the three tests with real usage data, not demo anecdotes.
- Set cost guardrails before launch, not after the first scary invoice.
- Rewrite your UX to remove prompts, clicks, and copy/paste wherever possible.

Want structured upskilling on practical AI for product teams? Explore courses by job at [Complete AI Training](https://completeaitraining.com/courses-by-job/).

---

This is the Markdown form of https://completeaitraining.com/news/we-built-a-brilliant-ai-search-and-a-bad-business/. The whole site is mapped at https://completeaitraining.com/llms.txt.
