Agentic AI in Support: Shipping Is Harder Than the Hype
Agentic AI has become a buzzword. Building a reliable customer service agent is the real test. Intercom's Fin has been live since March 2023 and its latest version launched this year. With a 65% average resolution rate, the goal is clear: keep learning until "the mythical 100%."
From Factory Floor to Lab Bench
Classic SaaS development was linear: research, design, build, test, ship, iterate. AI broke that certainty. You can't assume a feature works the same way twice. You need a lab mindset.
- Run experiments constantly (and expect some to fail).
- Decide with empirical evidence, not intuition.
- Evaluate frequently with consistent test suites.
- Refresh data, prompts, and policies as often as code.
Train the Agent-and the Humans
Fin is not a single model; it's a system with dozens of moving parts that get tuned and tested. The surprising unlock isn't raw intelligence. It's access to the right knowledge. Procedures, product changes, and policy details drive accuracy.
The loop matters. Fin learns from what it fails to resolve. Each handoff to a human becomes training data. One canonical answer written once should prevent repeat escalations.
- Map your top intents and write canonical, step-by-step resolutions.
- Pipe every human-resolved ticket back into the knowledge base.
- Sync product change logs, release notes, and policy updates automatically.
- Set guardrails for tone, refunds, escalations, and compliance.
Shortening the Human Loop
The handoff isn't a throw-over-the-wall moment. The agent follows the conversation to see what actually solved the issue. Over time, fewer tickets reach a human. Quality goes up, costs go down.
What's Next: Self-Training Agents
The near future: agents that read product updates, policies, and press releases by default. The bigger vision: parsing source code to infer behavior. If an agent can interrogate internal systems and code, coverage climbs and edge cases shrink.
Customer behavior matters too. People get better outcomes when they speak to a bot like it's a person-clear context, one question at a time, with specifics.
What Product Leaders Should Own
A recent survey indicates most people expect customer service organizations to identify AI opportunities, roadmap the evolution, and drive adoption. That's a product problem, not just a tooling problem. Own the system end to end.
- Define success: containment rate, CSAT, time to resolution, cost to serve.
- Build a data pipeline: live knowledge base, change logs, access control, approvals.
- Create an evaluation harness: golden test sets, regression checks, hallucination tests, adversarial prompts.
- Design clear workflows: escalation trees, human-in-the-loop reviews, canonical answer publishing.
- Risk and compliance: PII handling, redaction, audit trails, fallback policies (see NIST AI RMF).
Four Qualities of AI Product Winners
- Real usage, not shelf-ware. If it isn't used weekly by real teams, it doesn't count.
- Solves a hard business problem. Not a cool demo-measurable operational impact.
- Deeply differentiated AI. A gnarly core problem that's hard to copy limits convergence.
- Path to profit. Unit economics that improve with scale and learning.
A Practical Playbook You Can Apply This Quarter
- List your top 50 support intents by volume and cost; write canonical procedures for each.
- Instrument every handoff; convert the final human solution into a reusable answer.
- Automate ingestion of release notes, policy changes, and outage updates.
- Ship a weekly evaluation run against a fixed test set; publish a scorecard.
- Set thresholds for safe autonomy; below threshold, auto-escalate with full context.
- Track three numbers: containment, CSAT on bot-resolved tickets, and cost per resolution.
- Run a "first-time, last-time" sprint: eliminate repeat escalations for your top 20 intents.
Proactive Support Is the Next Battleground
Treat every error state, red banner, or confusing screen as the start of a support conversation. Don't wait for tickets-prevent them. Product and support will blend, and the teams who design for proactive help will win.
If you're upskilling your product org for AI-driven support, explore practical curricula by role at Complete AI Training.
Your membership also unlocks:
