Vendor demos are designed to impress. But Eshaan Jain, a lead product owner who has built real machine learning systems at Amazon, PwC, and Accenture, has watched too many of those impressive demos fail within two weeks of a pilot. He calls the disconnect "AI-washing" - labeling static, rules-based software as autonomous agents because "AI-powered" has no agreed definition.
The cost of that confusion is measurable. A 2025 MIT NANDA study found that 95% of enterprise generative AI pilots delivered no measurable P&L impact, against $30 to $40 billion in spend. A large share of that money went to products that were sold as adaptive and were actually static.
Jain has developed a nine-question audit designed to strip fake AI labels off product roadmaps before they ship. He shared the questions in a guide for product teams.
Nine questions that separate AI from if-statements
Jain's first question is the most direct: what decision does the system make without a human? An agent takes an action and accepts the consequence. A chatbot returns text and waits for a person to act. If the only answer is "it suggests, then a human clicks," it's a recommendation feature, not an agent.
Other questions target the system's behavior under real-world conditions. Ask what happens on an input the system has never seen. Deterministic systems fail loudly with an error or a default return; a real model degrades in a more graceful, probabilistic way. Ask whether the system can explain why it made a choice. "If you ask 'why did it choose that?' and the answer is a flowchart, the intelligence lives in the flowchart, not in a model," Jain said.
The toughest question on the list may be architectural: trace the request end to end and mark where the model output changes an outcome. "Many 'AI' features call a model for one cosmetic step, like rewording a confirmation message, while every real decision runs through hard-coded logic," Jain explained. If no such line exists, the model is decoration.
The audit also covers economics and data. Jain asks teams for token cost per action, because a team running genuine agents knows that number from its monthly cloud invoice. He also asks what the team spent on data preparation, not just model access. "Real machine learning needs clean, labeled, structured data," he said.
Three honest labels
After running the nine questions, Jain recommends retagging every feature into one of three categories:
- Agent: makes an unsupervised decision, adapts to data, has a real model in the decision path, and carries a measurable inference cost.
- Model-assisted: uses a model for a real step, but a human makes the call.
- Rules: deterministic logic. Often the right tool for the job.
Jain said the point is not to chase the most sophisticated label. He has moved features from his own roadmaps back through these categories. It is not a downgrade. "A clean rules engine that closes a quote in seconds beats a fragile agent that fails an audit," he said. The label needs to match the build so the next person who funds the work knows what they're paying for.
For product professionals building or evaluating
roadmaps, the budget stakes are enough to justify auditing any AI claim. The MIT finding alone suggests that most projects burn money without measurable impact. But the more durable issue is trust - internally with engineers and externally with customers. A team that calls deterministic software an agent reveals that the gap between its claims and its code widens, according to Jain. If leadership reviews the repo and sees static rules, every future AI claim gets scrutinized. Real machine learning requires the unglamorous data work first: labeling, validation, catalog hygiene. Teams that spend less time on labels and more on that foundation are the ones whose features ship and work.
Your membership also unlocks:
