Health systems have moved past the question of whether to deploy generative AI. The urgent challenge now is verifying that clinical AI remains accurate, transparent, and accountable once it enters daily practice. For CIOs and governance committees, the conversation has shifted from productivity gains to building an ongoing assurance process that detects failures and protects patients as models evolve.
When confidence becomes the problem
Lior Eshel, CEO of AI-powered medical imaging company TestDynamics, said healthcare leaders should worry less about dramatic AI failures than about ordinary workflow tools that quietly become trusted. Generative scribes can introduce facts that never appeared in a patient encounter. Predictive algorithms may influence care for years before anyone evaluates their real-world accuracy.
"A model cannot flag its own hallucination because the model that is wrong, by construction, is confident that it is right," Eshel said. Detection cannot depend on the model itself. Hospitals need independent comparison against verified clinical data and, where possible, against other tools performing the same task.
For clinicians, inaccurate AI creates two opposite risks. Frequent false alerts contribute to alert fatigue, while confident, polished responses encourage overreliance. Patients may receive inaccurate information, and organizations inherit legal, financial, and reputational exposure.
Bias rarely announces itself
Hallucinations represent only one dimension of AI risk. Bias may be embedded within training data, proxy variables, or commercial optimization objectives long before a hospital purchases the technology. "Evaluating biases before deployment is, therefore, concrete work, not a values statement," Eshel said.
He recommends that buyers request detailed information describing training populations, demographic composition, validation methodology, and participating clinical sites. Organizations should also evaluate products using local patient populations rather than assuming vendor-reported performance will translate across different communities and workflows. Validation is local - models developed under one set of clinical conditions may perform differently once exposed to another institution's patients, documentation practices, and disease prevalence. For healthcare organizations building internal expertise in this area, AI for Healthcare training resources can help teams develop the evaluation skills these assessments demand.
Governance after go-live
Healthcare already monitors pharmaceuticals and medical devices after they reach the market. Eshel believes clinical AI deserves similar treatment. Hospitals should validate systems before implementation, continuously monitor performance afterward, and preserve the underlying source material needed to audit AI-generated outputs. Without those records, organizations may never determine whether documentation accurately reflects the clinical encounter.
"Shared accountability is the only workable model, and it carries a precondition: shared visibility," he said. Shared visibility requires comprehensive logging, reproducible testing, and independent measurement rather than accepting static validation performed elsewhere. Governance becomes a continuous operational discipline instead of a one-time procurement exercise.
The CIO agenda
Eshel described the immediate steps as straightforward. "The steps to take now are not exotic. They are procurement disciplines applied to software that makes clinical claims." Health systems should begin with a complete inventory of AI models already operating across the enterprise, followed by contractual rights to independent testing, local validation, and ongoing access to performance data.
As AI applications multiply, evaluating products individually becomes impractical. Eshel envisions an independent evaluation layer capable of comparing competing systems using common benchmarks over time. Such a framework would give healthcare organizations objective evidence about how clinical AI performs after deployment instead of relying primarily on marketing claims. For CIOs navigating these responsibilities, the AI Learning Path for CIOs addresses governance, strategy, and implementation specifically for technology leaders in this role.
Whether that model ultimately emerges through regulators, industry collaboratives, or independent organizations remains uncertain. What appears increasingly clear is that AI governance is becoming a core executive responsibility. For CIOs, success will be measured not simply by how quickly AI is implemented, but by whether leaders can demonstrate that systems remain transparent, auditable, and clinically trustworthy throughout their operational life.
Why this matters for healthcare professionals
Clinicians and operational staff will encounter AI outputs daily - in documentation, imaging analysis, and decision-support tools. The safeguards that catch errors before they reach patients depend on governance decisions made now. Healthcare professionals should press for clarity on how their organization validates AI after go-live, what logging exists to audit outputs, and what independent testing occurs on local patient populations. Trusting the vendor's pre-deployment testing is not enough when the model is operating on your patients, in your workflows, with your data.
Your membership also unlocks:
