Most health systems have moved past the experimental phase with artificial intelligence and are now relying on the technology for clinical documentation, billing, and patient engagement. A new report from UPMC finds that many of these organizations are operating without clear guardrails to manage the risks, signaling a gap between deployment and governance.
The report, produced by UPMC's Center for Connected Medicine in collaboration with KLAS Research, surveyed more than two dozen healthcare leaders, including executives from health systems and ambulatory care organizations. It paints a picture of an industry eager to adopt AI tools, but often lacking the infrastructure to apply them safely.
Widespread adoption, weak oversight
The survey found that 93% of health systems have deployed third-party AI tools, but 63% are still developing the internal structures to support them. Those organizations described their policies as "ad hoc" rather than established or advanced. Even fewer - 44% - reported having a dedicated data platform to test AI solutions using models adapted from real-world patients.
Among the most cited uses of AI were clinical documentation, imaging, revenue planning, workforce tools, and productivity assistants. Despite widespread adoption, a majority of organizations are effectively building the plane while flying it when it comes to safety and risk management.
"I think we all see the promise in AI, but systems need to have true guardrails … to implement those tools in an effective way across the health system," said Ken Howard, vice president of UPMC Enterprises, the innovation and venture capital arm of the health system.
Data quality and internal standards
The report argues that without good internal infrastructure, health systems may struggle to validate what their AI tools produce. Respondents said data quality varies widely due to inconsistent definitions across teams, manual workarounds, and the use of spreadsheets filled with unstructured data. Many systems still rely on labor-intensive processes to reconcile data before it can be used by an AI tool.
UPMC has built its own platform, called Ahavi, to test AI applications without disrupting patient care. Howard said the platform uses de-identified patient data to mimic real-world scenarios. For healthcare professionals looking to build governance in practice, this type of internal testing is exactly where an AI for Healthcare framework can support decision-making.
No consensus on what success looks like
The survey found a broad lack of consensus among health systems about what a successful AI strategy looks like. There is no universal metric to determine the value AI brings to patients or system efficiency. That challenge, the report suggests, extends beyond technical bugs and into procedural issues that persist even after a tool is deployed.
UPMC contends that ongoing governance, testing, and evaluation are essential for AI deployment in healthcare. "What's emerging from this research is a clear recognition that implementation is only the first step," said Rob Bart, M.D., chief medical information officer at UPMC. "Health systems are now focused on building the governance structures, testing capabilities, and organizational strategies necessary to ensure AI delivers meaningful and measurable value."
Why this matters for healthcare and IT professionals
For IT and development teams in healthcare, the report signals that technical deployment is no longer the challenge - governance is. Health systems will be hiring for roles that focus on data quality, validation, and internal policy, not just integration and testing. Professionals who can build the frameworks for responsible AI use - not just select tools - will be in higher demand as the industry moves to close the gap between adoption and oversight. For those in the public sector, studying record on AI safety in this report directly connects to questions of policy guidance, which makes AI for Government resources a relevant additional read.
Your membership also unlocks:
