Healthcare organizations cannot rely on vendor assurances that artificial intelligence tools are secure and trustworthy. As AI adoption accelerates across the sector, providers need stronger governance, better oversight of third-party vendors, and greater scrutiny of how AI affects patient data and clinical decisions, said Tom Walsh, founder and principal consultant at tw-Security.
Many healthcare organizations lack the staff and time to thoroughly assess hundreds of third-party vendors. Walsh recommends prioritizing attention on vendors - including AI technology firms - that pose the greatest risks based on their access to protected health information and personally identifiable information. Along with those efforts, organizations also need AI governance frameworks, updated vendor assessments, and greater transparency into how vendors develop, validate, and oversee AI capabilities.
Demanding proof of human oversight
"I would ask for better documentation from the vendors to prove what human oversight they've implemented into their AI products and services," Walsh said. "We do not want something bad happening to a patient because we relied on an AI that wasn't accurate."
Walsh warns that AI-powered clinical documentation tools - including ambient scribes - can introduce errors into electronic health records if clinicians fail to review AI-generated notes before signing them. He advises healthcare organizations to strengthen human oversight, revise business associate agreements, and update privacy notices so patients understand how AI collects, processes, and uses their information. For professionals working with medical records, understanding these documentation risks is essential as AI tools become standard in clinical workflows - see AI for Medical Records Clerks for practical guidance.
"I don't think a lot of patients know or recognize that their conversation with a doctor or clinician is being scribed by an AI agent and then placed into their record," Walsh said. "I think that's one area that patients are not aware of how AI is being used in hospitals or healthcare systems today."
Prioritizing high-risk vendors
Not every vendor deserves the same level of scrutiny. Walsh recommends segmenting the vendor population and focusing security and privacy reviews on those with the deepest access to sensitive data. That includes AI vendors that process clinical notes, imaging data, or patient communications.
Shadow AI use is another concern. Employees may adopt consumer AI tools without organizational review, creating data exposure risks that leaders never approved. Walsh said organizations need visibility into these practices and clear policies for sanctioned AI tools. For teams working on healthcare AI governance, AI for Healthcare training offers a foundation for building those policies.
Risks to data integrity and patient safety
Inaccurate AI output in clinical documentation can compound over time. If a clinician signs an AI-generated note without careful review, the error becomes part of the patient's permanent record. That can affect future diagnoses, treatment decisions, and billing accuracy.
Walsh also stresses the importance of governance frameworks that extend beyond procurement. Organizations need ongoing monitoring of how AI performs in real clinical settings, not just a one-time assessment at purchase. That includes tracking accuracy, drift, and patient outcomes.
Why this matters for healthcare professionals
Clinicians and administrators who use AI tools can no longer treat vendor claims as sufficient. The people reviewing AI-generated notes, managing vendor contracts, and communicating with patients about AI use are the last line of defense against errors. Healthcare professionals should verify that their organizations have documented human oversight practices for every AI tool that touches patient data, and they should push vendors for evidence that their systems are validated and accurate.
Your membership also unlocks:
