In parts of Europe, general-purpose AI tools are being deployed in primary care settings to summarize patient records, suggest diagnoses, and support clinical decisions - often without any regulatory clearance. These deployments expose a growing accountability gap that puts patients at risk while undermining companies that follow the rules.
Some consulting firms and technology companies are installing ChatGPT-like systems directly into clinical workflows. The tools are not limited to administrative tasks. They are beginning to influence how care is delivered, crossing a line that triggers existing medical device regulations - whether the deploying organizations acknowledge it or not.
The core principle is straightforward: what matters is not the technology itself, but its intended use. The moment a system is designed to influence diagnosis, treatment, or patient management, it is no longer productivity software. It becomes a medical device. In Europe, medical devices are governed by the Medical Device Regulation.
The MDR Already Has the Framework
Some public institutions and at least one Ministry of Health from a southern European country have suggested that AI-driven software may require a new regulatory interpretation. The MDR was designed to handle this kind of innovation. It is a risk-based framework that does not depend on whether a product is built on classical algorithms or modern AI. What it cares about is impact: on patients, on clinicians, and on clinical decisions.
The regulation already includes software as a medical device, requires clinical evidence, and enforces traceability and post-market surveillance. As the original analysis put it: "AI does not reduce the need for oversight, it increases it. The more powerful and opaque the system, the greater the need for validation, monitoring, and accountability."
Every CE-marked medical device has a clear chain of responsibility. There is a legal manufacturer, a quality management system, and a technical file documenting how the product was built, validated, and intended to be used. Someone signs off. Someone takes responsibility. For AI for Healthcare tools deployed without clearance, those questions often have no answers. Who is the technical responsible person? Who signed off on its release for clinical use? Who is monitoring performance once deployed?
Accountability Cannot Be Optional
If a court determines that a patient was harmed due to an incorrect diagnosis influenced by one of these systems, the question of who carries legal responsibility is not theoretical. Medicine requires clinical accountability. Decisions must be attributable, auditable, and defensible. General-purpose AI tools are not built to carry that burden. They are powerful, but they are not accountable medical entities.
The more likely future is not a world without regulation. General AI systems will be embedded within or connected to clinically validated and regulated solutions. The intelligence layer may be general, but the clinical application will remain specific, controlled, and accountable.
An Uneven Playing Field
Many companies have spent years building products under the MDR framework, investing in clinical studies, quality systems, and regulatory processes to obtain CE marking. They now compete with new entrants who deploy similar capabilities as "assistive tools" or "decision support," avoiding the same scrutiny. Those who follow the rules are penalized. Those who bypass them gain speed.
Primary care is where most clinical journeys begin, where uncertainty is highest, and where decisions have cascading consequences. Introducing unregulated AI into this environment has the potential to influence diagnoses, delay treatments, or redirect patient pathways in ways that are difficult to detect and harder to correct.
The MDR is not a barrier to progress. "It is the foundation that allows innovation to scale safely. We do not need to bend it. We just need to apply it."
Why this matters for healthcare professionals
Clinicians and healthcare leaders should ask direct questions when AI tools are proposed for clinical use: Has this system received regulatory clearance as a medical device? Who is the legal manufacturer? What clinical evidence supports its use? If the answers are unclear, the safest course is to treat the tool as unvalidated - regardless of how capable it appears. The cost of getting it wrong is measured in patient outcomes, not in lost efficiency.
Your membership also unlocks:
