An AI transcription tool used during a urology appointment generated a false medical record claiming patient Rebecca Green micro-dosed psychedelic mushrooms, an error she only discovered after kidney stone surgery. The fabricated detail appeared in a post-operative letter sent to her GP, where it was cited as a potential cause of prior bleeding around her kidneys.
"I was gobsmacked and in tears … it made up that I take [psychedelic] mushrooms. I have never done mushrooms in my life," Green said.
The mistake carried real consequences. Green was receiving workers' compensation and feared the false drug reference could jeopardize her case. She also said the experience damaged her trust in the level of care she received.
"For this letter to say something that wasn't based in any evidence but could potentially have huge consequences for me … it was quite scary," she said.
The urologist later sent a written apology, saying she could not determine how the claim was generated but that it appeared to stem from an error during "dictation or transcription." She corrected the record and said she would review her use of AI. The doctor did not respond to follow-up questions about what changes were made.
Hallucinations pose clinical risks
Research from advocacy group Digital Rights Watch (DRW) found the incident reflects a broader pattern. AI medical scribes in Australia are largely unregulated, often operate without proper consent, and have produced errors that could change clinical decisions. One documented case involved a scribe recording the wrong breast in a breast cancer diagnosis; another claimed a patient had epilepsy when they did not.
Elizabeth Deveny, chief executive of the Consumers Health Forum of Australia, said patients frequently report finding errors only by chance. She added that doctors may become overly trusting of the technology over time.
"There's evidence that for the first few weeks of using AI, [people] check things pretty carefully and then at a point they stop checking because they assume it will be right," she said.
Perth GP Sean Stevens, who has used a scribe for three years, said the onus is on the doctor to verify every transcript. He acknowledged the technology still produces plausible-sounding but incorrect results.
"The classic is, it will get the side of the body incorrect and say right when you've said left … so you need to check things pretty closely, especially things like drug doses," he said.
Consent is inconsistent across practices
DRW found consent procedures vary, with some practices using only a sign in the waiting room and others requiring consent during online booking or refusing appointments if the patient declines. Deveny said consent should be "enthusiastic and fully informed," and that patients should feel comfortable asking questions even if it feels intimidating.
Stevens said he asks for consent at every appointment, including with regular patients, because people may withhold sensitive information like family violence when they know AI is recording.
Regulation gap remains unresolved
Around a dozen AI scribes operate in Australia, and none are approved by the Therapeutic Goods Administration (TGA). Scribes that only transcribe and summarize are exempt from oversight, while tools that suggest diagnoses or treatment plans technically require approval. The TGA has acknowledged some scribes may be operating in breach of the law.
The TGA's review findings are expected in the coming weeks. DRW is calling on the AI Safety Institute to test scribes and publish a list of approved tools, similar to New Zealand's approach, where two AI tools were approved for clinical use last year.
Deveny said safer use could also be enforced through mandatory checks built into the software or during the three-year practice accreditation cycle used for medical practices.
Patients are often unaware of the mistakes in their records, which can affect their health, employment, or insurance. As healthcare professionals integrate these tools, the question is not whether AI scribes will be part of the clinical workflow, but how to ensure they are used in a way that does not compromise patient safety.
Why this matters for healthcare workers
This situation is a practical reminder for clinicians to verify every AI-generated note before it is used or sent. For those developing the current AI tools, the stakes are clear. The TGA is reviewing whether transcription-only tools that make clinical suggestions need to be regulated, and errors like the one in Green's case could accelerate that process for your product.
For professionals in health administration, the case highlights the importance of building systematic checks into AI documentation. It is worth considering whether your practice can bin the sole responsibility of the treating doctor, which clearly can fail, with other professional checks that can be integrated into the workflow.
If you're looking for training that covers these practical challenges, AI for Healthcare Courses offer guidance on using AI tools in clinical settings. For those managing records directly, the AI Learning Path for Medical Records Clerks covers the fundamentals of handling sensitive medical data with AI.
Green said her experience has made her far more cautious about AI in medical consultations going forward.
"I love AI … but the essential ingredient is the human. At the end of the day what's happened here is more about the lack of human involvement," she said.
Your membership also unlocks:
