AI medical scribe invents drug use claim, distressing Australian patient
Rebecca Green agreed to have her first appointment with a urologist transcribed by an AI scribe, saying yes because she did not want to be, in her words, "that annoying patient" who took up more of the doctor's time. The AI tool fabricated a claim that she used psychedelic mushrooms, something she says she has never done. She did not discover the error until March, when she read the post-operative letter from her kidney stone surgery: it stated she had micro-dosed mushrooms and suggested that could explain earlier bleeding around her kidneys. Green called the experience "gobsmacked and in tears," and said the fabricated detail carried real stakes for her since she was receiving workers' compensation at the time and worried that any mention of illegal drugs on her medical record could affect her claim.
After Green complained, the urologist sent a written apology, corrected the letter and said she would review how AI is used in her practice, but could not determine how the mushroom claim was generated beyond suspecting an error in the "dictation or transcription process." She did not answer follow-up questions from the ABC about what, if anything, had changed since. Australia's regulator of medical practitioners, AHPRA, says clinicians must always check AI scribe output for accuracy to meet their professional obligations; Green said it was clear her doctor had not done so in her case.
The incident lands alongside a new report from advocacy group Digital Rights Watch (DRW), which found Australians are being exposed to unregulated AI medical scribes, sometimes without their consent, in ways that can alter clinical decisions. DRW's own examples include a scribe that recorded the wrong breast in a breast cancer diagnosis and another that said a patient had epilepsy when they did not. The Royal Australian College of GPs (RACGP) estimated last year that 40% of GPs were already using AI scribes regularly, though that figure was itself considered conservative. Elizabeth Deveny, chief executive of the Consumers Health Forum of Australia, said patients often only find these errors by chance and then struggle both to trace their source and to get the record corrected; she added that clinicians tend to check AI output carefully for the first few weeks of use, then relax once they assume it will be right.
Not every account is negative. Perth GP Sean Stevens, who has used an AI scribe for three years and chairs the RACGP's Digital Health and Innovation committee (he is also an advisor to one of the biggest AI scribe vendors), said the technology has visibly improved: three years ago he corrected multiple errors per transcript, and now it is closer to one correction every third or fourth transcription. He still checks every transcript, citing a recurring failure where the tool swaps the left and right sides of the body, and says drug doses in particular need close review. He said using AI has not let him fit in more patients, but has let him run closer to schedule and write more thorough referrals.
DRW also found wide inconsistency in how consent for AI transcription is obtained: some practices rely only on a sign in the waiting room, which does not satisfy privacy law, while others require consent at online booking, and some patients report being refused an appointment outright if they decline. Deveny said consent needs to be enthusiastic and fully informed, and that patients should feel able to ask questions even when it feels intimidating to do so. DRW's report separately warned that patients may withhold sensitive disclosures, such as family violence, once they know AI is listening. Stevens said that is why he asks for consent at every visit, even with long-standing patients, since people need to know they are free to say no.
Around a dozen AI medical scribes are sold in Australia, and none carries approval from the country's regulator, the Therapeutic Goods Administration (TGA): a tool used solely for transcription and summarising is exempt from TGA oversight, which applies only when a product suggests a diagnosis or treatment plan. No scribe has gone through the approval process, despite some vendors claiming to offer clinical suggestions, and the TGA has acknowledged that this leaves some products potentially in breach of the law; it says findings from its year-long review of AI scribes will be published "in the coming weeks," with any breaches acted on. Lisa Robins, chief executive of Patients Australia, argues every scribe should be regulated since even summarising a patient involves AI decision-making, while Stevens worries tighter rules would stifle innovation. DRW wants Australia's newly established AI Safety Institute to test AI scribes and publish an approved list, on the model of New Zealand, where the national AI advisory group has already approved two tools, Heidi Health and iMed, for clinical use. Deveny suggested other possible safeguards too, including mandatory checks built into the software itself or scrutiny during practice accreditation, which currently happens every three years.
Green says the episode has fractured her trust in AI transcription: she will now only consent to it with a doctor she already has an established relationship with. "I love AI … but the essential ingredient is the human," she said. "At the end of the day what's happened here is more about the lack of human involvement."
Key facts
- The AI scribe transcribing Rebecca Green's urology visit invented a claim that she used psychedelic mushrooms; the fabrication reached a post-operative letter to her GP and surfaced only when she read it after her March kidney stone surgery.
- AHPRA, Australia's regulator of medical practitioners, says clinicians must always check AI scribe output for accuracy; Green said her doctor clearly had not done so before the letter was sent.
- A Digital Rights Watch report found other AI-scribe errors capable of changing clinical decisions, including a scribe that recorded the wrong breast in a breast cancer diagnosis and one that falsely said a patient had epilepsy.
- None of the dozen or so AI medical scribes sold in Australia is approved by the Therapeutic Goods Administration, because tools used solely for transcription and summarising are exempt from device oversight; the TGA's year-long review findings are due "in the coming weeks."
- Consent practices vary widely: some clinics rely on only a waiting-room sign or refuse care to patients who decline, while New Zealand's national AI advisory group has approved two scribes, Heidi Health and iMed, for clinical use.
Why it matters
The case shows an AI scribe can hallucinate a specific, damaging clinical detail, a claim of illegal drug use, and have it flow straight into an official medical record with real consequences: it affected how Green's own doctor read her case, and she worried it could affect a workers' compensation claim. Australian oversight has not caught up with the technology: none of the dozen or so AI medical scribes sold in the country are approved by the TGA, because tools billed purely as transcription and summarising are exempt from device regulation.
Who it affects
Patients whose visits are recorded: the RACGP estimated last year that 40% of Australian GPs already use AI scribes regularly, a figure itself considered conservative. Doctors carry the compliance burden too, since AHPRA's rules require clinicians to check every line of AI output themselves. And the dozen or so AI scribe vendors selling into the Australian market currently operate without going through TGA approval.
How to use it
The source names no product involved in Green's own case, and no pricing for any AI scribe. It does name two tools cleared for clinical use in New Zealand, Heidi Health and iMed, by that country's national AI advisory group, and reports that Digital Rights Watch wants Australia's new AI Safety Institute to test scribes and publish a similar approved list. Until such a list exists, the practical safeguard described by both a patient advocate and a working GP is the same: check every AI-generated line yourself, especially details like drug names, doses and which side of the body is involved.
How solid is it
The reporting rests on a named, on-the-record case (Rebecca Green) plus a cited watchdog report (Digital Rights Watch) that supplies two further concrete error examples, a misidentified breast in a cancer diagnosis and a false epilepsy claim, and on responses from named regulators, AHPRA and the TGA. The RACGP's 40% usage figure is flagged in the source itself as a conservative estimate, and the claim that AI transcription eases GPs' administrative burden is attributed only to unspecified "research," not a named study.
Risks and caveats
The urologist could not explain how the mushrooms claim entered the transcript, only that it looked like a "dictation or transcription process" error, so the actual failure mode is unconfirmed. The TGA's review of AI scribes was still unpublished at the time of reporting, with results promised only "in the coming weeks" and no detail on what enforcement would follow. And DRW's own findings suggest some patients may be recorded without adequate legal consent to begin with: some practices rely on nothing more than a waiting-room sign, which does not meet privacy law, and some refuse an appointment outright to anyone who declines.
“I love AI … but the essential ingredient is the human. At the end of the day what's happened here is more about the lack of human involvement”
— Rebecca Green