Google's SymptomAI beats clinicians in 13,917-person diagnosis study

Google Research published a paper called "SymptomAI: Towards a Conversational AI Agent for Everyday Symptom Assessment," describing what it calls a first-of-its-kind national-scale study of AI for symptom checking and differential diagnosis (DDx, the ranked list of plausible diagnoses a clinician draws up from a patient's symptoms). The team says prior evaluations of language models on diagnosis relied on curated, sometimes synthetic case vignettes that do not reflect how ordinary patients actually describe symptoms, with incomplete information and varying medical literacy. This study instead enrolled 13,917 consenting research participants who each described their symptoms to one of five randomized SymptomAI agents, all running on Gemini Flash 2.0. Two weeks later, participants were surveyed on any diagnosis they received from an actual healthcare provider visit, giving the researchers a real-world outcome to check against.

The five study arms tested different interview styles: two agents (Dynamic Live and Dynamic Final) had full agency to ask unrestricted follow-up questions, two more (Fixed Canonical and Flexible Canonical) worked from a set of standard history-taking questions taught in medical school, and a Base arm was an unprompted language model representing the user-driven experience typical of a general chatbot today. Every agent-driven strategy, meaning any variant that actively asked follow-up questions, significantly outperformed the unprompted Base condition on diagnostic accuracy.

To grade SymptomAI's output, the researchers ran a blinded clinical-expert annotation study: a panel of three board-certified clinicians independently read the conversation transcripts, produced their own differential diagnosis, and then each clinician ranked the DDx generated by SymptomAI against those produced by the other two clinicians. SymptomAI's DDx was preferred over a peer clinician's in more than half of cases. Using top-5 accuracy, meaning whether the diagnosis the participant later reported from their own doctor visit appeared among the five candidates in a DDx, the clinicians also judged SymptomAI's DDx to be accurate more often than the DDx produced by other clinicians. The researchers report SymptomAI's advantage over clinical baselines was largest in exactly the cases where the reviewing clinicians felt least confident in their own diagnosis.

The team then cross-checked SymptomAI's diagnoses against biosignal data collected for up to 30 days before each participant's conversation from their Fitbit wearables. For cases where SymptomAI's top candidate diagnosis fell into a respiratory-infection category (excluding non-infectious conditions such as allergic rhinitis or COPD), the researchers found clear shifts in cardiovascular function, respiration, skin temperature and sleep quality in the days leading up to the symptom conversation, peaking around the date of symptom reporting. They present this as independent physiological evidence aligning with participants' reported illness, and argue that an accurate AI symptom checker could eventually enable automated, clinical-quality labeling of population-scale health datasets, something too costly to do with clinician labels alone today.

The authors flag several limitations. Differential diagnosis is inherently ambiguous and can shift over time, and because the study did not control for when participants chose to report symptoms, some may have reported early, unclear signs while others reported only after years of experience with a chronic condition. The clinician panel also reviewed static transcripts and could not ask their own follow-up questions, unlike a real consultation, and the researchers note that conversational AI in general may still miss cues such as body language, direct visual assessment, medical records or existing patient rapport. The paper stresses throughout that all diagnoses, labels and disease associations from the study were generated for research analysis only and do not constitute confirmed clinical diagnoses or official medical assessments. The work is credited to equal contributions from Joseph Breda, Jake Sunshine and Daniel McDuff, with the blog post written by Breda, a Student Researcher, and Sunshine, a Research Scientist, at Google Research.

Key facts

13,917 consenting participants each held a symptom conversation with one of five randomized SymptomAI agents built on Gemini Flash 2.0.
A blinded panel of three board-certified clinicians preferred SymptomAI's differential diagnosis over a peer clinician's in more than half of cases.
On top-5 accuracy against participants' real later diagnoses, clinicians judged SymptomAI's DDx accurate more often than DDx produced by other clinicians.
Every agent-driven interview style beat an unprompted Base language model, and SymptomAI's edge over clinicians was largest when clinicians felt least confident in their own diagnosis.
Cross-checked against up to 30 days of Fitbit biosignal data, respiratory-infection diagnoses lined up with shifts in cardiovascular function, respiration, skin temperature and sleep quality around the time of symptom reporting.

Why it matters

Doctor-patient interviews remain the gold standard for diagnosis but are limited by financial, geographic and systemic access barriers, and prior AI diagnosis research leaned on curated or synthetic case vignettes rather than how real patients actually describe symptoms. This study is Google Research's attempt to validate a conversational diagnostic agent at population scale, with clinicians as the judges rather than the model grading itself.

Who it affects

The result speaks most directly to researchers and health systems weighing AI-assisted triage or symptom checking, and to Google's own health AI work built on its Gemini models. It also matters to telehealth and population-health researchers, since the paper argues an accurate symptom checker could enable automated labeling of health datasets at a scale that is currently too costly with clinician review alone.

How to use it

SymptomAI is a research prototype described in an academic paper, not a shipped product; the blog post gives no launch date, pricing or public availability, and every diagnosis it produced during the study is explicitly labeled research-only rather than a confirmed clinical assessment.

How solid is it

The study design is comparatively rigorous for this space: 13,917 participants, five randomized agent variants, a three-clinician blinded panel judging both preference and top-5 diagnostic accuracy, and an independent physiological check using Fitbit biosignal data that lined up with respiratory-infection diagnoses. The researchers frame it explicitly as a first-of-its-kind national-scale validation rather than a finished, deployment-ready system.

Risks and caveats

Differential diagnosis is inherently ambiguous and can change over time, the study could not control when participants chose to report symptoms, and the clinician panel reviewed static transcripts without the ability to ask their own follow-up questions or observe patients directly. The authors also note conversational AI may miss cues like body language, visual assessment, medical records or existing clinician rapport, and they repeatedly stress that no diagnosis from the study is a confirmed clinical result.

“We found that the clinicians preferred the DDx generated by SymptomAI over those provided by the other clinicians in over 50% of the cases.”

— Joseph Breda and Jake Sunshine, Google Research