AI Chatbot Aids Physicians in Groundbreaking Study
Researchers have wrapped up a fascinating study showing how an AI chatbot has been assisting primary care physicians by engaging with patients in a clinical setting.
An eight-month clinical study, which tracked the interactions between a chatbot and patients, shared its findings recently in the medical journal The Lancet. This research took place at the Beth Israel Deaconess Medical Center (BIDMC) in Boston, as reported by the New York Times.
The chatbot, known as Articulate Medical Intelligence Explorer (AMIE), was first introduced in 2024. Developed by a team at Google, this AI system leverages large language models (LLMs) tailored for diagnostic reasoning and conversational interactions, according to a post by Google Research.
One resident physician at BIDMC, Peter Brodeur, mentioned that the chatbot was compared to a highly capable medical student or resident.
After addressing various programming issues and running detailed clinical trials at BIDMC’s primary care clinic, where physicians supervised the chatbot’s patient interactions, researchers found that around 100 patients consulted the AMIE chatbot before urgent care visits. This is rather interesting, don’t you think?
The clinicians involved in the study noted that in 75% of cases, AMIE’s summaries helped prepare for patient visits and influenced their care approach in more than half of the instances. Impressively, the differential diagnoses provided by AMIE matched the final diagnoses 90% of the time, as reported by the Times.
There were some hiccups, though. The study revealed that the chatbot did fabricate information—albeit only once throughout hundreds of conversations, which is, well, a bit concerning.
Adam Rodman, an internist and medical historian at BIDMC who was part of the trial, pointed out an interesting dynamic: “None of our patients trusted it until they realized that it was handing off to their doctor.”
In their initial 2024 report on Google Research, researchers noted that recent advancements in LLMs have demonstrated their capacity for planning, reasoning, and engaging in meaningful conversations. However, they acknowledged that good diagnostic dialogue involves nuances that are particularly relevant to the medical field—like understanding patient history and asking insightful questions without misleading anyone.
The study set up online consultations where patients could interact with AMIE in a way that resembled an objective structured clinical examination (OSCE), which is frequently used in training new clinicians and physicians for residency.
AMIE is just one of the many AI technologies increasingly integrated into healthcare. For instance, Epic Systems, the largest medical records company in the U.S., is utilizing Anthropic’s Claude to identify security vulnerabilities in hospital databases. At the same time, Arkana Labs in Arkansas has teamed up with the same AI to assist with kidney biopsies. Moreover, Utah recently approved a health AI to issue prescriptions for acne treatments.
