AI alone won’t make you a better doctor, finds study

American researchers have found that integrating large language models (LLMs) into medical practice does not significantly enhance physicians' diagnostic reasoning compared to conventional resources.


The study, conducted by a team from Stanford University and other institutions and published in JAMA Network Open, lays bare the complexities of incorporating artificial intelligence (AI) into clinical settings, suggesting that access to advanced technology alone is insufficient to improve medical outcomes.

Overview of the study

The trial involved 50 physicians trained in family medicine, internal medicine, and emergency medicine. Participants were divided into two groups: One had access to an LLM, specifically ChatGPT Plus (GPT-4), alongside traditional diagnostic resources, while the other relied solely on conventional tools. They were tasked with reviewing clinical snapshots and making diagnostic decisions within a 60-minute timeframe.

The primary outcome measured was diagnostic performance, evaluated through a tool that assessed differential diagnosis accuracy, the appropriateness of supporting and opposing factors, and the next steps in diagnostic evaluation. Secondary outcomes included the time spent on each case and the accuracy of final diagnoses.

Key findings

The results were surprising. The median diagnostic reasoning score for the LLM group was 76%, while the conventional resources group scored 74%. This difference of only 2 percentage points was not statistically significant, indicating that the LLM did not provide a meaningful advantage in diagnostic reasoning. Furthermore, the time spent per case was slightly lower for the LLM group, but again, the difference was not significant.



Interestingly, when the LLM was evaluated independently, it outperformed both physician groups, achieving a median score of 92%. This finding raises important questions about the role of AI in medical decision-making and the potential for LLMs to enhance diagnostic processes when used effectively.

Limitations of AI in medicine

The study brings to the fore an important point — while LLMs can process vast amounts of information and generate human-like responses, they do not replace the nuanced understanding and clinical judgment that experienced physicians bring to patient care. The researchers noted that simply providing access to AI tools does not guarantee improved performance; effective integration requires training and a comprehensive understanding of how to leverage these technologies.

The trial's authors said there was a need for further development in human-computer interactions to maximise the potential of AI in clinical settings. They suggested that training clinicians in effective prompting techniques could enhance their interactions with LLMs, ultimately leading to better diagnostic outcomes.

Human expertise

Diagnostic errors remain a significant challenge in healthcare, contributing to patient harm and increased healthcare costs. The study highlights that improving diagnostic performance requires a multifaceted approach, combining advanced technology and human expertise. While AI can assist in gathering and analysing data, the interpretation and final clinical decisions must rely on the physician's judgement.

The trial's findings align with previous research indicating that AI can augment, but not replace, human decision-making in medicine.

Implications for medical education

The implications of this study extend beyond clinical practice to medical education. As LLMs and other AI tools become more prevalent, medical training programmes must adapt to incorporate these technologies into their curricula. Educators should focus on teaching future physicians how to integrate AI into their diagnostic processes, maintaining a strong foundation in clinical reasoning.

Moreover, the study advocates for a shift in how medical professionals approach technology in their practice. Rather than viewing AI as a standalone solution, physicians should see it as a complementary tool that enhances their capabilities. This perspective fosters a collaborative approach between human expertise and machine intelligence, ultimately aiming to improve patient care.

In conclusion

AI alone won’t make you a better doctor. Integrating AI into clinical practice must be accompanied by rigorous training, effective human-computer interaction strategies, and a commitment to maintaining standards of patient care. The study stated that by fostering a relationship between AI and human expertise, the medical community can work towards a future where technology enhances, rather than diminishes, the art of medicine.