Could AI Help Doctors Diagnose OCD Faster?

August 20, 2024 – by Rebecca Handler

In recent years, large language models (LLMs), the AI systems behind popular chatbots like ChatGPT, have been making waves in healthcare. They can provide medical information, help with research, and even assist in decision-making. But can these AI tools really diagnose mental health conditions, like obsessive-compulsive disorder (OCD), with the accuracy of trained professionals? A new study from Stanford’s Center for Digital Health (CDH) suggests they might do even better.

The Challenge of Diagnosing OCD

Obsessive-compulsive disorder (OCD) is a mental health condition that affects about 1 in 40 adults in the United States. People with OCD experience intrusive thoughts (obsessions) and feel compelled to perform repetitive actions (compulsions) to manage their anxiety. These behaviors can severely disrupt daily life. 

However, diagnosing OCD isn’t always straightforward. On average, it takes 17 years from the onset of symptoms for someone with OCD to receive the correct diagnosis and start treatment. This delay can worsen the long-term outcomes for patients, leading to more severe symptoms and a harder road to recovery.

A New Approach: Testing AI's Diagnostic Skills

To explore whether AI could help reduce this delay, a team of researchers at Stanford – including Jiyeong Kim, PHD, MPH, Eleni Linos, MD, MPH, DrPH, with the Center for Digital Health and Carolyn I. Rodriguez, MD, PhD in the Department of Psychiatry and Behavioral Sciences – set out to test the diagnostic abilities of LLMs for OCD. They compared the performance of three different LLMs against that of primary care physicians, licensed mental health professionals, psychology doctoral trainees, and even clergy members who often provide mental health support in areas with limited healthcare resources. 

The researchers used "vignettes," i.e. short descriptions of fictional patient cases that reflect real-world OCD symptoms. These vignettes included various types of OCD, such as contamination fears, obsessive thoughts about harm, and a need for symmetry. 

They then fed the chatbots these vignettes, assessing the performance of three different AI chatbots – ChatGPT-4, Gemini Pro, and Llama 3 – to reduce any potential biases that may come from a specific LLM.  The chatbots were asked to provide the three most likely medical diagnoses, rank their choice in order of likelihood, and offer clinical reasoning behind their diagnoses. 

The goal was to see if the LLMs could correctly identify OCD in these scenarios, just as a healthcare professional would.

Surprising Results

The findings were striking. Not only did the LLMs match the professionals in diagnosing OCD, but in many cases, they outperformed them. One of the models, ChatGPT-4, correctly identified OCD in every vignette it was given, resulting in a 100% accuracy rate. This performance was notably higher than that of the human professionals in the study. For example, psychology doctoral trainees – the most accurate human group – correctly diagnosed OCD 81.5% of the time, while primary care physicians correctly diagnosed it 49.5% of the time.

This exceptional performance by the LLMs suggests that AI tools could play a significant role in improving the accuracy and speed of OCD diagnosis. The research team was particularly impressed by the LLMs' ability to provide detailed reasoning behind their diagnoses, a key factor in clinical decision-making.

The Future of AI in Mental Health Care

While the results are promising, the researchers are cautious about the immediate implementation of LLMs in clinical practice. “As a first step, we used vignettes, simplified versions of real-life cases. However, to truly understand how effective these AI tools could be, more research is needed using actual patient data from electronic health records,” explains Kim. 

“This will help identify any potential issues, like false positives or negatives, which could arise when applying AI in real-world settings,” Rodriguez adds.

Moreover, there are ethical considerations to address, and challenging questions to answer. For instance, how should AI be integrated into the workflow of healthcare professionals? How can we ensure that these tools are used responsibly, without over-relying on them at the expense of human judgment?