20 May 2023 • For answering online medical questions, AI outperformed humans in both facts and empathy. A recent study published in JAMA Internal Medicine compared physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. Researchers selected about 200 anonymous medical questions posted by members of the public in 2022 to a Reddit forum, r/AskDocs. opens in new tab, which were answered online by anonymous volunteer physicians with verified credentials. The same questions were then posed verbatim to ChatGPT. A team of physicians compared the responses in a blinded fashion.

WHAT DID THE STUDY FIND?

• Overall, the evaluators preferred the chatbot responses almost 80% of the time.
• The chatbot's responses were longer (mean, 180 vs. 52 words) and ranked higher in quality of information (78% of chatbot responses were categorized as “good” or “very good” as opposed to 22% of physicians' responses)
• Chatbot responses also were far kinder (45% of the bot's responses were “empathetic” or “very empathetic,” as opposed to 5% of physicians' responses).
• A table comparing a few sample answers confirms that physicians tended to be curt and often dismissive, whereas bots were pleasant and concerned.

In this cross-sectional study, a chatbot generated quality and empathetic responses to patient questions posed in an online forum. Further exploration of this technology is warranted in clinical settings, such as using chatbot to draft responses that physicians could then edit. Randomized trials could assess further if using AI assistants might improve responses, lower clinician burnout, and improve patient outcomes.

Source: JAMA Internal Medicine | Read full story