Emotion recognition in doctor-patient interactions from real-world clinical video database: Initial development of artificial empathy

Background and objective: The promising use of artificial intelligence (AI) to emulate human empathy may help a physician engage with a more empathic doctor-patient relationship. This study demonstrates the application of artificial empathy based on facial emotion recognition to evaluate doctor-patient relationships in clinical practice. Methods: A prospective study used recorded video data of doctor-patient clinical encounters in dermatology outpatient clinics, Taipei Municipal Wanfang Hospital, and Taipei Medical University Hospital collected from March to December 2019. Two cameras recorded the facial expressions of four doctors and 348 adult patients during regular clinical practice. Facial emotion recognition was used to analyze the basic emotions of doctors and patients with a temporal resolution of 1 second. In addition, a physician-patient satisfaction questionnaire was administered after each clinical session, and two standard patients gave impartial feedback to avoid bias. Results: Data from 326 clinical session videos showed that (1) Doctors expressed more emotions than patients (t 326 > = 2.998, p < = 0.003), including anger, happiness, disgust, and sadness; the only emotion that patients showed more than doctors was surprise (t 326 = -4.428, p < .001) (p < .001). (2) Patients felt happier during the latter half of the session (t 326 = -2.860, p = .005), indicating a good doctor-patient relationship. Conclusions: Artificial empathy can offer objective observations on how doctors' and patients' emotions change. With the ability to detect emotions in 3/4 view and profile images, artificial empathy could be an accessible evaluation tool to study doctor-patient relationships in practical clinical settings. © 2023