Iqbal, Usman and Lee, Leon Tsung-Ju and Rahmanti, Annisa Ristya and Celi, Leo Anthony and Li, Yu-Chuan Jack (2024) Can large language models provide secondary reliable opinion on treatment options for dermatological diseases? Journal of the American Medical Informatics Association, 31 (6). pp. 1341-1347. ISSN 10675027
![[thumbnail of 207_usman iqbal.pdf]](https://ir.lib.ugm.ac.id/style/images/fileicons/text.png)
207_usman iqbal.pdf - Published Version
Restricted to Registered users only
Available under License Creative Commons Attribution.
Download (1MB) | Request a copy
Abstract
Objective: To investigate the consistency and reliability of medication recommendations provided by ChatGPT for common dermatological conditions, highlighting the potential for ChatGPT to offer second opinions in patient treatment while also delineating possible limitations. Materials and Methods: In this mixed-methods study, we used survey questions in April 2023 for drug recommendations generated by ChatGPT with data from secondary databases, that is, Taiwan's National Health Insurance Research Database and an US medical center database, and validated by dermatologists. The methodology included preprocessing queries, executing them multiple times, and evaluating ChatGPT responses against the databases and dermatologists. The ChatGPT-generated responses were analyzed statistically in a disease-drug matrix, considering disease-medication associations (Q-value) and expert evaluation. Results: ChatGPT achieved a high 98.87 dermatologist approval rate for common dermatological medication recommendations. We evaluated its drug suggestions using the Q-value, showing that human expert validation agreement surpassed Q-value cutoff-based agreement. Varying cutoff values for disease-medication associations, a cutoff of 3 achieved 95.14 accurate prescriptions, 5 yielded 85.42, and 10 resulted in 72.92. While ChatGPT offered accurate drug advice, it occasionally included incorrect ATC codes, leading to issues like incorrect drug use and type, nonexistent codes, repeated errors, and incomplete medication codes. Conclusion: ChatGPT provides medication recommendations as a second opinion in dermatology treatment, but its reliability and comprehensiveness need refinement for greater accuracy. In the future, integrating a medical domain-specific knowledge base for training and ongoing optimization will enhance the precision of ChatGPT's results. © 2024 The Author(s). Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved.
Item Type: | Article |
---|---|
Additional Information: | Cited by: 4 |
Uncontrolled Keywords: | Databases, Factual; Dermatologic Agents; Humans; Natural Language Processing; Referral and Consultation; Reproducibility of Results; Skin Diseases; Taiwan; dermatological agent; acne vulgaris; adult; Article; artificial intelligence; atopic dermatitis; ChatGPT; contact dermatitis; controlled study; data analysis; data base; decision making; dermatologist; dermatology; diseases; drug therapy; drug use; female; herpes simplex; herpes zoster; human; ICD-10; impetigo; institutional review; internal consistency; knowledge base; major clinical study; male; medical informatics; national health insurance; psoriasis; rosacea; scabies; seborrheic dermatitis; skin disease; statistical analysis; urticaria; vitiligo; wart; factual database; natural language processing; patient referral; reproducibility; Taiwan |
Subjects: | R Medicine > RL Dermatology |
Divisions: | Faculty of Medicine, Public Health and Nursing > Non Surgical Divisions |
Depositing User: | Ngesti Gandini |
Date Deposited: | 10 Mar 2025 03:29 |
Last Modified: | 10 Mar 2025 03:29 |
URI: | https://ir.lib.ugm.ac.id/id/eprint/15672 |