Chatbots for conducting systematic reviews in pediatric dentistry

Rata Rokhshad; Fateme Doost Mohammad; Mahsa Nomani; Hossein Mohammad-Rahimi; Falk Schwendicke

doi:10.1016/j.jdent.2025.105733

Chatbots for conducting systematic reviews in pediatric dentistry

J Dent. 2025 Apr 5:158:105733. doi: 10.1016/j.jdent.2025.105733. Online ahead of print.

Authors

Rata Rokhshad¹, Fateme Doost Mohammad², Mahsa Nomani³, Hossein Mohammad-Rahimi⁴, Falk Schwendicke⁵

Affiliations

¹ Department of Pediatric Dentistry, Loma Linda School of Dentistry, Loma Linda, United States. Electronic address: Ratarokhshad@gmail.com.
² Student Research Committee, Faculty of Dentistry, Mazandaran University of Medical Sciences (MAZUMS), Sari, Iran.
³ Faculty of Dentistry, Kerman University of Medical Sciences, Kerman, Iran.
⁴ Department of Dentistry and Oral Health, Aarhus University, Vennelyst Boulevard 9, Aarhus C, Aarhus 8000, Denmark; Clinic for Operative Dentistry and Periodontology, LMU Klinikum, Munich, Germany.
⁵ Clinic for Operative Dentistry and Periodontology, LMU Klinikum, Munich, Germany.

PMID: 40194755
DOI: 10.1016/j.jdent.2025.105733

Abstract

Objectives: The performance of chatbots for discrete steps of a systematic review (SR) on artificial intelligence (AI) in pediatric dentistry was evaluated.

Methods: Two chatbots (ChatGPT4/Gemini) and two non-expert reviewers were compared against two experts in a SR on AI in pediatric dentistry. Five tasks: (1) formulating a PICO question, (2) developing search queries for eight databases, (3) screening studies, (4) extracting data, and (5) assessing the risk of bias (RoB) were assessed. Chatbots and non-experts received identical prompts, with experts providing the reference standard. Performance was measured using accuracy, precision, sensitivity, specificity, and F1-score for search and screening tasks, Cohen's Kappa for risk of bias assessment, and a modified Global Quality Score (1-5) for PICO question formulation and data extraction quality. Statistical comparisons were performed using Kruskal-Wallis and Dunn's post-hoc tests.

Results: In PICO formulation, ChatGPT outperformed Gemini slightly, while non-experts scored the lowest. Experts identified 1261 records, compared to 569 (ChatGPT), 285 (Gemini), and 722 (non-experts). Screening showed chatbots having 90 % sensitivity, >60 % specificity, <25 % precision, and F1-scores <40 %, versus non-experts' 84 % sensitivity, 91 % specificity, and 39 % F1-score, respectively. For data extraction, ChatGPT yielded a (mean±standard deviation) score of 31.6 ± 12.3 (max. was 45), Gemini 29.2 ± 12.3, and non-experts 30.4 ± 11.3, respectively. For RoB, the agreement with experts was 49.4 % for ChatGPT, 51.2 % for Gemini 48.8 % for non-experts (p > 0.05).

Conclusion: Chatbots could enhance SR efficiency, particularly for the study screening and data extraction steps. Human oversight remains critical for ensuring accuracy and completeness.

Clinical significance: Chatbots can streamline SR tasks like screening and data extraction, potentially accelerating evidence synthesis, though human oversight remains needed for reliability. The applicability of chatbots for SR steps was found dependent on the specific step, indicating reviewers need to make informed choices when employing chatbots for this purpose.

Keywords: Artificial intelligence; ChatGPT; Chatbot; Large language models; Pediatric dentistry.