Comparative benchmarking of three artificial intelligence chatbots (ChatGPT-5.1, Qwen- 3 Max, and Perplexity AI) on faculty validated undergraduate respiratory physiology multiple-choice examinations

##plugins.themes.academic_pro.article.sidebar##

Published May 24, 2026
https://doi.org/10.62438/tunismed.v104i05.6620
Download
PDF
Vol. 104 No. 05 (2026): Tunis Med May 2026

##plugins.themes.academic_pro.article.main##

Khouloud Kchaou
Laboratory of Physiology, Faculty of Medicine of Tunis, University of Tunis El Manar, Tunis, Tunisia
https://orcid.org/0000-0001-7116-1336
Salma Mokaddem
Laboratory of Physiology, Faculty of Medicine of Tunis, University of Tunis El Manar, Tunis, Tunisia
Soumaya Khaldi
Laboratory of Physiology, Faculty of Medicine of Tunis, University of Tunis El Manar, Tunis, Tunisia
Yacine Ouahchi
Laboratory of Physiology, Faculty of Medicine of Tunis, University of Tunis El Manar, Tunis, Tunisia
Khadija Ayed
Laboratory of Physiology, Faculty of Medicine of Tunis, University of Tunis El Manar, Tunis, Tunisia
Rym Baati
Laboratory of Physiology, Faculty of Medicine of Tunis, University of Tunis El Manar, Tunis, Tunisia

Abstract

Background: Artificial intelligence (AI) chatbots are increasingly used by medical students for learning and examination preparation. However, their reliability in mechanistically demanding disciplines such as respiratory physiology remains insufficiently evaluated using authentic examination
material. This study aimed to assess and compare the performance of three AI chatbots on validated undergraduate respiratory physiology examination questions.
Methods: We conducted a cross-sectional analytical study including all respiratory physiology multiple-choice questions (MCQs), each comprising five propositions, used in official undergraduate examinations at the Faculty of Medicine of Tunis during the academic years 2023–2024 and 2024–2025
(101 questions). Three AI chatbots (ChatGPT-5.1, Qwen-3 Max, and Perplexity AI) were evaluated using standardized French-language prompts. Exact question-level concordance with the faculty-validated answer key, established by the teaching staff responsible for respiratory physiology examinations,
was compared using Cochran’s Q test and pairwise McNemar tests. At the proposition level, responses were analyzed as binary outcomes (true/false) to compute accuracy, sensitivity, and specificity.
Results: Question-level exact concordance differed significantly across models (Cochran’s Q = 10.18, p = 0.006). ChatGPT showed the highest concordance rate (79.2%), followed by Qwen (74.3%) and Perplexity (62.4%). At the proposition level (505 propositions), ChatGPT achieved the highest overall accuracy (94.1%) and sensitivity for true propositions (96.5%), whereas Qwen demonstrated the highest specificity for false propositions (94.2%).
Conclusion: Although all evaluated AI chatbots performed well on respiratory physiology MCQs, substantial variability was observed across models. These findings ..( abstract truncated at 250 words).

Keywords:

Machine Learning, Medical Education, Multiple-Choice Questions, Physiology

##plugins.themes.academic_pro.article.details##

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

References

  1. Vrdoljak J, Boban Z, Vilović M, Kumrić M, Božić J. A Review of Large Language Models in Medical Education, Clinical Decision Support,
  2. and Healthcare Administration. Healthcare. 2025;13(6):603. doi:10.3390/healthcare13060603
  3. Shool S, Adimi S, Saboori Amleshi R, Bitaraf E, Golpira R, Tara M. A systematic review of large language model (LLM) evaluations in
  4. clinical medicine. BMC Med Inform Decis Mak. 2025;25(1):117. doi:10.1186/s12911-025-02954-4
  5. Lu Z, Peng Y, Cohen T, Ghassemi M, Weng C, Tian S. Large language models in biomedicine and health: current research landscape and
  6. future directions. J Am Med Inform Assoc. 2024;31(9):1801-1811. doi:10.1093/jamia/ocae202
  7. Mondal H, Karri JKK, Ramasubramanian S, Mondal S, Juhi A, Gupta P. A qualitative survey on perception of medical students on the
  8. use of large language models for educational purposes. Adv Physiol Educ. 2025;49(1):27-36. doi:10.1152/advan.00088.2024
  9. Elmoselhi A. Clinically oriented laboratory illuminates challenging respiratory concepts to the pre-clinical medical students. Medical
  10. Teacher. 2012;34(3):258-258. doi:10.3109/01421591003719012
  11. Crozier T. Respiratory Physiology and Terminology. In: Lapinsky SE, Plante LA, eds. Respiratory Disease in Pregnancy. 1st ed. Cambridge
  12. University Press; 2020:1-8. doi:10.1017/9781108163705.001
  13. Cheng HM, Hoe SZ. Students’ responses under “negative pressure” to respiratory questions at the 15th physiology quiz international
  14. event: 100 medical school teams from 22 countries. BLDE Univ J Health Sci. 2017;2(2):105. doi:10.4103/bjhs.bjhs_27_17
  15. Badenhorst E, Mamede S, Abrahams A, et al. First-year medical students’ naïve beliefs about respiratory physiology. Adv Physiol
  16. Educ. 2016;40(3):342-348. doi:10.1152/advan.00193.2015
  17. Bharatha A, Ojeh N, Rabbi AMF, et al. Comparing the Performance of ChatGPT-4 and Medical Students on MCQs at Varied Levels of
  18. Bloom's Taxonomy. AMEP. 2024;15:393-400. doi:10.2147/AMEP. S457408
  19. Sadeq MA, Ghorab RMF, Ashry MH, et al. AI chatbots show promise but limitations on UK medical exam questions: a comparative
  20. performance study. Sci Rep. 2024;14(1):18859. doi:10.1038/s41598-024-68996-2
  21. Zhang R, Cai Q, Sartori A, Gayed N, Collette H. Comparing Artificial Intelligence Large Language Models in Medical Training: A
  22. Performance Analysis of ChatGPT and DeepSeek on United States Medical Licensing Examination (USMLE) Style Questions. Cureus.
  23. ;17(8):e90212. doi:10.7759/cureus.90212
  24. Uppalapati VK, Nag DS. A Comparative Analysis of AI Models in Complex Medical Decision-Making Scenarios: Evaluating ChatGPT,
  25. Claude AI, Bard, and Perplexity. Cureus. 2024;16(1):e52485. doi:10.7759/cureus.52485
  26. Agarwal M, Goswami A, Sharma P. Evaluating ChatGPT-3.5 and Claude-2 in Answering and Explaining Conceptual Medical
  27. Physiology Multiple-Choice Questions. Cureus. 2023;15(9):e46222. doi:10.7759/cureus.46222
  28. Alshammari AF, Madfa AA, Anazi BA, Alenezi YE, Alkurdi KA. Comparison of accuracy and consistency of AI Language models
  29. when answering standardised dental MCQs. BMC Med Educ. 2025;25(1):1507. doi:10.1186/s12909-025-07624-7
  30. Bhayana R, Fawzy A, Deng Y, Bleakney RR, Krishna S. Retrieval- Augmented Generation for Large Language Models in Radiology:
  31. Another Leap Forward in Board Examination Performance. Radiology. 2024;313(1):e241489. doi:10.1148/radiol.241489
  32. Masanneck L, Epping PZ, Meuth SG, Pawlitzki M. Evaluating Web Retrieval–Assisted Large Language Models With and Without
  33. Whitelisting for Evidence-Based Neurology: Comparative Study. J Med Internet Res. 2025;27:e79379. doi:10.2196/79379
  34. Kim Y, Shin S, Yoo SH. Performance of large language models in non-English medical ethics-related multiple choice questions:
  35. comparison of ChatGPT performance across versions and languages. BMC Med Ethics. 2025;26:168. doi:10.1186/s12910-
  36. -01316-z
  37. Perez-Toro PA, Dineley J, Iniesta R, et al. Exploring biases related to the use of large language models in a multilingual depression
  38. corpus. Sci Rep. 2025;15(1):36197. doi:10.1038/s41598-025- 19980-x
  39. Qin L, Chen Q, Zhou Y, et al. A survey of multilingual large language models. Patterns (N Y). 2025;6(1):101118. doi:10.1016/j.
  40. patter.2024.101118
  41. Soulage CO, Van Coppenolle F, Guebre-Egziabher F. The conversational AI “ChatGPT” outperforms medical students
  42. on a physiology university examination. Adv Physiol Educ. 2024;48(4):677-684. doi:10.1152/advan.00181.2023
  43. Chytas D, Noussios G, Salmas M, Vasiliadis AV, Troupis T. Investigation of Studies on ChatGPT’s Ability to Answer Anatomy
  44. Questions: A Self-Evaluation by ChatGPT and Comparison with an Evaluation by Gemini. Cureus. 2025;17(8):e90572. doi:10.7759/cureus.90572
  45. Blank IA. What are large language models supposed to model? Trends Cogn Sci. 2023;27(11):987-989. doi:10.1016/j.
  46. tics.2023.08.006
  47. Banerjee A, Ahmad A, Bhalla P, Goyal K. Assessing the Efficacy of ChatGPT in Solving Questions Based on the Core Concepts in
  48. Physiology. Cureus. 2023;15(8):e43314. doi:10.7759/cureus.43314
  49. Ullah E, Parwani A, Baig MM, Singh R. Challenges and barriers of using large language models (LLM) such as ChatGPT for diagnostic
  50. medicine with a focus on digital pathology – a recent scoping review. Diagn Pathol. 2024;19:43. doi:10.1186/s13000-024-01464-7
  51. Omar M, Sorin V, Collins JD, et al. Multi-model assurance analysis showing large language models are highly vulnerable to adversarial
  52. hallucination attacks during clinical decision support. Commun Med (Lond). 2025;5:330. doi:10.1038/s43856-025-01021-3
  53. Roustan D, Bastardot F. The Clinicians’ Guide to Large Language Models: A General Perspective With a Focus on Hallucinations.
  54. Interact J Med Res. 2025;14:e59823. doi:10.2196/59823
  55. Herrera-Poyatos D, Peláez-González C, Zuheros C, et al. An overview of model uncertainty and variability in LLM-based
  56. sentiment analysis: challenges, mitigation strategies, and the role of explainability. Front Artif Intell. 2025;8:1609097. doi:10.3389/
  57. frai.2025.1609097
  58. Waldo J, Boussard S. GPTs and Hallucination: Why do large language models hallucinate? Queue. 2024;22(4):19-33.
  59. doi:10.1145/3688007
  60. Sallam M, Alasfoor IM, Khalid SW, et al. Chinese generative AI models (DeepSeek and Qwen) rival ChatGPT-4 in ophthalmology
  61. queries with excellent performance in Arabic and English. Narra J. 2025;5(1):e2371. doi:10.52225/narra.v5i1.2371
  62. Kalam KA, Masoud FD, Muntaser A, Ranga R, Geng X, Goyal M. ChatGPT as a Learning Tool for Medical Students: Results From a
  63. Randomized Controlled Trial. Cureus. 17(6):e85767. doi:10.7759/ cureus.85767
  64. Gravina AG, Pellegrino R, Palladino G, Imperio G, Ventura A, Federico A. Charting new AI education in gastroenterology: Cross-sectional
  65. evaluation of ChatGPT and perplexity AI in medical residency exam. Dig Liver Dis. 2024;56(8):1304-1311. doi:10.1016/j.dld.2024.02.019
  66. Chimenea A, García-Díaz L, Antiñolo G. Exploring the Potential of Artificial Intelligence Language Models in Obstetrics with a Focus
  67. on Fetal Medicine: An Evaluation of the Perplexity AI Model. Fetal Diagn Ther. 2024;51(2):125-132. doi:10.1159/000535345
  68. Zhu S, Hu W, Yang Z, Yan J, Zhang F. Qwen-2.5 Outperforms Other Large Language Models in the Chinese National Nursing Licensing
  69. Examination: Retrospective Cross-Sectional Comparative Study. JMIR Med Inform. 2025;13:e63731. doi:10.2196/63731