Docimological analysis of criterion-referenced checklists in Objective Structured Clinical Examinations (OSCE) in acute medicine: Evaluation of validity and reliability

##plugins.themes.academic_pro.article.sidebar##

Published Apr 1, 2026
https://doi.org/10.62438/tunismed.v104i02.6440
Download
PDF
Vol. 104 No. 02 (2026): Tunis Med Fev 2026

##plugins.themes.academic_pro.article.main##

Syrine Keskes
Regional hospital of Ben Arous, Medical faculty of Tunis, university of Tunis El Manar, tunis, Tunisia
Hanene Ghazali
Regional hospital of Ben Arous, Medical faculty of Tunis, university of Tunis El Manar, tunis, Tunisia
Ines Chermiti
Regional hospital of Ben Arous, Medical faculty of Tunis, university of Tunis El Manar, tunis, Tunisia
Héla Ben Turkia
Regional hospital of Ben Arous, Medical faculty of Tunis, university of Tunis El Manar, tunis, Tunisia
https://orcid.org/0000-0002-3364-3067
Naoual Ben Salah
Regional hospital of Ben Arous, Medical faculty of Tunis, university of Tunis El Manar, tunis, Tunisia
Nadia Boujelbene
Regional hospital of Ben Arous, Medical faculty of Tunis, university of Tunis El Manar, tunis, Tunisia
Sami Souissi
Regional hospital of Ben Arous, Medical faculty of Tunis, university of Tunis El Manar, tunis, Tunisia

Abstract

Introduction:The Objective Structured Clinical Examination (OSCE) is an essential tool for assessing clinical competencies. It has been used at the Faculty of Medicine of Tunis since 1998, yet few studies have analyzed its psychometric quality, particularly in emergency medicine. This study aims to evaluate the validity, reliability, and fairness of the OSCE, with a focus on the contribution of checklist-based evaluation grids.


Methods:A prospective descriptive study was conducted in 2024–2025 among 60 third-year medical students (DCEM3). The examination included 12 stations covering knowledge, skills, and professional behavior, scored using standardized checklist-based grids. Analyses focused on success rates, difficulty and discrimination indices, Cronbach’s alpha, and score distribution parameters.


Results:Sixty student scripts were analyzed. The overall pass rate was 96 %, with a mean score of 12.50 ± 1.53/20. The average difficulty index (0.58) indicated acceptable difficulty, and Cronbach’s alpha (0.85) reflected good internal consistency. The mean discrimination index was low (0.16), but stations S11 and S12 (technical and communication skills) showed satisfactory discriminative ability, unlike the cognitive stations S1, S6, and S8. Items of moderate difficulty offered the best discrimination (D ≈ 0.21). The correlation between difficulty and discrimination was negligible (r = –0.0316).


Conclusion:The OSCE in emergency medicine generally meets psychometric recommendations, showing balanced difficulty, good reliability, and standardization through checklist-based grids. Certain stations and items require revision to enhance discriminative power and improve differentiation among students.

Keywords:

Docimology, Pedagogy, Competence

##plugins.themes.academic_pro.article.details##

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

References

  1. LEclercq D, Nicaise J, Demeuse M. Docimologie critique: des difficultés de noter des copies et d’attribuer des notes aux élèves. In: Demeuse M, éditeur. Introduction aux théories et aux méthodes de la mesure en sciences psychologiques et en sciences de l’éducation. Liège: Les éditions de l’Université de Liège; 2004. p. 273-92.
  2. De Ketele JM. L’évaluation conjuguée en paradigmes. Revue française de pédagogie. 1993;103:59-80.
  3. Harden RM, Stevenson M, Downie WW, Wilson GM. Assessment of clinical competence using objective structured examination. Br Med J. 1975;1(5955):447-51.
  4. Chisnall B, Vince T, Hall S, Tribe R. Evaluation of outcomes of a formative objective structured clinical examination for second-year UK medical students. Int J Med Educ. 2015;6:76-83.
  5. Mahjoub F, Mizouri R, Amor NB, Ounaissa K, Achour W, Jamoussi H. L’évaluation clinique objective en stations multiples en nutrition: analyse docimologique. Tunis Med. 2020;98:548-54.
  6. Limaiem F, Farah F, Belkacem O, Bouraoui S, Achour W. Objective multi-station clinical evaluation in pathological anatomy: critical analysis and perception. Tunis Med. 2020;98:674-81.
  7. Martin J. Aux origines de la « science des examens » (1920-1940). Hist Educ. 2002;(94):177-99.
  8. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297-334.
  9. George S, Haque MS, Oyebode F. Standard setting: comparison of two methods. BMC Med Educ. 2006;6:46.
  10. Carricano M, Poujol F. Analyse de données avec SPSS. Pearson.Paris;2010.p.33-8
  11. Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ. 2003;37(9):830-7.
  12. Pell G, Fuller R, Homer M, Roberts T. How to measure the quality of the OSCE: a review of metrics. AMEE guide no. 49. Med Teach. 2010;32:802-11.
  13. Streiner DL. Starting at the beginning: an introduction to coefficient alpha and internal consistency. J Pers Assess. 2003;80(1):99-103.
  14. Newble D. Techniques for measuring clinical competence: objective structured clinical examinations. Med Educ. 2004;38(2):199-203.
  15. /17Patrício MF, Julião M, Fareleira F, Carneiro AV. Is the OSCE a feasible tool to assess competencies in undergraduate medical education? Med Teach. 2013;35:503-14.
  16. Nunnally JC, Bernstein IH. Psychometric theory. 3rd ed. New York: McGraw-Hill; 1994. 786 p.
  17. Ware J, Vik T. Quality assurance of medical multiple-choice questions by a simple computerized analysis. BMC Med Educ. 2009;9:40.
  18. Haladyna TM, Rodriguez MC. Developing and validating test items. New York: Routledge; 2013. 454 p.
  19. Ilgen JS, Ma IWY, Hatala R, Cook DA. A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ. 2015;49(2):161-73.
  20. Regehr G, MacRae H, Reznick RK, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med. 1998;73(9):993-7.