Fig. 4

Example of a clinical case question associated with a radiological image. Graph A displays the students’ responses, while graph B represents the distribution of students according to the probability of answering the question correctly (y-axis) based on their ability level in the exam (x-axis). “Ability” refers to the theoretical estimation of the student’s knowledge in the exam. Graph C represents the point at which this question best discriminates among the knowledge levels of the entire sample (x-axis). Both graphs belong to the Item Response Theory (IRT) using the Two-Parameter Logistic (2-PL) probability model. This question exemplifies how errors in its design result in poor discrimination, rendering it inadequate for assessing students’ knowledge. The clinical description is insufficient and too nonspecific to effectively evaluate students’ knowledge. In this case, although it may represent a real-world scenario, better discrimination results are obtained in exams with “ideal” or “typical” cases. Attempting to assess scenarios that are not theoretical, scientific, or supported by clear scientific evidence is not cost-effective with multiple-choice - single select questions. Furthermore, the answers are lengthy, imprecise, subjective in nature, and lacking clear scientific evidence, as they may vary among different hospital protocols. Additionally, the image, while once again typical of routine medical practice in the setting of acute abdomen, does not depict a specific radiological finding but rather a common image associated with numerous pathologies that do not contribute to the diagnosis of a specific condition. With these methodological errors according to the 2PL model, graph B demonstrates how there are no differences between students with varying levels of knowledge when faced with the question