Table 1 MERSQI domain and item scores for two cohorts of medical education research studies (published between 2002 and 2003 and 2013–2014)
Original Study Manuscripts (n = 210) | Replication Study Manuscripts (n = 482) | ||||||||
|---|---|---|---|---|---|---|---|---|---|
Domain | Item | Item Score | Maximum Domain Score | Domain Score mean (SD) | Studies % (N) | Domain Score mean (SD) | Studies % (N) | P valueb | |
Study Design | 3 | 1.28 (0.47) | 1.41 (0.66) | 0.01b | |||||
1. Study Design | |||||||||
Single group cross-sectional or single group post-test only | 1 | 66.7 (140) | 64.1 (309) | ||||||
Single group pre and post-test | 1.5 | 15.7 (33) | 11.4 (55) | ||||||
Non-randomized, 2 group | 2 | 14.8 (31) | 13.5 (65) | ||||||
Randomized controlled experiment | 3 | 2.9 (6) | 11.0 (53) | ||||||
Sampling | 3 | 1.90 (0.65) | 1.92 (0.61) | 0.71 | |||||
2. Institutions | |||||||||
Single institution | 0.5 | 64.3 (135) | 62.2 (300) | ||||||
Two institutions | 1 | 3.8 (8) | 3.5 (17) | ||||||
More than 2 institutions | 1.5 | 31.9 (67) | 34.2 (165) | ||||||
3. Response Rate | |||||||||
Not applicable | 14.3 (30) | 14.2 (68) | |||||||
Response rate < 50% or not reported | 0.5 | 33.3 (60/180) a | 34.7 (143/414)a | ||||||
Response rate 50–74% | 1 | 21.7 (39/180) a | 21.4 (88/414)a | ||||||
Response rate ≥ 75% | 1.5 | 45.0 (81/180) a | 44.0 (181/414)a | ||||||
Type of Data | 3 | 1.91 (0.99) | 2.09 (.99) | 0.03b | |||||
4. Type of Data | |||||||||
Assessment by study subject | 1 | 54.3 (114) | 45.6 (220) | ||||||
Objective measurement | 3 | 45.7 (96) | 54.4 (262) | ||||||
Validity of Evaluation Instruments’ Scores | 3 | 0.69 (0.93) | 1.06 (1.07) | < 0.001b | |||||
5. Internal Structure | |||||||||
Not applicable | 11.9 (25) | 5.4 (26) | |||||||
Not reported | 0 | 74.6 (138/185) a | 71.1 (324/456)a | ||||||
Reported | 1 | 25.4 (47/185) a | 28.9 (132/456)a | ||||||
6. Content | |||||||||
Not applicable | 11.9 (25) | 5.4 (26) | |||||||
Not reported | 0 | 71.4 (132/185)a | 63.4 (289/456)a | ||||||
Reported | 1 | 28.6 (53/185)a | 36.6 (167/456)a | ||||||
7. Relationships to other variables | |||||||||
Not applicable | 11.9 (25) | 5.4 (26) | |||||||
Not reported | 0 | 84.9 (157/185)a | 59.4 (271/456)a | ||||||
Reported | 1 | 15.1 (28/185)a | 40.6 (185/456)a | ||||||
Data Analysis | 3 | 2.58 (0.65) | 2.64 (0.57) | 0.22 | |||||
8. Appropriateness of analysis | |||||||||
Data analysis inappropriate for study design or type of data | 0 | 13.8 (29) | 6.6 (32) | ||||||
Data analysis appropriate for study design and type of data | 1 | 86.2 (181) | 93.4 (450) | ||||||
9. Sophistication of analysis | |||||||||
Descriptive analysis only | 1 | 27.6 (58) | 29.9 (144) | ||||||
Beyond descriptive analysis | 2 | 72.4 (152) | 70.1 (338) | ||||||
Outcome | 3 | 1.44 (0.50) | 1.56 (0.61) | 0.01b | |||||
10. Outcome | |||||||||
Satisfaction, attitudes, perceptions, opinions, general facts | 1 | 48.6 (102) | 41.3 (199) | ||||||
Knowledge, skills | 1.5 | 19.5 (41) | 24.1 (116) | ||||||
Behaviors | 2 | 29.5 (62) | 25.5 (123) | ||||||
Patient/health care outcome | 3 | 2.4 (5) | 9.1 (44) | ||||||
Total | 18 | 9.95 (2.34) | 10.71 (2.62) | < 0.001b | |||||