Table 3 Risk of bias overview
From: Methods of teaching evidence-based practice: a systematic review
SELECTION BIAS | PERFORMANCE BIAS | DETECTION BIAS | ATTRITION BIAS | ||
|---|---|---|---|---|---|
Study | Random sequence generation | Allocation concealment | Blinding of participants and personnel | Blinding of outcome assessment | Incomplete outcome data |
Badgett 2001 [23] | High Risk ‘Group assignment occurred in a quasirandomized manner. At our institution students place themselves into 16 groups of 10 to12 members. The Dean's Office combines these 16 groups into 8 larger groups in order to achieve similar distributions of gender, ethnicity, and academic performance. The Dean's office randomly assigns medical students to a sequence of rotations. A student from each group draws from a bowl a number that indicates the order of clerkship rotations’ | High Risk Allocation concealment not detailed but it appears as if it was not concealed: 'A student from each group draws from a bowl a number that indicates the order of clerkship rotations' | High Risk No blinding | High Risk No blinding | High Risk Study 1: ‘We excluded 17 incomplete questionnaires.’ Study 2: ‘Of the students with missing results, 24 were on one rotation that we could not access during the survey period. These 24 students had been randomly assigned to a sequence of rotations that placed both their Internal Medicine and Family Medicine rotations in the latter part of the year, and thus the control group contains more missing data.’ |
Bradley 2005 [24] | Low Risk ‘For each cohort, author CO created an allocation sequence using random number tables’ | Low Risk ‘Sealed in opaque envelopes, which were ordered numerically before the day of randomisation. Another investigator, PB, completed a list of students meeting the inclusion criteria on the first day of each semester. CO used this list of students and numerically ordered envelopes to sequentially allocate the students to the intervention groups on the same day. The allocation sequence for each cohort was known only to CO until allocation was completed’ | High Risk ‘Blinding of tutors and students was not possible, but outcome assessors were blinded by use of study codes and examination candidate numbers, known only to CO. The codes were broken only when data analysis was completed.’ | Low Risk ‘The study was assessed by 2 blinded outcome assessors who could allot 0–5 marks to each of the 4 sections.’ | Low Risk Many more were lost to follow-up in the directed intervention group for the attitude questionnaire (34 vs 15). Reasons for loss to follow up: Non-attendance at examination, failure to answer question in examination or answer uninterpretable or Non-response despite repeated requests, reason unknown ‘The data were analysed using an intention-to-treat analysis, where participants partially or not receiving the interventions were analysed in the group to which they had originally been randomised. Internal consistency of the knowledge scale ranged from 0.52 to 0.61 across the 3 cohorts with a Cronbach’s alpha of 0.57 for the 3 cohorts combined.’ |
Cardoso 2021 [26] | Low Risk ‘…students from three clinical nursing courses were randomly assigned to the experimental group (EBP educational program) and students from another three clinical nursing courses were randomly assigned to the control group (no intervention—education as usual) before the baseline assessment. An independent researcher performed this assignment using a random number generator from the random.org website. This assignment was performed based on a list of the 12 optional courses provided through the nursing school’s website. | Unclear Risk Not detailed | High Risk ‘Due to the nature of the intervention, it was not possible to blind participants regarding treatment assignment nor was it feasible to blind the individuals delivering treatment’ | Low Risk ‘The rater that graded the answers to the Adapted Fresno Test was blinded to treatment assignment.’ ‘Three independent experts (one psychologist with a doctoral qualification and two qualified nurses, one with a master’s degree) performed a qualitative analysis of the selected monographs. All experts had experience with the EBP approach and were blinded to treatment assignment.’ | Low Risk Two from the intervention and six from the control group did not fill in the post-test. ‘To minimize the noncompliance impact, an intention-to-treat (ITT) analysis was used to analyze participants in the groups that they were initially randomized to by using the last observation carried forward imputation method.’ |
Cheng 2012 [29] | Low Risk ‘After rotating schedules were finalised, students were randomly allocated to the above 2 groups using a table of random numbers with even and odd in Group A and B, respectively. A research assistant who was blinded to outcome analysis performed the randomisation as well as allocation of participants.’ | High Risk ‘Blinding and allocation concealment were not possible in the present study because teachers and students were all aware of the courses they were going to attend.’ | High Risk ‘Blinding and allocation concealment were not possible in the present study because teachers and students were all aware of the courses they were going to attend. However, study hypothesis had not been disclosed to all participants.’ | High Risk ‘Blinding and allocation concealment were not possible in the present study because teachers and students were all aware of the courses they were going to attend.’ | Low Risk Low attrition: ‘A total of 99 undergraduate students rotating to the wards of general medicine were recruited into the EBM teaching courses. Two students were excluded because they didn’t finish the full intervention and complete the post-test. Forty-nine students were randomly allocated to Group A and 48 to Group B. As shown in Fig. 1, 47 subjects in each group completed the post-test questionnaires.’ |
Davis 2008 [31] | Low Risk ‘Students were randomized to either computer based session or lecture using sealed envelopes prepared by the Birmingham clinical trials unit. The randomization sequence was generated by computer’ | Low Risk ‘The envelopes were coded by a third party to ensure concealment of randomization.’ | High Risk Blinding not possible | Low Risk ‘The questionnaires were marked by an examiner blind to group allocation.’ | High Risk Higher amount of missing data for intervention group (44 incomplete ineligible questionnaires compared with 6 in control) |
Eldredge 2013 [34] | High Risk ‘The chair of the block (Bear) allocated all students into problem-based learning tutorial groups using their average score on two tests in the previous block. In this way, students were distributed evenly in terms of previous test performance across eleven tutorial groups. The authors used a random number generator to assign each of the eleven tutorial groups into either an intervention or a control group; in other words, randomization was applied at the tutorial group level rather than at the individual level. Scheduling and room size constraints meant that more tutorial groups (seven) were allocated to the intervention group than the control group (four).’ | Unclear Risk Not detailed. ‘Scheduling and room size constraints meant that more tutorial groups (seven) were allocated to the intervention group than the control group (four).’ | High Risk Blinding not possible | Low Risk ‘The instructor graded all students’ assignments and provided feedback seventy-two hours after the labs using the same three rubrics with all students’ identities concealed.’ | High Risk Difference in group sizes and number lost in each group is not detailed. ‘During the fall, 80 students began the genetics and neoplasia block. EBM knowledge and skills training constituted a component of this block. Six students were absent during the EBM labs, and 3 did not take the formative test, leaving a total study size of 71.’ |
Hadvani 2020 [35] | Low Risk ‘We created a computer-generated randomization scheme, where each 2-week block of students were randomized to receive either TDS or SPM | Low Risk The investigator (S.K.) created the scheme and was unaware of student rotation assignments. The rotation schedules for students were made by individuals outside the study team from the Undergraduate Medical Education office, who were unaware of the randomization schedule.’ | High Risk ‘Due to the nature of the intervention, we were unable to blind students and TDS facilitators.’ | Low Risk ‘Blinding was utilized for assessment of the primary outcome. A single evaluator (A.D.), blinded to the group assignment and student names, graded the forms.’ | High Risk No reasons given for why post-intervention test results were not completed by participants. Whilst there was minimal drop-outs post-intervention, more than one-third of participants across both groups did not reply to follow-up |
Ilic 2012 [37] | Low Risk ‘Participants were randomly assigned independently by the Southern Health clinical site administrator by block randomization to either the intervention or control groups (Fig. 1). A computer random number generator was used to generate a randomization list in blocks of four.’ | Unclear Risk Not detailed | High Risk ‘Blinding of investigators and participants was not possible as the subject librarian and students were aware of their allocation.’ | Low Risk ‘The outcome assessor and data analyst were blinded to the allocation.’ | High Risk ‘Data were analyzed using the principle of intention-to-treat.’ However, much more attrition occurred in the control group (22 compared with 2) |
Ilic 2015 [39] | Low Risk ‘Students were randomised according to their tutorial group (i.e. cluster) by a researcher independent to the study utilising a simple cluster randomisation procedure (computerised random numbers).’ | Low Risk ‘Students were randomised according to their tutorial group (i.e. cluster) by a researcher independent to the study utilising a simple cluster randomisation procedure (computerised random numbers).‘ | High Risk ‘Due to the educational nature of the intervention, it was not possible to blind either the educators or the students.’ | Low Risk ‘Student competency in EBM was assessed by a blinded outcome assessor. The outcome assessor and data analyst were kept blinded to allocation’ | Low Risk ‘Quantitative data was analysed using the principle of intention-to-treat.’ ‘A total of 147 (30%) (45 graduate-entry and 102 undergraduate entry) students completed the Berlin Questionnaire and ACE tool (Fig. 1). The remaining 350 students declined to complete the outcome assessment.’ |
Johnson 2013 [41] | Low Risk ‘This was a simulation-based randomized trial with balanced group allocation [1:1].’ | Low Risk ‘Following informed consent, study participants drew a sealed envelope from a cardboard box. Inside the envelope was a card with a unique study number and either an “A” or “B” letter associated with group allocation; A = intervention and B = control. In this study intervention group participants (A) were provided access to research abstracts only; control group participants (B) had access to research abstracts and full-text manuscripts.’ | Low Risk ‘Participants were blinded to their group assignment, as were all co-investigators. Only the principal investigator, who was not associated with the Graduate School of Nursing, had knowledge of individual participation that linked participant name, demographic data, and their questionnaire results.’ | Unclear Risk It is not clear whether outcome assessors knew which group participants where in | Low Risk ‘All participants regardless of allocation completed the exercise and their respective questionnaire; one student did not answer the target question regarding the perceived usefulness of abstract in clinical decision making. In final analysis, this changed the number of participants from 36 to 35 and the intervention group from 20 to 19.’ |
Johnston 2009 [42] | Low Risk ‘A two-stage randomisation strategy was adopted. First, students were divided into 13 standard learning groups of approximately equal size (9–10 students each) by the Faculty of Medicine's Medical Education Unit, independent of the research team. Seven such learning groups were randomly assigned to the usual teaching intervention arm in the first half of the trial followed by the PBL intervention in the second half and six groups were randomly assigned to the PBL intervention arm followed by usual teaching intervention.’ | Low Risk ‘Assignment of students and the randomisation process were concealed from both the participants and investigators.’ | High Risk Blinding not possible (crossover design) | Unclear Risk Not detailed in the study | Unclear Risk Not fully detailed in the study (shown how many per outcome not per group) ‘KAB response rates were 97% at baseline, 88% at first assessment and 89% at second assessment.’ |
Kloda 2020 [43] | Low Risk ‘Participants were randomly allocated to the experimental group (the alternative clinical question framework) or the control group (the PICO framework) using an online random number generator after consent was obtained.’ | Low Risk ‘The allocation was concealed from the students until the instruction session during class time.’ | Low Risk ‘Both the control and experimental groups were taught by the same librarian instructor who was also one of the researchers (Boruff), for the same ninety-minute duration, and in a face-to-face setting with identical teaching methods (i.e., a combination of lecture and hands-on activities) in a computer-lab classroom setting… The difference between the two sessions amounted to three slides.’ | Low Risk ‘All participants’ identities were masked and coded so that scores for the information literacy self-efficacy instrument could be compared pre- and post-instruction.’ | Unclear Risk Study flow diagram is presented, however there is no mention of why participants dropped out, didn't complete, or didn't provide completed/partially completed outcome measures ‘Out of a possible 151 eligible OT and PT students, 103 consented to participate in the study, but several were lost due to withdrawal, drop-out, and failure to follow-up, leaving 64 with data’ |
Koufogiannakis 2005 [44] | High Risk ‘Students were assigned to groups by the course coordinator before the start of classes, with the aim to have a representative mixture of students (male/female, ethnic background, educational background) in each group. Six librarians participated, all from the John W. Scott Health Sciences Library (one group had two librarians sharing the time equivalent to one librarian—they are counted as one librarian for the purposes of this paper). The librarians were assigned randomly to one of the 18 groups.’ | Unclear Risk ‘University staff not directly involved with the study blindly drew the librarians’ names from a hat, and a second staff member drew from a separate hat the number of the group to which the librarian would be assigned.’ | High Risk Blinding not possible | Low Risk ‘WebCT allowed individual students to be matched with their scores throughout the research study. This process was managed by the Faculty’s Under-graduate Medical Education office, so that the researchers were kept at an arms length from any information that would identify individual students.’ | High Risk No information provided on participant numbers for each group or completeness of data |
Krueger 2006 [45] | Low Risk ‘The students are assigned by lottery to eight groups, with each group comprising 8 to 10 students. These groups rotate together through the various clinical clerkships The rotating clinical clerkship groups for the 1998–1999 academic year were randomly assigned to a study (EBM) or a control (non-EBM) group. Eight groups of students rotated through the obstetrics and gynecology clerkship that year.’ | Unclear Risk Not detailed | High Risk Blinding not possible | Unclear Risk ‘The examination was independently reviewed. Two reviewers with expertise in medical education who were blinded to the study-arm assignment, established face validity. They agreed that the examination assessed critical analysis and the curriculum taught the body of knowledge students would need to demonstrate expertise in critical analysis.’ | Low Risk ‘Follow-up was available for 100% of the students. All 77 students completed the clerkship and took the examination.’ |
Leung 2003 [46] | Low Risk ‘The students were randomised in two stages. Firstly, they were randomly divided into three groups of about equal size. Secondly, the groups were randomly allocated to start the clerkship in one of the three teaching blocks (groups A to C).’ | Low Risk ‘Assignment and randomisation were concealed from the students and investigators’ | High Risk Blinding not possible | Low Risk ‘Investigators had access only to aggregate results and were blinded to data at the individual level’ | Low Risk ‘Response rates were 100% throughout.’ ‘One student initially randomised to block A (pocket card first) withdrew from medical school and therefore withdrew from medical school and therefore dropped out of the study during their first rotation.’ |
Long 2016 [47] | Low Risk ‘A table of random numbers was used to allot subjects in each of the RCTs to control or intervention groups.’ | Unclear Risk Not detailed | High Risk Blinding not possible | Unclear Risk Not detailed | High Risk Sampling was purposefully oversampled to account for attrition. Variance in lost to follow-up between intervention and control groups (nutrition 3 vs 19, pharmacy 9 vs 0). Subjects with missing data for questions 1 or 2 were removed from paired analysis |
Nango 2010 [49] | Low Risk ‘Medical students were randomly assigned to either multidisciplinary groups (MultiG) that consisted of six multidisciplinary students (two medical students, two pharmacy students and two nursing students) or medical student groups (MedG) that consisted of six medical students only, in a ratio of 2:3 using computer generated random numbers (Fig. 1). All pharmacy and nursing students were allocated to a MultiG group. Randomization was stratified by gender.’ | Low Risk ‘Allocation of randomization was concealed until the randomization was completed.’ | High Risk ‘However, based on study design, it was impossible to blind the nature of the intervention’ | Unclear Risk Does not appear that outcome assessors were blinded as detail is not provided | Low Risk ‘All analyses were performed … according to an intention—to—treat principle.’ ‘One medical student assigned to MultiG and two assigned to MedG did not attend the PBL program, and were excluded from further analysis.’ |
Sanchez-Mendiola 2012 [50] | Low Risk ‘The randomization was done by the medical school with a computer generated list, using the block randomization method with blocks of two to ensure equal sample sizes’ | Unclear Risk Not detailed | High Risk Blinding not possible | Unclear Risk ‘The data analysis was blinded in an attempt to decrease bias.’ | Low Risk ‘One student from the M5 EBM group was sick on the assessment day.’ |
Schilling 2006 [51] | High Risk ‘Alternating blocks of clerks were assigned to the intervention group.’ | Unclear Risk Not detailed | High Risk Know if have a librarian or not | Unclear Risk Not detailed despite study being called a blinded study | High Risk No description of attrition within each group or how analysis accounted for ‘The final EBM case was completed by 85.5% of clerkship students in the blocks that administered the case, with comparable response rates for other outcomes.’ |
Stack 2020 [52] | Low Risk ‘The sample was randomized into the intervention group and the control (wait) group by simple randomization.’ | Unclear Risk Not detailed | High Risk ‘Participants and course instructors were un-blinded.’ | Low Risk ‘Outcome measures were scored by trained research assistants who were blinded.’ | Low Risk ‘One participant dropped out of the control (wait) group due to personal issues before data collection. Thirty students were analysed in each group on an intention to treat basis’ ‘Control group PECA results were analyzed with n = 29 as one participant invalided their PECA results by not following protocol. No participants were lost to follow up. Each participant’s data was analyzed in the group to which they were originally randomized (intention to treat principle).’ |
Widyahening 2019 [53] | Low Risk ‘The students were randomly assigned to groups of 10 or 11. For the first two group discussion sessions, each group was randomly scheduled to be tutored either by medical staff or junior doctors. During the last two group discussion sessions, the groups were crossed over (see Fig. 1). Randomization was implemented using computer software. As medical staff tutors and near-peer tutors were unequal in number, five groups did not participate in the crossover and were tutored by medical staff throughout.’ | Unclear Risk Not detailed | High Risk Blinding not possible | Unclear Risk Not detailed | High Risk ‘The analysis was based on the intention-to-treat principle’ but there was varying numbers available for different outcomes |