Table 1 Summary of the Research Evidence of Stakeholder Views of Selection to Medicine
From: A systematic review of stakeholder views of selection methods for medical schools admission
Author /Year/ Type of article | Location / Setting | Study Design / Aim | Selection tool(s) | Stakeholder/ number / response rate(RR) | Data Collection method | Outcome variable relevant to this review | Overall findings relevant to this review | MERSQI Quality |
|---|---|---|---|---|---|---|---|---|
Adams 2009 a [52] Research paper | USA Medical Schools | Quantitative | Academic record (compared acceptability of different educational settings) | Medical school administrators N = 58, RR 49% | Postal questionnaire survey, with 3 hypothetical scenarios to rank, checkboxes and open questions. | Offer of interviews, based on applications that were identical except for the institution where the applicant received their academic qualification. | University academic record perceived as preferable to both community college courses and online courses by selector administrators, in deciding offers of interviews, citing concerns about quality and rigor. | 7 |
Agrawal et al. 2005a [53] Research paper | USA Medical Schools | Quantitative | Aptitude Test / MCAT/ Academic Record GPA / Other factors | Deans of Student Affairs N = 86, RR 59% | Postal questionnaire, based on extensive literature review & piloted. | Percentage responses | MCAT (90%), GPA (60%), lack of financial aid (48%), lack of role model (77%) seen as barriers to under-represented and minority applicants by deans | 8.5 |
Brown & Griffin 2012 a [54] Abstract | Australia 1 medical school | Quantitative | Academic record/ UMAT or GAMSAT and interviews | Mixed stakeholder group- comprising applicants medical students patients and doctors. N = 938 RR = UTD | Questionnaire | Collated views of perceived validity, familiarity, overall confidence in selection | Confidence in selection methods low for all groups, even lower in medical professionals. Cognitive ability tests least valid/ interviews most valid by this mixed stakeholder group. | 6 |
Brownell et al. 2007a [55] Research paper | Canada 1 Medical School | Quantitative | MMI | Applicants N = 277 RR 98.5% Interviewers N = 74, RR 91% | Questionnaire | Mean ratings and standard deviations of 5 point Likert scale responses (1 = strongly disagree, 5 strongly agree) | MMIs acceptable to both. Applicants free from gender (4.7) & cultural bias (4.6), adequate time (3.6) stressful (2.9). Interviewers had adequate orientation (4.4) & preparation (4.6), scoring sheet allowed differentiation (4), adequate time (4.1) MMI fair (4.3). | 8 |
Campagna-Vaillancourtet al 2013a [45] Research paper | Canada Postgraduate residency programme – Ears, Nose Throat, Head & Neck | Embedded mixed methods | MMI | Applicants N = 45, RR 100% Interviewers N = 18, RR 100% | Questionnaire survey with open ended qualitative comments | Mean ratings and standard deviations of 7 point Likert scale responses (1 = strongly disagree, 7 = strongly agree) | MMIs acceptable to both. Applicants: MMI free from gender (6.2), cultural (6.2), age bias (6.3); adequate time (6.1), preferred to TI (5.3), Interviewers: better than TI (5.3) adequate preparation (5.2), scoring allowed differentiation (4.7), adequate time (5.4) MMI fair (6). | 8.5 |
Christakis et al. 2010 [56] Research paper | Canada 1 Postgraduate residency – ophthalmology | Quantitative | Interviews day | Applicants from two years (2000, 2005). N = 26, RR 79% | Questionnaire – survey, with Likert scale responses & some overlap of questions between the two surveys. | Frequency of responses, from both years merged. | 14/14 applicants felt the interview day was valuable, 14/14 appropriate length, 14/14 adequate opportunity to express own ideas and opinions. 20/26 felt the interview day affected their opinion of the programme positively and increased their likelihood of accepting a place. | 7.5 |
Cleland et al. 2011a [46] Research paper | Scotland 5 Undergraduate Medical Schools | Mixed Methods | Aptitude Test/ UKCAT | 1st Yr Medical students Qualitative strand N = 28 Quantitative strand N = 883, RR = 88% | Focus groups & Questionnaire survey (4 items relevant to this review) | Framework analysis & Percentage responses | Medical students viewed UKCAT poorly, lacking face validity, poor predictive validity, coachable, discriminatory against less affluent applicants. Only 20% agreed that UKCAT was useful. | 9 |
Daram et al. 2014 [57] Research paper | USA 1 Medical Fellowship – Gastroenterology | Quantitative | Web based video conferencing interviews (WBVC) | Applicants N = 16, RR = 100% | Questionnaire | Percentage responses | 13 candidates (81%) felt WBVC interview met or exceeded expectations, 87% thought WBVC should be an option in fellowship interviews. 25% felt that WBVC was equivalent to or better than their traditional interview experience. | 6 |
Dennehy et al. 2013a [58] Research paper | Ireland Medical School Entry – Survey conducted in one geographical area | Quantitative | Aptitude Test- HPAT-Ireland/Academic record | General practitioners (GPs) N = 122, RR = 79% | 27 item Questionnaire survey | Percentage responses | GPs supported the use of aptitude tests in principle (69.7%) & academic record (96.7%) but 30% unhappy with reforms introducing aptitude tests. Concerns expressed re socio-economic bias of academic record (71%) & HPAT-Ireland (66%) | 7.5 |
Dhar et al. 2012a [59] Research paper | New Zealand 2 Undergraduate Medical Schools | Quantitative | Aptitude Test / UMAT | Medical students N = 1325, RR 65% | 35 Item Questionnaire survey- closed questions & 4 or 5 point Likert scales. | Binary logistic regression & percentage responses | 56% of students thought UMAT was not important for selection, 67% was not fair, 81% stressful, 54% felt it assessed non cognitive attributes not really or not at all. | 8 |
Dore et al. 2010a [60] Research paper | Canada Post graduate programmes in Obstetrics, Paediatrics, Internal Medicine | Quantitative | MMI | Applicants- N = 484 RR = UTD MMI Assessors – N = UTD, RR = UTD | Questionnaire survey | Percentage responses to satisfaction ratings | Applicants: 88% believed they could positively portray themselves & 70% felt they had adequate time. Assessors: 90% felt they had a reasonable portrayal of candidate’s abilities 74% felt MMI better than TI | 7.5 |
Dowell et al. 2012 [61] Research paper | Scotland 1 Undergraduate Medical School | Quantitative | MMI | Applicants N = 324, RR 75% Interviewers – N = 116, RR = 58% | Online Questionnaire | Percentage responses | 94% applicants thought MMIs fair, 33% more stressful than TI, 74% preferred MMI to TI 90% of interviewers thought MMIs fair, 23% felt they needed more specific training | 7.5 |
El Says et al. 2013 [62] Research paper | Saudi Arabia 1 Medical School | Quantitative | MMI | Applicants and interviewers N=UTD, RR = UTD | UTD | UTD | MMI was acceptable to both students & faculty | 3 |
Eva et al. 2004 [63] Research paper | Canada 1 Medical School | Quantitative | MMI (voluntary MMI did not actually contribute to selection) | Applicants N = 115 RR 98% Assessors N = 40, RR = UTD | Questionnaire with Likert scale responses and free text comment boxes | Mean ratings and standard deviations of Likert scale responses between 1 (definitely not) to 7 (definitely). | Applicants & Assessors felt candidates were able to accurately portray themselves (5.64, 5.7) adequate advance instructions (5.87, 6.1) clear station instructions (5.84, 6.2) respectively. Free text: applicants wanted more time and interviewers wanted more training | 7 |
Eva et al. 2004 [64] Research paper | Canada 1 medical School | Quantitative | MMI (voluntary MMI did not actually contribute to selection) | Applicants N = 54, RR UTD Interviewers N = 36 RR = UTD | Questionnaire - 8 items Likert scale responses and free text comment boxes | Mean ratings and standard deviations on 7 point Likert scale (1 = definitely not, 7 = definitely) | Applicants and assessors felt that candidates could accurately portray themselves (5.26, 5.23), adequate advance instructions (5.84, 5.54) clear station instructions (5.86, 5.53) respectively. Applicants found the MMI difficult (4.05). | 7 |
Eva & Macala 2014 [65] Research paper | Canada 1 medical School | Quantitative | MMIs-(voluntary MMI – comprising – free form stations, behaviour interview (BI) & SJT type interview | Applicants N = 41 RR = UTD Interviewers N = 48 RR = UTD | Questionnaire - 5 items for applicants and similar version for interviewers | Mean ratings, standard deviations on 7 point Likert scale (1 = definitely not, 7 = definitely, or modified slightly to match the question) | Applicants: Free form stations more anxiety (p < 0.05), difficult (p < 0.01). No difference between interviewers’ views of the 3 station types re difficulty, clear instruct-tions, difficulty for candidate, but slightly lower rating for ability to judge applicant ability on BI(p < 0.05) | 8 |
Gale et al. 2010 [66] Research paper | UK 1 Postgraduate training programme – anaesthetics | Quantitative | Selection Centre – 4 stations: Structured interview, Portfolio, Presentation, Simulation. | Applicants N = 178 RR 79% over two years 2007/ 2008 Assessors –N = 24 RR = UTD | Questionnaire - | Mean and standard deviations on 5 point Likert scale (1, poor; 3, satisfactory; 5, excellent) | All four methods were positively rated by applicants & assessors for relevance (3.6–4.7), fairness (3.9–4.4), opportunity to demonstrate ability (3.6–4.2). Both groups rated the simulation station significantly higher for relevance, opportunity to demonstrate ability (P < 0.001) | 7 |
Goulston & Oates 2009 a [47] Report | Australia 1 Medical School | Mixed methods | MMIs, Academic record/ GPA Aptitude test/ GAMSAT | Extensive consultation with applicants, medical students, Faculty, alumni, Health Services, clinical training, professional bodies & more N=UTD, RR = UTD | Focus groups, submissions, email surveys, face to face interviews, invited submissions | Medical School report –collating the submissions into 29 recommendations | Stakeholders confirmed a commitment to widening diversity; approved MMIs as the interview tool; recommended including community interviewers; affirmed final selection ranking based on GPA 25%, GAMSAT 25%, MMI 50% | 4 |
Griffin et al. 2008 [67] Research paper | Australia 1 medical school Medical School | Quantitative | MMIs, Aptitude test/ UMAT | Applicants – N = 287 RR =84% | Questionnaire survey | Perceptions of the usefulness of coaching, previous interviews experience and practice run on MMI and UMAT performance | Just over half (51.4%) had accessed coaching. Those who had attended coaching rated it more helpful than those who had not (P = 0.001). A MMI practice run was considered most effective way to prepare for MMIs compared with coaching or other interview experience. | 8.5 |
Gula et al. 2014 [68] Abstract | Canada 1 medical school | Quantitative | Standardised interviews | Applicants N=UTD RR = UTD Interviewers N=UTD RR = UTD | Views on atmosphere of interviews, confidence in interviews | Standardised interviews positively received by applicants and interviewers | 4.5 | |
Harris & Owen 2007a [69] Research paper | Australia – 1 medical school | Mixed methods | Non-cognitive characteristics MMIs | Medical students, early graduates, health academics, clinical health workers & administrators N = 105, RR UTD Also surveyed applicants, post MMI. N=UTD/RR = UTD | Using q method-ology stakeholders ranked non-cognitive characteristics. No details of survey | Ranking of statements Applicant feedback simply summarised – no details provided | 6 factors emerged & used to develop the MMI: Love of medicine and learning, groundedness, self-confidence, balanced approach, mature social skills and realism. Applicant feedback overall positive. | 7 |
Henry 2006 a [70] Research paper | USA 1 premedical preparatory programme | Quantitative | Aptitude Test/ MCAT Academic record/ GPA | N = 97 premedical students | Modified version of Perceived Educational and Career Barriers Inventory | Mean responses to 31 items –Likert-type response scale consisting of strongly disagree (1) - strongly agree (5). | Barriers- Not having a high enough GPA (22% mean 2.38) and MCAT (38% mean 2.94) were seen as the most significant barriers. Letter of recommendation was not seen as high a barrier (78% did not see it as barrier, mean 1.78). | 8 |
Hofmeister et al. 2008a [48] Research paper | Canada Residency Programme- Family Medicine at 2 Medical Schools | Embedded mixed methods | MMI | Applicants N = 69,RR = 97% MMI Interviewers N = 31, RR = 94% | Survey with quantitative and qualitative components | Analysis of Likert scale responses (5 point Likert scale 1 strongly disagree, 5 strongly agree) and qualitative data content analysis | Applicants: Preferred MMIs over other interviews (4.6), free from culture (4.6)/gender (4.8) bias. Interviewers: well prepared (4.1) fairness (3.9)/ ability to differentiate (3.6). MMIs helpful assessing professionalism. Needed more time to calibrate. | 8.5 |
Hopson et al. 2014a [71] Research paper | USA 3 Emergency medicine training sites | Quantitative | MMI | Emergency Medicine (EM) interns N = 71 RR =98.6%. | Pre and post experience surveys | Mean Likert responses using five point scale | MMI as part of an interview process would negatively influence their decision to accept offer of interview mean 2.7 pre and 2.8 post MMI. Preference for combined approach of mixed MMI and TI. MMI score did not correlate significantly with preference for MMI. MMI was viewed as an accurate assessment of communication skills (3.3), problem solving skills (3.3) | 10 |
Humphrey et al. 2008a [72] Research paper | UK 1 post graduate deanery Paediatric training programme | Quantitative | MMI | Applicants N = 72, RR 75%. Interviewers N = 15, RR 100%. | Questionnaires- Cronbach alpha for applicant and interviewer survey 0.88 &.62 respectively | Means, standard deviations Likert scale responses between 1 (strongly disagree) to 6 (strongly agree) and free text comments | Applicants: Fairness 4.3, organised well 5.1, understandable questions 4.8, adequate info 4.4, fairer than traditional interview 4, preferable to traditional interview 3.7, (IMGs preferred MMI significantly more) Wanted more information (n = 8), more time (n = 3). Gender, age or previous MMI experience did not impact opinion. Interviewers- MMI better than TI 4.8, fair 4.4, needed more stations 4.3, selects best candidates 4, tests appropriate range of competencies 3.6, performance at interview predicts future performance 3.2 | 9.5 |
Husbands et al. 2014 a [73] Abstract | UK Undergraduate Medical school | Quantitative | SJT | Medical School applicants N = 200 RR = 36.2% | UTD | Applicants perceptions of relevance and validity of SJT summarised – no details | Most applicants (no details) agreed that SJT appeared relevant and valid | 3.5 |
Jauhar et al. 2008a [74] Rsearch paper | Scotland National survey | Quantitative and open ended comments | Shortlisting / Traditional interviews (TI) | Doctors on Psychiatry training programme N = 123, RR =61.5% | Questionnaire - both open- and closed-ended questions, using a Likert scales | Percentage responses to Likert scale questions | 76% lack of confidence in shortlisting process with no significant difference between successful / unsuccessful candidates. 45% thought interviews were invalid. Poor communication & inadequate feedback were problematic. 92% felt references should be available at interviews 63% favour structured references. | 8.5 |
Jayasuriya et al. 2012 a [37] Abstract | UK 1 medical school | Qualitative | Not specified | Medical Students (N=UTD) | Focus groups (N=UTD) | Students perceptions | Students were aware of the components of selection but unsure how they were used. Inconsistency in student advice. Preferred non-academic interviews that used personal statements and communication scenarios | NR |
Johnson & Elam 2001a [75] Short research report | USA 1 Medical School | Quantitative | Letters of recommendation (LOR) | Admission committee members N = 14, RR 93% Premedical advisors N = 42 RR = 87.5%. | Using example letters of recommendation, rated usefulness on 5 point Likert scales from “Not at all” to “Extremely”. | Perceptions of usefulness | There was no difference between the two groups in their perceptions of usefulness and global impression of the sample letters. Both thought most helpful when they factual, descriptive and cited examples of specific behaviours. | 8 |
Kaffenberger et al. 2014a [76] Letter – original research | USA National Survey | Quantitative | Letters of recommendation (LOR) | Professors of Dermatology - N = 129 RR = 37% | Survey- no details | Percentage responses displayed graphically | LOR from Dermatology Professors and “Physicians I know” considered more reliable than other sources. Perceived problems with LOR are frequently having difficulty in ascertaining the strength of recommendation and reluctance to give honest account of weaknesses | 6.5 |
Kelly et al. 2014 a [77] Research paper | Ireland 1 Undergraduate medical School | Quantitative | MMI (experimental, did not contribute to selection) Aptitude Test / HPAT-Ireland | First Year Medical Students N = 71 RR = 65% MMI Interviewer N = 24, RR = 49% | Electronic questionnaire survey | Percentage responses | 90% students agreed that the MMI content was relevant, 60% felt content of TIs or HPAT-Ireland (38%) were relevant, 73% felt MMI suitable for selection, 79% supported academic record. 75% of interviewers felt that MMI was relevant, reasonably tested candidates’ ability (79%). The majority (71%) thought MMI would be a useful addition to selection | 8 |
Kelly et al. 2014 ba [38] Research paper | Ireland 1 Undergraduate medical School | Qualitative | Aptitude test/ HPAT-Ireland | Qualified doctors from various disciplines (n = 15) | Interviews – analysed using principles of grounded theory | Perceptions of job relevance, acceptability of HPAT-Ireland | Sections 1 and 2 perceived to have good job relatedness, but Section 3 non-verbal reasoning, criticised. Split views on acceptability, with those opposed being principally concerned re possible negative impact on diversity. | NR |
Kleshinski et al. 2008a [78] Research paper | USA 1 medical school | Quantitative | Interviews - regarding the value of including professionalism/ethics scenarios in selection interviews | Faculty interviewers N = 91 Applicants N = 107, RR = 54% | Survey | Percentage responses to questionnaire items with five point likert scale responses to statements | Applicants: 74% asking about professionalism positively impacted & 76% agreed it enhanced their view of the medical school. 88% agreed it was important to include in selection interviews. Applicants more positive than interviewers re importance of including professionalism (88% versus 69% p = 0.0001). | 7.5 |
Koczwara et al. 2012 [79] Research paper | UK Post graduate GP training in one geographical area | Quantitative | Cognitive ability tests clinical problem-solving test (CPST), situational judgement test (SJT) | Applicants N = 249, RR 96% | Validated candidate evaluation questionnaire | Percentage and frequency of responses | Cognitive ability tests: 30% not fair-fair, 35% content not appropriate, 54% not relevant. By contrast figures from the overall 2009 GP applicant pool (n = 2947) showed that the CPST and SJT were regarded as relevant by 89, 63%, appropriate 85, 68% and fair 85 and 53% respectively. | 8 |
Kumar et al. 2009 a [39] Research paper | Australia & Canada 2 Graduate Entry Medical Schools | Qualitative | MMI | MMI Interviewers N = 37, took part in focus groups MMI Interviewers (n = 75, RR 48%)- completed a survey Applicants (n = 442; RR = 91%) completed a survey | 6 Focus groups and open-ended survey | Framework analysis | Very positively viewed. Candidates valued interviewer independence & multiple opportunities,, but felt time pressured and absence of opportunity to present their motivations. Interviewers less anxious about decision making, but concerns re measuring communication skills and lack of opportunity to bench their marking. | NR |
Kumwenda et al. 2013a [80] Research paper | UK 6 Medical schools and 1 dental school | Quantitative | Application including personal statement / UCAS | First year entrants to medical and dental school N = 432 RR = 34% | Online questionnaire | Average and percentage responses Cronbach alpha 0.77. | 66% suspect peers stretch the truth, 16% deceptive practice is common, 84% lying unacceptable, 949% exaggerating on UCAS is dishonest but 14% think part of the admission game (males agree more p < 0.05) | 9 |
Lambe et al. 2012 [81] Research paper | UK 1Medical School | Quantitative | Aptitude Test/ UKCAT | Applicants N = 787, RR = 66%, | Online questionnaire | Percentage responses | 86% thought that you can prepare for the UKCAT, 44% felt that advice on the UKCAT was confusing, 55% felt test was fair and 44% agreed it was relevant | 6.5 |
Lievens 2013 [82] Research paper | Belgium National survey Medical and dental undergraduate | Quantitative- longitudinal multiple cohort study (1999–2002) | SJT & Cognitive tests | Applicants N = UTD, RR 61.8% | Validated questionnaire | Mean, Standard deviation of responses on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). | Mean rating for the face validity of the SJT (3.19 SD 0.88) was significantly higher than cognitive test (2.76 SD 0.68) p < 0.01. SJT viewed as significantly less difficult than cognitive tests. | 8.5 |
Lievens & Sackett 2006 [83] Research paper | Belgium National survey Medical and dental undergraduate selection | Quantitative | Two formats of SJT video based versus written formats & Cognitive tests | Applicants from two cohorts (N = 638, RR 55%; N = 1078, RR 61%) | Validated questionnaire Cronbach alpha (0.66, 0.76) | Mean, Standard deviation of responses rated on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). | No significant difference between the mean face validity perceptions of the video SJT (3.41) and written SJT (3.44). Both significantly higher than face validity of the cognitive test (2000 = 2.75, 2003 = 2.79). | 10.5 |
Lubarsky &Young 2013 [84] Abstract | Canada 1 hospital Neurology residency program | Quantitative | MMI | Applicants N = 29 RR = 94% MMI interviewers no details provided | UTD | UTD | Both applicants and interviewers felt MMIS allowed applicants to showcase their unique attributes and skills, but that the process felt somewhat ‘impersonal’ | 4 |
Marrin et al. 2004a [49] Research paper | Canada 1 Medical School | Quantitative | Key qualities of medical selection process, no particular tool identified | Admission stakeholders- N = 277 comprising Faculty, students and community Mean RR across the stakeholder groups 71% | Paired comparison approach- | Z scores for probability of each characteristic being chosen from the pairing | No significant difference between stakeholders. Fairness (mean z score 0.92), Validity (mean z score 0.87), comprehensiveness (0.44), accessibility (0.1), defensible (− 0.3), leads to diversity (− 0.31), affordable (− 0.8), public statement (− 0.9). | 8.5 |
Mathers & Parry 2010a [40] Research paper | UK 3 Medical Schools | Qualitative | Not specified | Older mature students (N = 15) | Unstructured one to one interviews | Framework analysis | Decision to apply made after careful consideration of university location/ access to family support/ identity and fit were key. Inflexibility and uncertainty of process/ UCAS inflexible/ Risks involved in making the application. Financial cost | NR |
Milne et al. 2001a [85] Research short report | USA 1 Medical Residency Programme | Quantitative | Interviews | Medical Interns n = 53, RR = 87%. | Questionnaire survey | Percentage responses to categories of five point Likert scale responses from 1 (strongly disagree) to 5 (strongly agree). | Interviews viewed as a chance to learn more about programme (84%), sell myself (80%), determine faculty satisfaction with institution (76%) and their own interest in the prog (71). 86% felt interview was necessary, 93% believing no interview was unacceptable. | 8 |
Mitchison 2009a [86] Research paper | UK 1 Post graduate deanery | Quantitative | Selection centre with 3 types of station: structured interview/ case based discussion/ simulated patient | Assessors-N = 53, RR 77%. | Feedback questionnaire | Frequency & percen-tage responses. Note – feedback given per station type, with some assessors responding to more than 1 station | 19/21 assessors found the Structured interview useful, 21/24 the communication station, 27/27 case based discussion. 69% felt SC was an improvement on TI, 4% felt it was worse. Positive free text: fairness &objectivity. Negative: inflexibility to explore other issues | 7 |
Monroe et al. 2013a [87] Research paper | USA / Canada Large scale survey of 142 medical school | Quantitative | Aptitude test / MCAT Academic record / GPA Interviews Letter of recommendation | Admission Deans from all US and Canadian medical schools using MCAT N = 120 RR = 85% | Online survey − 69 items - derived from qualitative interviews in 8 medical schools | Means Standard Deviations, frequency of responses. 5 point Likert scale (1 not important – 5 extremely important) | Rating differed depending on the stage of the process- MCAT & GPA viewed most important for shortlisting to interview but less important in the decision of who to admit where interview (mean rating 4.5) & letter of recommendation (3.7) more valued (no p value given), followed by GPA (3.6), community service/volunteering (3.5), MCAT (3.4) and personal statement (3.3) | 10 |
Niyomdecha et al. 2012a [88] Abstract | Thailand 1 medical school | Quantitative | MMIs | Medical students and instructors N=UTD, RR = UTD | UTD | UTD | 88% of instructors thought MMI process was good and 100% of students thought MMI was fair | 4 |
O’Brien et al. 2011 [89] Research paper | UK 1 Undergraduate medical School | Quantitative | MMI (experimental, not used for selection) Standardised interviews (SI) | Applicants N = 47,RR = UTD Interviewers-N=UTD RR = UTD | Questionnaire survey with free text comment boxes | Means, standard deviation, of responses on a 5 point Likert scale scoring | No statistical difference between Interviewers’ ranking of SI and MMI with respect to overall opinion, fairness, accuracy and ability to pick best candidate. School leavers: MMI more accurate, less difficult than SI (p = 0.03 and 0.01). Graduate entrants: MMI more difficult than SI (p = 0.005). | 7 |
O’Flynn et al. 2013 a [90] Research paper | Ireland National survey | Quantitative | Academic Record – School leaving certificate examination Aptitude test - HPAT –Ireland | Guidance counsellors N = 187, RR = 15%. | Questionnaire – 26 items and free text comment s | Percentage responses to likert scale answers and simple content analysis of free text comments | 52% in favour of the introduction of HPAT-Ireland, 49% felt new system was fair. Those opposed were concerned re negative impact on socially disadvantaged. Majority felt non-verbal reasoning least relevant. | 7 |
Patel et al. 2011a [91] Abstract | USA Post graduate − 1 medical residency programme | Quantitative | Group Interview | Applicants N = 77 RR = 38% | Online anonymous questionnaire – 15 items | Percentage responses to survey questions | 75% liked group interviews, 62% would recommend continued usage. 89% felt group interviews effective. IMGs felt they had much harder time impressing interviewers than local candidates (p = 0.004) | 6 |
Patterson et al. 2009a [92] Research paper | USA 1 medical school with mission to recruit from and serve underserved populations | Quantitative | Aptitude test/MCAT Academic record / GPA | Applicants - American Indians and Alaskan natives N = 34, RR = 38%, from three chorts. Included and compared accepted (N = 21) and rejected (N = 13) applicants . | Questionnaire developed from research and pilot – containing numerical and open ended questions | Frequency and percentage responses, chi square and t-test, and groupings of free text comments | Access to supports was limited for both groups but rejected applicants had significantly less support (p < 0.05).MCAT viewed as a barrier by 65% & financing the application by 42% (rejected more so, p < 0.05) Free text comments identified 3 obstacles – finance including cost of MCAT preparation, lack of time to volunteer/ build a CV etc. as working & discouraging/wrong information | 8 |
Patterson et al. 2011 a [7] Research paper | UK Postgraduate training in General Practice - national survey | Quantitative | SJT Clinical problem solving test (CPST) Selection Centre (SC) comprising: Simulated patient Group exercise Written exercise | Applicants to GP training- 3 cohorts 2007–09. N = 9067, RR 56% | Online and paper questionnaire survey. | Mean, Mode Standard deviation of responses Cronbach alpha survey 0.7–.94. | Shortlisting: SJT viewed very job relevant but CPST as more so p < 0.001. Perceptions of fairness (formal test characteristic and interpersonal treatment) were good. Selection centre- All three tasks positively viewed in terms of job relevance, SP more positively p < 0.001. Perceptions of fairness high | 9.5 |
Patterson et al. 2013 [93] Abstract | UK Large scale survey | Quantitative | SJT | Candidates to Foundation Year Training Programme N=UTD, RR = UTD | UTD | Candidate reaction | Feedback from candidates indicate SJT relevant and fair | 3 |
Randall et al. 2006 [94] Research paper | UK 1postgraduate Paediatric Deanery | Quantitative | Selection centre (SC) comprising group discussion, simulated patient written exercise | Applicants N = 27, RR UTD | Questionnaire survey | Frequency of responses on 5 point Likert scale (1 strongly disagree to 5 strongly agree) | N = 24agreed that the SC content was appropriate, 24 agreed more relevant than other selection tools & provided better opportunity to demonstrate their skills | 6 |
Razack et al. 2009a [50] Research paper | Canada 1 Undergraduate medical school | Embedded mixed methods | MMI | Applicants -International and non-local. N = 82 RR = 82%. Unsuccessful applicants N = 50 RR = 60%. Interviewers N = 38 RR = 100% | Questionnaires with quantitative and qualitative components | Mean ratings and standard deviations of Likert scale responses between 1 (strongly disagree) to 6 (strongly agree). Content Analysis of free comments. | Applicants: MMI more fair than SI (p = 0.001) & more effective at evaluating non-academic aptitudes (p = 0.001) more stressful (p = 0.016). Interviewers’ mean scores: Fair 5.2, effective 5.1, appropriate for use with home and international applicants 4.9, transparent 5.2. Concern candidate difference may affect performance, misses some of benefits of TI, some practical issues. | 8 |
Rich 2011 a [95] Abstract | UK 1 medical school- with a widening access agenda | Quantitative | Traditional Interviews (TI) | Medical student admitted via a widening access (WA) route N = UTD RR = UTD Interviewers n = UTD, RR = UTD | Questionnaire | Satisfaction with TI for widening access in selection | 49% students in early clinical years felt TI should be retained. Only 25% of students in clinical years and 20% of interviewers agreed favouring multi station interviewing. | 5 |
Rodgerson et al. 2013 a [96] Abstract | UK Medical / Dental School | Quantitative with free text | MMI Traditional Interview | Medical N = 451) and dental applicants (N = 224) RR = 75% | Online survey post | Percentage responses and mean ratings of responses using five point Likert scale | 94% agreed MMI suitable for assessing potential 45% agreed TI suitable. MMI more favourable re enjoyment, stressfulness & fairness. 80% gave free text comments relating to fairness, with 40% approving of the opportunity to impress multiple interviewers. | 4.5 |
Samarasekera et al. 2014 [97] Abstract | Singapore 1 medical school | Quantitative | SJT and Focused skills assessment | Applicants and assessors N=UTD, RR = UTD | UTD | UTD | 92% of candidates happy with format. 82% assessors positive perceptions of the process which Evaluated empathy, communication, integrity, general knowledge, resilience, personality profile | 5 |
Stagg & Rosenthal 2012a [41] Research paper | Australia 1 Medical School | Qualitative | Not specified | Community Members and Members of the rural based Community Liaison N = 12 | Semi structured individual interviews | Thematic analysis | Overwhelmingly saw involvement in selection of students as positive. Opportunity for professional &personal growth; responsibility to represent the broader community; protecting the student and public interest and self-interest in shaping the future workforce. | NR |
Stevens et al. 2013a [51] Research paper | Ireland 3 medical schools | Mixed methods – embedded | Aptitude Test/ HPAT –Ireland | First Year medical students N = 291, RR = 77%, | Questionnaire survey | Percentage responses and simple content analysis of free text | Almost all support academic record as suitable tool, 78% interviews, 74% personality tests, 68% adjunct admission tests. International students more likely to support interviews, knowledge about course, references and personal statements (all p < 0.01), Of those who had sat HPAT (N = 175) – 76% felt it fair, 37% felt it was easier for males, 32% felt non-verbal reasoning section irrelevant . 54% had accessed prep course, of these 79% felt it improved performance | 8 |
Tiller et al. 2013 [98] Research paper | Australia Graduate entry medical & dental school | Quantitative | Internet based MMI (iMMI) | Applicants N = 119 RR = 41% Interviewers N = 78, RR = UTD | Online survey | Percentage responses and mean ratings to responses using five point Likert scale(5 = very satisfied, 1-very unsatisfied) | Mean satisfaction ratings with use of skype technology 4.25, overall interviews process 4.2, being interviewed online as part of overall selection process 4.10, video quality 4.09, audio quality 4.08. 68% would prefer an in-person MMI and 32% a skype interviews. 78% of Interviewers satisfied with the iMMI, 71% with the technology. Free text comments re concerns re operational & technical issues | 8 |
Turner & Nicholson 2011a [42] Research paper | UK 1 Undergraduate Medical School | Qualitative | Written application / UCAS Personal Statement/ Letter of reference | Medical school selectors –in three focus groups- clinical / non-clinical and lay members N = 17 | Focus groups and document review | Thematic framework analysis and triangulation with recorded reasons | Four themes: Work experience/ commitment to study medicine/ teacher reference/ personal statement. Most common reason for rejection was poor medically related work experience. Teacher reference viewed as influential esp. for rejection but hard to interpret. Personal statement – useful but considered highly subjective. Ideal candidate extremely difficult to judge | NR |
Uijtdehaage et al. 2011 [99] Research paper | USA 1 medical School Specialised to develop leadership & serve disadvantage | Quantitative | MMI | Applicants Cohort 1: N = 76, RR =100% Cohort 2: N = 77, RR =99% Interviewers N = 26 RR = 93% | Questionnaire surveys – 8-10 items | Mean ratings and standard deviations of Likert scale responses between 1 (definitely not) to 7 (definitely). Cohort results presented separately | Applicants: able to present abilities (5.6, 5.7), adequate instructions (6.2, 6.5), sufficient time (3.7, 4), free from gender (6.6–6.7) or cultural bias (6.3, 6.6), stressful (4.2, 3.7) . Interviewers – accurate portrayal (5.6), prepared (6), clear instructions (6.2), adequate time (4.7), allow differentiation (5.7), Overall fair (6.2) | 9 |
UKCAT Consortium 2009/2010 [100] Report | National Survey UK Medical Schools using UKCAT | Quantitative | UKCAT / Aptitude | Applicants N = 6821, RR =27% | Questionnaire | Percentage responses and majority opinions | Considered a difficult test. Unconvinced it tests right attributes. 40% felt their college or school were not well informed about UKCAT/ ¾ had used online practice materials/ books, and found them useful. 90% happy with testing environment. | 6 |
UKCAT Consortium 2011 [101] Report | National Survey UK Medical Schools using UKCAT | Quantitative | UKCAT / Aptitude | Applicants RR = 19.5% | Questionnaire | Percentage responses and majority opinions | 44% found out it from websites /prospectuses. 33% found out from their schools. 36% of candidates from independent schools rated their advice as good or very good, only 18% from comprehensive schools agreed with this. Majority very supportive of practice tests (93%) and books (90%). Timing in the test is crucial. | 6 |
Vermeulen et al. 2012 [102] Abstract | Post Graduate GP training. Nether-lands | Quantitative | Behaviour specific interviews, knowledge test, SJT and simulated consultation versus traditional interviews (T) | Applicants N = 47 RR = UTD | UTD | UTD | Both TI & behaviour specific interviews were considered job relevant & fair/ but latter offered better opportunity to show competencies. Both SJT and knowledge based test were considered job relevant. SJT considered fair (95.7%), simulated consultation (78%) knowledge based test (64%). | 4.5 |
Waheed et al. 2011a [103] Research paper | Pakistan 1 medical school | Quantitative | Interviews that include a scenario based on professionalism as part of the interview | Medical Students N = 100 Faculty members N = 100 | Students and faculty attended a lecture about profesion-alism scenarios that could be included in interviews and then discussed these and completed a survey | Frequency of responses | 77% of students highly positively influenced by the lecture compared to 10% faculty, 85% students / 76% Faculty agreed it influenced their impression of Medical school values Faculty more likely to feel important to include such scenarios in admission interviews (p = 0.01) | 7 |
Westwood et al. 2007a [104] Research paper | UK 1 Post graduate Deanery Cardiology | Quantitative | Structured Interviews | Applicants N = 94, RR = 80%. | Questionnaire | Median and interquartile range of Likert scale responses between 1 (strongly agree) to 5 (strongly disagree) | Satisfaction rating high, (2), objective (2), appropriate duration (2) offered sufficient scope to express individuality (2) and was relevant to the job (2). | 7 |
White et al. 2011a [43] Research paper | Canada 1 Medical School | Qualitative | Essay | Applicants - N = 20 | Review of 240 randomly selected essays and interviews. Qualit-ative analysis using modified grounded theory | How applicants approach writing the essay | Applicants expressed the idea that they had approached the essays as a way to “show themselves” and “tell their own story” in a subjective way which they felt was missing from other parts of the admission process. | NR |
Wilkinson & Wilkinson 2013 [105] Research paper | New Zealand 1 Medical School | Quantitative | Aptitude Test/ UMAT | Medical Students – two cohorts 2010/2011 N = 263 RR = UTD | Online survey of self -reported- forms of preparation used for UMAT and received confidence | Percentage response rates, comparisons of scores mean and standard deviation | Commonest forms of preparation were ACER practice materials, MED Entry course, and student led tutorials. Students who took a MED Entry course had significantly higher confidence (mean diff Likert score 0.6), No significate differences for those taking student led tutorials. Moderately strong positive correlation between amount of money spent and confidence r = 0.3, p < 0.001. | 10.5 |
Wright 2012 [44] PhD Thesis | UK 1 Medical School | Mixed Methods thesis, with Qualitative strand used to explore stakeholder views | UK Medical School Admission processes- personal statements, interviews | Medical Students (n = 13) | Interviews - Qualitative analysis using Framework analysis | Students’ views of influences on decision to apply to medical school and preparedness | Family & School were highly influential on decision, support for application activities such as work experience, preparing personal statements and interviews practice. Students from medical /professional backgrounds and fee paying schools were better supported/prepared. | NR |
Ziv et al. 2008 [106] Research paper | Israel 1 Medical School | Quantitative | Selection centre/ known as MOR | Applicants (two cohorts) N = 510 RR = 90.6% - MOR Raters – largely Senior Faculty members n = 352 RR = UTD | Questionnaire | Frequency of responses with four point Likert scale | 76% of applicants rated MOR as fair 76% felt they had opportunity to express their capabilities 85% (n = 299) of rater found it fair. 92% rated MOR assessment items as appropriate | 7 |