Predicting residents' performance: A prospective study

Abstract

Background

Objective criteria for predicting residents' performance do not exist. The purpose of this study was to test the hypothesis that global assessment by an intern selection committee (ISC) would correlate with the future performance of residents.

Methods

A prospective study of 277 residents between 1992 and 1999. Global assessment at the time of interview was compared to subsequent clinical (assessed by chief residents) and cognitive performance (assessed by the American Board of Pediatrics in-service training examination).

Results

ISC ratings correlated significantly with clinical performance at 24 and 36 months of training (r = 0.58, P < .001; and r = 0.60, P < .001 respectively). ISC ratings also correlated significantly with in-service exam scores in the 1^st, 2^nd, and 3^rd years of training (r = 0.35, P = .0016; r = 0.39, P = 0.0003; r = 0.50, P = 0.005 respectively).

Conclusions

Global assessment by an ISC predicted residents' clinical and cognitive performances.

Background

Intern selection committees exist to select appropriate medical students from the universe of students willing to enter that training program. This is often a challenging task. Members of intern selection committees devote huge resources toward achieving the dual goals of selecting applicants who hold future promise and predicting the future performance of applicants [1, 2]. There have been several attempts in the past to develop objective and valid criteria for predicting the performance of medical students as residents [2–4]. Many studies have found no relationship between objective measures of medical school performance and subsequent performance during residency [5–8]. However, other studies have demonstrated a modest correlation between medical school performance and residency performance [9–15].

Many training programs, including ours, rely on global subjective assessments made by experienced medical educators during the intern selection process. However, we found very little published data to support this practice. Thus, this study was designed to test the hypothesis that global assessment by an intern selection committee would correlate with the future performance of residents. Specifically, we aimed to answer the following question: Does the global assessment of an intern selection committee correlate with the clinical and cognitive performance of a resident?

Methods

Subjects and setting

We studied prospectively 277 residency applicants who subsequently matched to our pediatric training program between 1992 and 1997. The study was based at the Children's Hospital at Montefiore/Albert Einstein College of Medicine, Bronx, New York. All applicants to the residency program were rated by an Intern Selection Committee (ISC), on a numerical scoring system ranging from 1 (best) to 4 (worst). Applicants judged to be "outstanding, extremely strong, superior" were assigned a score of "1". Those considered to be "strong, excellent" received a score of "2", and those judged as "acceptable, competent" received a score of "3". The full ISC committee determined the final score for each applicant. The ISC was composed of a twenty-member panel with an average of 10 years experience (range 7 to 25 years) in intern selection. The same 20 members were on the selection committee during the duration of the study. Factors that were considered by the ISC included performance during the interview, dean's letter, narrative comments from clinical rotations, clinical grades, and letters of recommendation. The process of assigning scores was two-staged. First, a member of the ISC reviewed the applicant's file and assigned a preliminary score. Then, all applicants were reviewed by the full committee to determine a final ISC score. During the study period, the full committee upheld the initial ISC score 99.3% of the time. The ISC rating was assigned at the end of the interview process, prior to the submission of rank lists to the National Resident Matching Program.

Evaluation of clinical and cognitive performance

Clinical performance was assessed at the end of 12, 24, and 36 months of training for each resident. Four chief residents, at the Post-Graduate Year 4 level, were asked to rate the clinical performance of each resident using the same scoring scale as the ISC. In our program, the chief residents provide the most direct clinical supervision of all residents and are intimately familiar with the residents' clinical work. The chief residents were blinded to the ISC scores and to residents' scores on the in-service training examination. Chief residents were also blinded to each other's global assessments.

Cognitive performance was assessed by absolute test scores on the American Board of Pediatrics (ABP) in-service training examination administered at the beginning of the 1^st, 2^nd, and 3^rd years of training.

Statistical methods

Agreement among chief resident raters was assessed with the multi-rater hierarchical kappa-type statistics for categorical variables [16]. Linear regression analysis was used to compare ISC ratings with subsequent mean clinical performance ratings and standardized examination scores. Differences in proportions were tested by chi-square statistics. Significance was set at p < 0.01.

Results

There was significant agreement among the four chief residents in the clinical ratings assigned to the 227 residents (kappa = .75, P < .0001).

After 24 months of training, there was a significant correlation between the ISC rating and clinical performance rating (Table 1). This association was maintained at 36 months of observation (Figure 1). Applicants with the highest ISC scores (corresponding to poorer rankings) were more likely to perform poorly clinically, as illustrated by a greater degree of clustering of scores in that range (Figure 1).

Figure 1

Linear Regression Scattergram for Relationship between Intern Selection Committee (ISC) Scores and Global Clinical Scores at 36 months for Individual Residents. Y axis = ISC score X axis = Clinical global assessment score

Table 1 Association between ISC rating and mean clinical performance rating.

There was a significant correlation between the ISC rating and cognitive performance on the American Board of Pediatrics in-service training examinations (Table 2). The level of this correlation became stronger as residents advanced through the training program. The relationship between ISC rating and cognitive performance on the American Board of Pediatrics in-service training examinations is also displayed in Figure 2.

Figure 2

Linear Regression Scattergram for Relationship between Intern Selection Committee (ISC) Scores and Board Scores for Individual Residents. Y axis = ISC score X axis = American Board of Pediatrics Certifying Examination score

Table 2 Association between ISC rating and absolute scores on the American Board of Pediatrics examinations.

Discussion

In this study, the combined experience of a group of faculty had modest but significant validity in predicting resident's clinical and cognitive performances. Overall, the committee was better at identifying those applicants who were less likely to perform well clinically. The observed degrees of correlation between ISC ratings and future clinical performance compare quite favorably with those reported for objective measures of medical school performance and even exceed those reported for objective criteria such as National Board of Medical Examination scores, medical school grade point average, computerized academic class ranking and Alpha Omega Alpha membership [9, 11, 17, 18].

Other studies have found that performance in medical school can be predictive of performance in different specialties [19–21]. Hojat et al examined the issue of predicting performance during residency and found that associations varied by specialty and at different levels of performance for different measures [22]. Brown and colleagues found that most residents who did poorly had problems with attitude and motivation, as opposed to knowledge and skills [23]. However, the authors found little evidence in the medical school record that might have predicted such sub-optimal performance during residency [23].

Noteworthy in our study was the consistent increase in the strength of the relationship between ISC ratings and performance as residents advanced during residency. This higher level of correlation between ISC ratings and clinical ratings as the residents progressed through their training may be related to the fact that the roles of interns and residents are quite different in our training program. While residents are expected to synthesize clinical data and problem-solve around patient care issues, interns are mostly required to be compulsive about executing the management plans on their patients. It is possible, therefore, that the potential expressed in the ISC score would not become manifest until applicants assume the resident role in our training program.

While it is possible that these findings represent a "self-fulfilling prophecy", we do not believe that this is the case. First, the ISC interviewed and rated over 1,500 applicants during the study period. Next, all ratings are deleted from applicant files once the resident matching process is completed. Also, in subsequent discussions during resident performance reviews, the members of the ISC were often unable to consistently recall the ratings assigned to specific residents.

Conclusions

This study was designed to test the hypothesis that the subjective global assessment of an intern selection committee was a valid predictor of an applicant's clinical and cognitive performance during residency. We found that there was a significant relationship between the assessment of applicants at the time of selection and their subsequent performance as residents, both clinically and academically. The specific components of faculty "experience" that contributed to the determination of the ISC global subjective rating are currently under investigation. However, our findings suggest that the opinions of experienced faculty should be regarded as a valuable asset deserving of consideration on the list of criteria for residency applicant selection.