Do USMLE Steps and ITE Results Predict Performance in the American Board of Internal Medicine Certifying Exam?

Objective To evaluate if United States Medical Licensing Examination (USMLE) Step 1, USMLE Step 2, USMLE Step 3, and residency third-year in-service training exam (ITE) results predict performance of Internal Medicine (IM) residents in American Board of Internal Medicine Certifying Exam (ABIM-CE). Methods A retrospective review of USMLE Step 1, USMLE Step 2, USMLE Step 3 scores, residency third year ITE and ABIM-CE of IM residents at Michigan State University from 2004 through 2017 was conducted. Pearson correlation coecient and two sample t-tests were used to assess the relationship between various scores and pass or fail results in ABIM-CE. Results Among 114 MD residents included in the study; 92% (n=105) passed the ABIM-CE. There was a signicant correlation of passing ABIM-CE with USMLE Step 1 (OR 1.042; 95% CI 1.004- 1.082), Step 2 (OR 1.043; 95% CI 1.004-1.085) and Step 3 (OR 1.046; 95% CI 1.004- 1.089) independently, with the combined odds of all USMLE exams being 1.044 (p=0.031). Increase in ITE percentage increased the likelihood of passing ABIM-CE (OR 1.28, 95% CI 1.160-1.419). The increase in the gap was associated with decreased chances of passing ABIM-CE (OR 0.774; 95% CI 0.670-0.893). All residents who failed ABIM-CE had Step 1 scores <220. Among 31 residents with Step 2 score <220, 20% (n=6) failed ABIM. Similarly, 9% of residents with USMLE Step 3 score less than 220 failed ABIM-CE; all residents who failed had scores less than 220. The probability curve predicted that the chance of passing ABIM- CE was around 80% with USMLE scores greater than 200 and increased to almost 100% with USMLE scores of 250 or more. Conclusion There is a strong correlation between ABIM-CE results with scores of USMLE Steps and third-year ITE.


Introduction
Competency assessment of residents in a training program is a complex process. Numerous tools are used to evaluate the six medical core competencies in order to assess the overall performance of a resident [1]. American Board of Internal Medicine (ABIM) uses various factors to test all the competencies of the physicians and their ability to deliver high-quality care. The factors include ful llment of the graduate medical education training requirements, demonstration of clinical competence inpatient care, and passing the certi cation examination in internal medicine (IM) [2]. The ABIM Certifying Exam (ABIM-CE) is administered annually and is taken by the residents just after completion of their training. The performance of the residents in ABIM-CE is important not only for the residents but also for the programs.
In fact, for a residency program to maintain its accreditation, the graduates must have an 80% pass rate in ABIM-CE from the rst time takers of the exam in the latest three-year period [3].
Various tools are used by IM residency programs to select the best candidates who will perform well in their program and maintain the 'Board-pass rate'. United States Medical Licensing Examination (USMLE) scores are commonly used by residency programs to assess the applicants for residency [4][5][6]. During the residency training, the annual in-service training exam (ITE) is one of the tools used to assess the medical knowledge of the residents. A national standardized ITE has been developed in different specialties including IM, to provide feedback to the residents and the training program [7]. By assessing the knowledge gap, ITE exams inform the residents and the training program about areas in need of focused education. Thus, performance in ITE exams is used by the training programs to evaluate the readiness and ability of their residents to take and pass ABIM-CE [7].
However, there is the paucity of research and data on the correlation of baseline mandatory licensing exam scores to the success in ABIM-CE. In the past, there have been studies done in various elds that have reported some correlation between USMLE and ITE scores [5,6,8,9]. A few other studies have also reported an association between USMLE scores and IM-ITE performances [10,11]. One study reported a modest correlation between USMLE Step 1 and ITE scores with ABIM-CE [12]. Here we analyzed whether the USMLE Step 1, USMLE Step 2, USMLE Step 3 and residency third-year ITE results correlated with passing the ABIM-CE. The number of osteopathic residents (DO) was considerably less than allopathic residents (MD) and most of them took COMLEX exams instead of USMLE Steps. Additionally, data were unavailable for a few MD residents. Thus, after excluding DO residents (n = 18) and MD residents with incomplete data (n = 9), the remaining MD residents were included in the study.

Study sample and characteristics
Scores of ITE taken in the third year of residency, the exam closest to ABIM-CE in the timeline, were used in our analysis as third-year ITE scores are supposed to re ect the readiness of the residents to take ABIM-CE.

Statistical analysis
All descriptive data are presented as mean ± SD, and statistical analyses were on the basis of scaled scores. Pearson correlation coe cient was used to assess the correlation between USMLE Step 1, USMLE Step 2, USMLE Step 3, and third-year ITE scores and the results in ABIM-CE. Additionally, two-sample ttests were also done to look at the relationship between various scores and pass or fail results in ABIM-CE. Multiple logistic regression was used to examine the relationship between the scores and the ABIM-CE results controlling for other confounding variables such as gender, medical school country, the gap between medical school and residency program. Signi cance was set at p < 0.05. Statistical tests were performed using SAS 9.4 software (SAS Institute Inc., Cary, North Carolina).

Ethics approval
We received an exemption from the approval of Michigan State University Human Research Protection Program-MSU institutional board review (IRB# x16-029e). Verbal consent was obtained from all participants. Verbal consent was obtained from all participants, and the ethics committee approved this procedure.

Results
A total of 114 MD residents were included in the study; 92% (n = 105) passed the ABIM-CE. The characteristics of the residents along with their mean scores in different exams are shown in Table 1.
Mean scores in different exams for residents who passed ABIM-CE versus those who did not are shown in Table 2.
There was no relationship between ABIM-CE performance and gender, type of degree (MD versus DO) or the country of medical school (AMG versus IMG).  (Table  3). Similarly, an increase in ITE percentage increased the likelihood of passing ABIM-CE (OR 1.28, 95% CI 1.160-1.419). The increase in the gap was associated with decreased chances of passing ABIM-CE (OR 0.774; 95% CI 0.670-0.893) ( Table 3).
The probability curve (with third-year-ITE percentage and the gap set at mean values) predicted that the chance of passing ABIM-CE was around 80% with USMLE scores greater than 200 and increased to almost 100% with USMLE scores of 250 or more ( Figure 1). A ROC (receiver-operating characteristic) curve was computed to assess the accuracy of the model's ability to predict passing the ABIM-CE. The area under the ROC curve was 0.93 ( Figure 2). This indicates the probability of our model will rank a randomly chosen "pass the board" higher than "not pass the board" is 0.93.

Discussion
Passing in ABIM-CE is important for a resident [13], and it is equally important for residency training programs since the performance of their residents since ACGME requires rst-attempt examinee pass rate of at least 80% for continued accreditation of the program. Hence, a great deal of effort is put in by the programs to select the candidates who will do well in ABIM-CE and also to train their residents to perform well in ABIM-CE.
Residency programs use USMLE scores as initial screening tools to select their potential candidates from residency applicants. USMLE comprises 3 steps -Step 1, Step 2 and Step 3, which are intended to evaluate the appropriate use of medical knowledge in patient care by the examinees. The USMLE Step 2 exam consists of two sections: Step 2 CK (Clinical Knowledge) and Step 2 CS (Clinical Skills). Among them, the USMLE Step 1 score was included in a predictive tool to assess applicants during the interview for internal medicine residency [14]. However, there is limited data to prove that USMLE scores have a strong correlation with the performance in ABIM-CE [12]. The results from our study showed that the scores in all USMLE Steps have a predictive value for passing the ABIM-CE, USMLE step 1 score being the strongest one. The chance of failing ABIM-CE is higher with USMLE Scores below 220, more so with USMLE Step 1 than USMLE Step 2 or Step 3. Kay et al. reported a modest correlation between USMLE Step 1 and ABIM-CE scores [12]. Consistent results showing a correlation between USMLE scores and the results of the certi cation examination have been reported in various specialties and subspecialties [15][16][17][18][19]. Higher USMLE Step 1, Step 2 and composite scores were also associated with better performance on Emergency Medicine boards, with Step 2 being the strongest predictor [18]. Low USMLE Step 1 score was also predictive of failing certifying exams in surgery and pediatrics [19,20].
Performance in ITE is another tool used to predict the score of a resident in the ABIM-CE. ITE scores usually improve from the rst year to the third year in IM [12]. Previous ITE scores, more than USMLE scores, have been shown to be strongly associated with subsequent ITE [21]. ITE scores are useful for the programs to recognize the residents who might need assistance or interventions from the program before taking ABIM-CE. Since third-year-ITE is the exam closest to the ABIM-CE the results in this exam can be used to gauge the readiness of a resident to take ABIM-CE. In our study, third-year-ITE percent had a positive correlation with ABIM-CE performance, which was slightly better than USMLE scores. Kay et al. also found a modest correlation between ITE and ABIM-CE scores [12]. Additionally, it was reported that residents scoring in the bottom quartile on their ITE were at increased risk of failing boards while those who scored in the top quartile of the ITE had a 100% pass rate [12]. In the past, few other studies looked at the association between ABIM-CE results and ITE scores with similar results [22][23][24][25]. Babott et al.
reported that second-year ITE scores of more than 61% predicted a 100% pass rate in ABIM-CE with 41% sensitivity and 100% speci city [7]. Brateanu and colleagues developed a nomogram to predict the ABIM-CE performance which included the ITE scores of each year and the number of overnight calls in the last 6 months of residency [15]. In fact, third-year-ITE was reported to be the most important predictor in their analysis. Although USMLE scores were showed a good correlation with ABIM-CE during univariate analysis, they did not predict the ABIM results in the nal multivariable logistic regression model. This could be because the residents have already been selected for the program based on the USMLE scores. Additionally, the ITE scores are the most proximate data to ABIM-CE and would, to some extent, have some correlation with USMLE scores. The predictive value of ITE scores with the performance in certifying exams has been reported in other specialties as well [26][27][28]19]

. A study in Emergency
Medicine showed that third-year-ITE scores were most predictive of the score in the certi cation exam [27]. Similarly, a low score in ITE at any time during residency increased the chances of failing a certi cation exam in surgery [19].
Brateanu et al. reported otherwise with a weak positive correlation between the gap and the performance in ABIM-CE [15]. Another study on IMGs by Kanna and colleagues showed that the gap between medical school and residency was not signi cant in predicting ITE scores [29]. However, our study also showed a positive correlation between the shorter gap between medical school and residency with higher performance in ABIM-CE. Thus, the residents with continued medical training without interruption from clinical practice are likely to perform better in ABIM-CE. Thus, the results regarding the gap have been mixed with no discernible explanation and further studies may be needed to nd the exact cause.
Our study has a few limitations. This was a single-center study with limited demographic information of the residents. Since there is no tool to calculate the equivalence of COMLEX to USMLE, the analysis and comparison of osteopathic residents to allopathic residents could not be carried out. Additionally, there were too few subjects to explore the impact of the various combinations of either passing or failing USMLE Step 1 or Step 2 and passing the boards.
The other important factor to consider is the implication of these results in nding the residents who might need help with ABIM-CE. The exact nature and e cacy of remediation plans, including board review courses, conferences or self-study courses, in helping the resident pass the ABIM-CE are unclear [30]. Although there have been reports of improved ABIM-CE results with a directed reading program and individual education plan for residents, literature regarding the e cacy of these strategies is limited [31,32]. Furthermore, there are many other unexplained factors determining whether a resident passes an ABIM-CE. The scores described above do not measure factors such as interpersonal skills, professionalism, or shared decision-making capabilities which are equally important for a good physician.

Conclusion
There is a strong correlation between ABIM-CE results with scores on USMLE Steps and third-year ITE. Both USMLE Step 1 and Step 2 scores >220 are independent predictors of success in ABIM-CE, Step 1 scores being the strongest predictor. Among all the mean scores, the most signi cant difference between ABIM-CE Pass vs Fail groups was seen in their score in third-year-ITE. Importantly, performance in these exams may identify the IM residents who might need more help during the residency in preparing for ABIM-CE.