Comparative analysis of Physician Assistant/Associate National Certifying Examination (PANCE) performance trends: pre and during COVID-19 era

Abstract

Background

COVID-19 significantly impacted physician assistant/associate (PA) education programs. Most programs transitioned didactic and clinical education from in-person to remote, and clinical training opportunities diminished. Graduates of accredited PA programs take the Physician Assistant National Certifying Examination (PANCE), a five-hour exam with 300 multiple-choice questions, and must attain or exceed the scaled passing score of 350 (range: 200–800). We examined first-time examinees’ trends in PANCE scores and passing rates three years prior to the pandemic and three years during.

Methods

We analyzed data (N = 59,459) from the National Commission on Certification of Physician Assistants. The two primary outcomes were PANCE scores and pass rates. The main exposure was the timeframe: three years pre-pandemic (2017–2019) and three years during the pandemic (2020–2022). The 2017–2018 scores were equated to the new passing standard implemented in 2019. Covariates included age, gender, years the PA program has been accredited, program region, and rural-urban setting. Analyses consisted of descriptive, bivariate, and multivariate statistics.

Results

The mean PANCE score and pass rate during the six-year study period were 463 and 93%, respectively. In unadjusted analyses comparing each year individually, mean PANCE score was highest in 2020 and lowest in 2022 than in all other years except for 2017. When comparing each pandemic year to the pooled three pre-pandemic years and adjusting for test-taker and PA program covariates, examinees scored significantly higher in 2020; there was no difference in 2021, and they scored lower in 2022. When controlling for covariates, examinees had 1.24 higher odds of failing in 2022 compared to the pooled pre-pandemic period.

Conclusion

Findings suggest that PANCE scores and pass rates were impacted during the third year of the pandemic. PANCE assesses if examinees have the essential clinical knowledge to enter the PA profession. It is crucial to determine whether the pandemic affected PANCE scores and pass rates to ensure PAs provide safe and high-quality patient care.

Background

The COVID-19 pandemic significantly disrupted and created extraordinary challenges for higher education institutions, including medical schools and physician assistant/associate (PA) programs [1, 2]. Academic learning and mental health of medical school and PA students were affected during the pandemic [3,4,5]. Given that PA students complete their medical education in a more condensed timeline than medical school matriculants, the pandemic’s effects on their educational opportunities may be even more pronounced. PA program duration is 26.7 months on average (ranging from 24 to 36 months) and consists of two phases: didactic and clinical [6]. Most programs start in August or May, and the didactic phase typically occurs in the first year, while the clinical period takes place in the second [6]. To adapt to the new reality of the COVID-19 environment, PA programs transitioned didactic education from in-person to online virtual platforms [2]. The Association of American Medical Colleges (AAMC) recommended suspending clinical rotations for medical students [7], this resulted in substantially diminished clinical training opportunities for both medical and PA students [2].

The Physician Assistant Education Association (PAEA) launched multiple surveys during the pandemic to capture data on how COVID-19 has impacted programs and how they are adapting. The first was conducted between April and May 2020, showing that 20.9% of programs delayed transitioning students to the clinical phase, and 68.8% reported that student stress and emotional health have worsened since the onset of the pandemic [8]. A few months later, in July 2020, PAEA identified that 66.9% of programs had suspended supervised clinical practice experiences, 85.7% indicated that clinical sites are taking fewer students compared to before the pandemic, and a higher proportion than in the previous survey (84.8%) confirmed that student stress and mental health concerns have increased [9]. The third survey collected program data from January through February 2021, revealing that 67.5% of programs experienced suspension of rotations by clinical sites and/or preceptors, 44.8% had unplanned changes to or from online instruction, and 35.6% had cancellations by instructors, guest lecturers, community partners, or other individuals outside of principal faculty [10]. By December of 2021, lower proportions than in the previous survey experienced suspension of rotations (50.6%), and a much lower percentage had unplanned changes to instruction modalities (16.7%) but a similar proportion in lecture cancellations (34.6%) [11].

The pandemic also affected the certification and licensing exam administration processes and testing conditions [12]. Many testing centers were closed or limited the number of examinees to maintain social distancing. The American Board of Medical Specialties (ABMS) and member boards allowed for greater flexibility in certification requirements and extended deadlines [13]. Similarly, the National Commission on Certification of Physician Assistants (NCCPA) increased support and flexibility with its certification programs during the pandemic. To become certified, a requirement to obtain a license to practice medicine, graduates of accredited PA programs must pass the Physician Assistant National Certifying Examination (PANCE). Testing for NCCPA exams was closed for about two weeks at the beginning of the pandemic. NCCPA worked closely with Pearson VUE to increase testing capacity (more seats and scheduling opportunities) under social distancing guidelines. This was implemented by expanding beyond the Pearson Professional Centers to allow exams to be administered through third-party sites. Pearson VUE worked with various local and state governments to have PAs included as essential workers. NCCPA expanded the timeframe to allow PA students to schedule their exams earlier to help them get a preferred seat and location, moving from 90 days prior to graduation to 180 days.

Although valuable data exist on how PA programs were affected by COVID-19, the pandemic’s impact on PANCE performance has not yet been investigated. To address this research gap, we sought to examine whether the pandemic impacted first-time examinees’ PANCE scores and passing rates while controlling for covariates. Based on the findings regarding the pandemic’s effect on PA programs, we anticipated that PANCE scores for examinees in 2022 would be most affected, given that PA students experienced two years of pandemic-disrupted education (both their didactic and clinical training years), followed by examinees in 2021 who completed their clinical year during COVID-19. First-time PANCE takers in 2020 would have mostly completed their PA education before the start of the pandemic. However, given that they took PANCE during COVID-19, we speculated that their scores might be influenced by changes to exam administration conditions along with societal, economic, and health challenges faced during this difficult time.

Methods

Study design and data

This retrospective repeated cross-sectional study utilized six years (2017 to 2022) of administrative data (N = 59,459) from the NCCPA, the only certifying board for PAs in the US. The primary exposure of interest, outcome variables, and covariates had less than 1% missing data and are described below.

Pre-pandemic vs. during timeframe

The main exposure was the timeframe. We categorized the three-year pre-pandemic period as separate years for the descriptive and bivariate analyses and pooled 2017–2019 for the multivariate analyses to use as a reference category. The three pandemic years (2020, 2021, and 2022) were analyzed separately, given that examinees may have been impacted differently in each of those years.

PANCE score and passing rate

Graduates of accredited PA programs take PANCE, a five-hour exam with 300 multiple-choice questions. The vast majority (99%) of PA graduates take PANCE within a year of completing PA school. The scaled scoring involves the number correct and the relative item difficulty. The cut score is determined via standard setting using the modified Angoff method [19], and scores can range from 200 to 800. Examinations using this scale are built to standard specifications defined in the NCCPA PANCE blueprint [20]. Item difficulty and the ability estimate of examinees are equated to the same scale; thus, direct comparisons of performance from year to year can be made. However, scores on PANCE tend to decrease temporarily after a new standard setting and cut score is implemented. The most recent passing standard was established in 2019, and pass rates decreased slightly during that year. For this study, we rescaled PANCE scores in the two prior years (2017 and 2018) to be on the same metric as 2019 through 2022. We applied the new standard retroactively so that we could compare the pre and during-COVID yearly cohorts using the same policy-based standard. Essentially, we converted the same IRT-based scale to the new scale score metric so we could make direct comparisons. This was done to prevent bias, as the old pass score was more permissive and would have exaggerated the performance difference. Examinees must attain or exceed the scaled passing score of 350. There were 64,436 total PANCE administrations (including multiple attempts) between 2017 and 2022; only first-time PANCE scores were included for the purposes of this study (59,459).

Covariates

We included examinee (age when taking PANCE and gender) and PA program characteristics (number of years the program has been accredited, US region, and rural-urban setting) as control variables. The number of PA programs grew substantially during the study period: 194 in 2017, 207 in 2018, 221 in 2019, 226 in 2020, 243 in 2021, and 252 in 2022.

Data analysis

First, we calculated descriptive statistics (mean [M] and standard deviations [SD] for continuous variables and counts and percentages for categorical) for all examinee and PA program characteristics. Then, we conducted bivariate analyses (chi-square tests of independence, Pearson correlations, t-tests, or one-way analysis of variance [ANOVA] followed by Scheffé post-hoc tests, as appropriate) to determine if there were significant associations between examinee and program characteristics and time period. Similarly, to assess if there were statistically significant associations between test-taker and PA program characteristics with PANCE scores, we conducted Pearson correlations, independent t-tests, or ANOVA followed by Scheffé post-hoc tests, as appropriate. Independent t-tests or chi-square tests of independence were used to assess significant associations between the examinee and program characteristics and pass rates. Given that directly comparing unadjusted results from before to during the pandemic can provide misleading estimates due to differences in examinee characteristics [21], multiple linear regression was performed to examine the association of timeframe (pooled pre-pandemic years [reference group] vs. each pandemic year) with PANCE scores controlling for all examinee and PA program characteristics. Finally, multivariate logistic regression examined the relationship between time period and odds of failing PANCE when adjusting for covariates. To ensure that multicollinearity was not an issue in linear regression and multivariate logistic regression, we computed the variance inflation factor (VIF). All VIF values were well under 5, indicating an absence of multicollinearity [22]. All statistical tests were two-tailed; analyses were conducted in R version 4.2.1.

Results

Examinee and PA program characteristics

Table 1 presents test-taker and PA program characteristics overall and by time period. During the study period, the mean age when taking PANCE was 28.3 (ranging from 21 to 72), and the majority (74.0%) were female. There was a statistically significant difference in mean age by time period, F(5, 59453) = 19, p < .001). Scheffé test for multiple comparisons showed that the mean age in 2017 (M = 28.6, SD = 5.2) was not significantly different from 2018 (M = 28.5, SD = 5.0, p = .597) but higher compared to 2019 (M = 28.2, SD = 4.7, p < .001), 2020 (M = 28.2, SD = 4.7, p < .001), 2021 (M = 28.1, SD = 4.5, p < .001) and 2022 (M = 28.1, SD = 4.4, p < .001). Similarly, mean age was higher in 2018 vs. 2019 (p = .032), 2020 (p = .003), 2021 (p < .001) and 2022 (p < .001). None of the other age-by-year comparisons were statistically significantly different. There was also a significant association of gender with time period X² (5) = 30, p < .001. The mean number of years PA programs have been accredited was 22.0 (ranging from 1 to 50 years), and the vast majority were located in urban settings. The highest proportion of examinees was from programs in the southern US region (35.7%), and the lowest was from the West (14.5%).

Table 1 Participant and PA program characteristics overall and by time period

Figure 1 illustrates the overall and three years before and during the pandemic mean PANCE scores and failure rates. There was a statistically significant association between time period and mean PANCE scores F(5, 59453) = 85, p < .001). Post-hoc tests showed that the mean PANCE score in 2017 (M = 456.0, SD = 80.1) was lower than in 2018 (M = 466.0, SD = 77.5, p < .001), 2019 (M = 464.5, SD = 77.1, p < .001), 2020 (M = 473.1, SD = 79.1, p < .001), 2021 (M = 464.6, SD = 80.3, p < .001) but not 2022 (M = 453.1, SD = 77.4, p = .280). Mean PANCE scores in 2018 were not significantly different from 2019 (p = .867) but lower than 2020 (p < .001), not significantly different than 2021 (p = .887), and significantly lower than 2022 (p < .001). Mean PANCE scores in 2019 were lower than in 2020 (p < .001), not significantly different than 2021 (p = 1.000) and significantly higher than in 2022 (p < .001). Mean PANCE scores were higher in 2020 than in 2021 (p < .001) and 2022 (p < .001). Finally, mean PANCE scores were higher in 2021 than 2022 (p < .001). There was also a significant association between time period and PANCE failure rate X² (5) = 97, p < .001. The highest failure rate was observed in the third year of the pandemic (8.3%) and the lowest in the first (5.4%).

Fig. 1

PANCE mean scores and failure rates by time period. Note PANCE Physician Assistant National Certifying Examination

In terms of examinee characteristics, we found that age during PANCE was negatively correlated with exam performance (r = − .153, p = < 0.001), such that with increasing age, scores decreased (Table 2). When assessing differences by gender, we found that female examinees had significantly higher PANCE scores (M = 465.9, SD = 78.5) than males (M = 454.2, SD = 79.3), t(59447) = 15.97, p < .001. Regarding PA program characteristics, there was a positive correlation between the number of years programs have been accredited and PANCE scores (r = .024, p = < 0.001). One-way ANOVA revealed a significant relationship between PA program US region and PANCE scores F(3, 59418) = 83.04, p < .001). Scheffé test for multiple comparisons demonstrated significant differences between all regions (all p < .001). PANCE scores were highest in the Midwest (M = 470.5, SD = 76.9), followed by Northeast (M = 464.9, SD = 79.9), South (M = 461.1, SD = 78.0) and West (M = 453.5, SD = 80.0). We also detected a significant association between PA rural-urban setting and PANCE scores F(2, 58795) = 49.76, p < .001). Results of the post-hoc analysis showed that PA programs located in urban areas (M = 463.6, SD = 79.0) had higher scores than those in large rural (M = 445.9, SD = 74.5, p < .001) and small rural (M = 453.0, SD = 73.3, p = .015). There was no significant difference in PANCE scores between PA programs located in large vs. small rural settings.

Table 2 Bivariate associations of test-taker and program characteristics with PANCE scores

Table 3 shows the results of bivariate analyses examining test-taker and PA program characteristics and passing vs. failing PANCE. Examinees who passed had significantly lower age during the exam (M = 28.2, SD = 4.6) than those who failed (M = 30.0, SD = 6.1), t(59457) = -24.86, p < .001. We detected a statistically significant difference in passing rate by gender χ² (1) = 76.49, p < .001, whereby female vs. male examinees were more likely to pass (93.6% vs. 91.5%). There was also a significant difference in passing rate by PA program region χ² (3) = 115.85, p < .001 and program rural-urban setting χ² (2) = 15.68, p < .001.

Table 3 Bivariate associations of test-taker and program characteristics with passing vs. failing PANCE

Given the significant associations of the examinee and PA program characteristics with PANCE scores as well as with time period in bivariate tests, the next set of analyses sought to adjust for these covariates when assessing the relationship between pre-pandemic vs. each of the three pandemic years on PANCE performance. As shown in Table 4, results of multiple regression accounting for covariates demonstrate that compared to the pooled three-year pre-pandemic period, during the first pandemic year, PANCE scores increased (B = 10.23, SE = 0.91, p < .001); were not significantly associated with the second pandemic year (B = 0.93, SE = 0.88, p = .290); and significantly decreased in the third pandemic year (B = -10.27, SE = 0.89, p < .001). This model accounted for only 3.5% of the variance in PANCE scores.

Table 4 Multivariate regression results: associations of time period, test-taker, and program characteristics with PANCE scores

Similarly, when controlling for all covariates, multivariate logistic regression (see Table 5) showed that when compared to the pre-pandemic period, the first pandemic year was associated with lower odds of failing PANCE (Odds Ratio [OR] = 0.76, p < .001); there was no significant association with the second pandemic year (OR = 1.00, p = .975). In the third year, examinees had 1.24 higher odds of not passing PANCE than in the pooled three-year pre-pandemic period.

Table 5 Multivariate logistic regression results: associations of time period, test-taker, and program characteristics with failing PANCE

Discussion

The purpose of this study was to investigate the trajectory of PANCE performance from before to during the COVID-19 pandemic. Using six years of data (three pre-pandemic years and three during), we compared PANCE scores and passing rates. In unadjusted analyses comparing each year individually, mean PANCE score was highest in 2020 and lower than all other years in 2022 except for 2017. In adjusted analyses, we observed that PANCE scores and pass rates increased in 2020, did not differ in 2021, and decreased only in the third pandemic year compared to the pooled three-year pre-pandemic period. There could be a number of potential explanations accounting for this pattern in our findings.

The observation that PANCE scores and pass rates increased in 2020 during the first year of the pandemic compared to before is counterintuitive. We expected performance to either remain the same, given that the pandemic did not impact examinees’ education, or deteriorate, given the immense disruption caused by COVID-19 to daily life and changes to exam administration at testing centers. However, testing for NCCPA exams was closed for only about two weeks at the pandemic’s beginning, and NCCPA worked closely with Pearson VUE to maximize capacity. Moreover, NCCPA expanded the timeframe enabling PA students to schedule their exams up to 180 days prior to graduation to help them get a preferred seat and location. One plausible explanation for the increase in PANCE scores and pass rates in 2020 is that the most confident prospective PAs self-selected to complete the exam at the beginning of the pandemic. Although we did not assess and control for this unobservable, almost all (99%) of PA program graduates eligible to take the PANCE do so within a year after graduation. Another potential explanation is that given the 2019 standard increased the amount of medical knowledge required for successfully passing the examination, PA faculty may have implemented additional preparations for their students. PA programs may have adapted teaching and implemented flagging and remediation of at-risk PA students right after NCCPA set a new passing standard. The observed increase in performance in 2020 is similar to increases in the years following when the passing standard was previously changed. Nonetheless, more research is needed to better understand the increased PANCE performance in 2020.

A concerning finding was that compared to before the pandemic, in 2022, performance decreased: examinees’ scores were approximately 10 points lower and had 1.24 higher odds of failing when adjusting for covariates. As demonstrated by prior reports, many PA programs had to rapidly adapt their operations because of the pandemic, including transitioning to online instruction and temporarily suspending supervised clinical practice experiences for students [2, 8,9,10,11]. First-time PANCE takers in 2022 completed both their didactic and clinical years during the pandemic. The pattern in PANCE pass rates from before to during each of the pandemic years contrasts with descriptive data from the American Board of Family Medicine (ABFM) [17]. The Family Medicine Certification Examination first-time pass rates remained similar between 2018 and 2022. This is not surprising given that family physician training is longer (typically four years of medical school followed by three years of residency and potentially at least one year of fellowship); students who started medical school in 2020 will not be completing Family Medicine Certification Examination until at least 2027. Those who took the board exam in 2020–2022 had completed the majority of their training before the onset of the pandemic. However, there were decreases in the ABFM In-Training-Examination performance, which is offered to residents; it has similar content and predicts passing the Family Medicine Certification Examination [18;23–24]. Moreover, the American Board of Internal Medicine indicated that initial certification pass rates for general internal medicine physicians were lower in 2021 (88%) and 2022 (87%) than in the previous three years (91-93%) [15]. The American Board of Pediatrics observed similar trends for general pediatrics, with pass rates decreasing from 91% in 2018 to 87% in 2019 and 2020 and further dropping to 81% in 2021 and 80% in 2022 [16].

Some boards changed examinations from in-person to remote, such as the American Board of Surgery’s oral certifying exam (CE) [14]. Barry et al. determined that performance on the CE from before to during the pandemic did not meaningfully differ [14]. During COVID-19 surges, many elective operations were canceled, limiting surgical residents’ and fellows’ training and learning opportunities from surgical cases. However, the analysis conducted by the authors used data from CE candidates who completed their surgical training either before or in the initial stages of the pandemic; thus, their training was most likely unaffected [14].

We acknowledge that this repeated cross-sectional study has several limitations. The most important is the threat to internal validity when determining the causal effect of the pandemic on PANCE performance. Our research was exploratory, relying on an observational study design, and thus was hypothesis-generating rather than confirming. Cross-sectional designs can suffer from substantial bias due to unmeasured shifts in the characteristics of the examinees over time or due to other unmeasured contextual confounders affecting performance as time progresses [25]. We assessed PANCE score and pass rate data from six years, of which three years prior to the onset of the pandemic served as the counterfactual and three years during as the exposure. However, how examinees in 2020, 2021, and 2022 would have scored if the pandemic had never occurred can never be determined. We statistically controlled for potentially confounding variables that were significantly associated with year and PANCE performance. These covariates included examinee gender and age, and number of years PA program has been accredited and program US region and rural-urban setting. They were derived from NCCPA administrative records and had little missing data. NCCPA also collects detailed demographic information on the PA workforce that would have been useful and important to control for in our linear and logistic models. However, this information is self-reported after examinees pass the PANCE, and it usually takes a few years for the data to be provided and complete. Thus, we could not include this information in our analyses, given that most of the demographic information for 2021 and 2022 was still missing. Another limitation was that in our analysis, PANCE scores from the first few months of 2020 were included as part of the first pandemic year. Lastly, there was a change in passing standard in 2019. However, we rescaled PANCE scores in the two prior years (2017 and 2018) to be on the same metric as 2019 through 2022 to prevent bias, as the previous passing score was more permissive and would have exaggerated the performance difference. Moreover, when only using 2019 as the pre-pandemic year, the pattern in results was similar. Compared to 2019, scores in 2020 were higher, with no difference in 2021 and lower in 2022. Future studies are needed to further validate the findings.

Despite these limitations, our study is the first to explore the potential impact of the pandemic on PANCE scores and pass rates. Our findings demonstrate that PANCE performance decreased in 2022 but not in the first two years of the pandemic. However, more work is needed to understand what direct impact the pandemic has had on PA student didactic and clinical education and how this affects performance on PANCE. Future research is needed to assess and closely monitor the trajectory of PANCE performance over the next few years. Moreover, given that the pandemic disproportionally affected racial and ethnic minority groups [26, 27], research is needed to elucidate if different subgroups of PANCE examinees were impacted differently by the pandemic and, as a result, differences exist in PANCE performance.

Conclusion

Our study indicates a potential influence on the scores and pass rates of the PANCE examination during the pandemic’s third year. The observed decline in performance is likely due to a multitude of factors, and our findings serve as a basis for generating hypotheses that necessitate further research. PANCE is designed to assess if examinees have the essential clinical knowledge to enter the PA profession. It is crucial to assess whether the pandemic is associated with decreased PANCE scores and pass rates to ensure PAs are provided with the necessary support to successfully transition into practice and render safe and high-quality patient care.