- Research article
- Open Access
- Open Peer Review
Validation of a clinical rotation evaluation for physician assistant students
BMC Medical Education volume 18, Article number: 123 (2018)
We conducted a prospective validation study to develop a physician assistant (PA) clinical rotation evaluation (PACRE) instrument. The specific aims of this study were to 1) develop a tool to evaluate PA clinical rotations, and 2) explore associations between validated rotation evaluation scores and characteristics of the students and rotations.
The PACRE was administered to rotating PA students at our institution in 2016. Factor analysis, internal consistency reliability, and associations between PACRE scores and student or rotation characteristics were determined.
Of 206 PACRE instruments sent, 124 were returned (60.2% response). Factor analysis supported a unidimensional model with a mean (SD) score of 4.31 (0.57) on a 5-point scale. Internal consistency reliability was excellent (Cronbach α=0.95). PACRE scores were associated with students’ gender (P = .01) and rotation specialty (P = .006) and correlated with students’ perception of being prepared (r = 0.32; P < .001) and value of the rotation (r = 0.57; P < .001).
This is the first validated instrument to evaluate PA rotation experiences. Application of the PACRE questionnaire could inform rotation directors about ways to improve clinical experiences. The findings of this study suggest that PA students must be adequately prepared to have a successful experience on their rotations. PA programs should consider offering transition courses like those offered in many medical schools to prepare their students for clinical experiences. Future research should explore whether additional rotation characteristics and educational outcomes are associated with PACRE scores.
Physician assistants (PAs) are vital to all aspects of health care delivery. The number of PA training programs continues to increase to meet the demand for access to health care [1,2,3]. In the past decade, the number of accredited PA programs and applicants to these programs has increased dramatically [4, 5]. An integral part of these programs are clinical rotations, which are limited by competition and shortages [6,7,8,9]. The educational quality of these clinical rotations can vary . Maintaining standards in clinical rotations, including validated assessment of performance, is a prerequisite to ensuring a high-quality PA workforce.
Data are sparse regarding the evaluation of training programs and individual clinical rotations in other educational settings. In medical school, students have evaluated clerkships using electronic Likert-scaled checklists . In graduate medical education, residents have used validated questionnaires to evaluate their programs with respect to rotation workload, faculty/learning environment, and stress [12,13,14,15]. Other studies have examined resident assessments of programs and rotations in surgery , internal medicine [17, 18], and anesthesiology . Although PAs are trained in the medical model, differences in clinical rotation length, content, supervision, and logistics make the use of existing clinical rotation evaluations less relevant to PA training settings. We are unaware of prior research on validated measures of PA clinical rotations.
To fill this gap, we conducted a prospective validation study to develop a PA clinical rotation evaluation (PACRE) instrument. The specific aims of this study were to 1) develop a tool to evaluate PA clinical rotations, and 2) once validated, explore associations between PACRE rotation evaluation scores and characteristics of the students and rotations. The purpose of the PACRE instrument was to determine and document a student’s perceptions of a rotation based on research on components of effective clinical teaching. We hypothesized that such an instrument would have strong internal structure validity and scores would be associated with rotation or student demographic variables.
Participants and clinical sites
This prospective validation study involved PA students who completed a clinical rotation in 2016 at1) the Mayo Clinic School of Health Sciences/Gundersen Medical Foundation/University of Wisconsin – La Crosse PA Program or 2) the Mayo Clinic or the Mayo Clinic Health System from other PA programs. The study was deemed exempt by the Mayo Clinic institutional review board (Identification number: 15–006040).
PACRE instrument development
A PACRE questionnaire was developed on the basis of existing literature [12, 14, 15, 20,21,22,23,24,25,26,27,28]. Items were developed for each of the Stanford Faculty Development Program (SFDP) for Clinical Teaching categories: learning climate, control of session, communication of goals, promotion of understanding and retention, evaluation, feedback, and promotion of self-directed learning [24, 29]. Two additional categories—rotation logistics and a rotation global assessment—were included. After iterative revision, 2 items were selected for each of the 9 domains, for a total of 18 items in the PACRE instrument (Table 1). Responses were based on a 5-point Likert scale (1, strongly disagree, to 5, strongly agree). The final PACRE instrument was pilot tested on 5 former PA students and 5 current PA colleagues, which led to minor rewording of some items.
Data collection and analysis
The PACRE was sent via an emailed link to each PA student at the completion of the clinical rotation. Our survey research center managed the data collection using Qualtrics software (Qualtrics LLC). A reminder email was sent 1 week after the completion of the rotation.
Demographic characteristics were collected, including gender, age, and number of previous rotations. Rotation characteristics were collected, including rotation specialty (general practice, medicine subspecialty, medical specialty, pediatrics, surgery, other), rotation type (required, elective), and length of rotation (4 or 6 weeks). These demographic and rotation characteristics were chosen based on the authors’ hypothesis for potential associations with PACRE scores and the availability of accurate date. Each student ranked the following questions on a 5-point Likert scale: 1) My program adequately prepared me for this rotation; and 2) This rotation prepared me for being a PA.
Factor analysis was completed on the PACRE instrument item scores. “To account for the clustering of multiple ratings by students completing more than 1 rotation evaluation, we generated an adjusted correlation matrix using generalized estimating equations. This adjusted correlation matrix was then used to perform factor analysis with orthogonal rotation. For a sensitivity analysis, we performed a factor analysis using an unadjusted correlation matrix and within-student combinations” . Specifically, the sensitivity analysis involved conducting the factor analysis at lowest level of measurement (the student) and also higher levels of measurement (group-averaged scores), and then comparing these different levels of analysis to determine if they reveal similar or identical factor structures, which would then support reporting factor analysis of the higher, nested level of measurement. The number of factors to be retained was determined based on the eigenvalue criterion (factors with eigenvalues > 1). The final model was confirmed by reviewing the scree plot. Items with factor loadings ≥0.60 were retained. Internal consistency reliability was calculated using the Cronbach α, where α greater than 0.7 is considered acceptable . Additionally, for internal structure validity determination an evaluation-to-item ratio should range from 5:1 to 10:1 . The 18 items that make up the PACRE instrument would require between 90 and 180 completed instruments in order to be powered to complete the factor analysis.
Categorical variables are presented as count (percentage) and continuous variables are presented as mean (SD). Associations between PACRE instrument scores and categorical student demographic and clinical rotation characteristics were determined using the Wilcoxon rank sum test (if 2 groups) or Kruskal-Wallace test (if more than 2 groups). Pearson correlation coefficients were calculated to explore the relationship between PACRE scores and continuous participant or rotation characteristics (0.1–0.3, small correlation; 0.3–0.5, medium correlation; 0.5–1, large correlation) . Given multiple comparisons, the threshold for statistical significance was set at P≤.01. Statistical analysis was conducted using SAS version 9.3 software (SAS Institute, Inc.).
Of 206 surveys sent to 41 unique PA students, 124 surveys were returned (60.2% response rate) by 33 students. Of the responses, 118 surveys from 32 students contained both evaluation and demographic data, and these are the data source for this study. There were 28 students from the La Crosse program and 4 students were from other PA programs. The 32 students completed between 1 and 7 rotation evaluations. Student demographics are shown in Table 2.
PACRE instrument validation
Factor analysis of the PACRE instrument showed a unidimensional model for assessing PA clinical rotations (Table 1). The eigenvalue for the PACRE instrument’s one factor was 9.63. Eigenvalues for all additional factors were < 1, which supports a one factor model (Fig. 1). Item factor loadings were all higher than 0.6, except for 2 items developed for “promotion of self-directed learning.” These items were removed from the remainder of the analysis and future iterations of the PACRE instrument. The internal consistency reliability was excellent (Cronbach α=0.95). The item mean (SD) scores ranged from 4.08 (0.81) to 4.56 (0.71). The mean overall PACRE score was 4.31 (0.57).
Associations between PACRE scores and student or rotation characteristics
PACRE scores were associated with the students’ gender. Men gave significantly higher PACRE scores than women (mean [SD], 4.7 [0.4] vs 4.3 [0.6]; P = .01). There were no significant associations between rotation evaluation scores and student age or PA program (tertiary center vs health system).
PACRE scores were associated with the specialty of the rotation. General practice rotations had the highest PACRE scores (4.6 [0.5]), and surgical rotations had the lowest (4.1 [0.5]; P = .006). There were no significant associations between rotation evaluation scores and rotation type or length.
Analysis indicated significant relationships between PACRE scores and Likert-scaled responses to 1) My program adequately prepared me for this rotation (Pearson correlation coefficient, 0.32; P < .001) and 2) This rotation prepared me for being a PA (Pearson correlation coefficient, 0.57; P < .001). These represent a medium correlation and a large correlation, respectively .
Regarding statistical power, for a binary variable with a prevalence of 50%, the sample size of 118 students has 80% power to detect a moderate effect size of 0.52 standard deviations or higher based on a two sample of t-test with equal variance. For continuous variables, a sample of size 118 students has 80% power to detect a correlation coefficient of 0.26 or higher between any two continuous variables, such as age versus PACRE score.
To our knowledge, this is the first study to report the validity of a clinical rotation assessment in PA education. The newly developed PACRE questionnaire represents a unidimensional model with strong internal consistency reliability. Student rotation evaluation scores were associated with the students’ gender and specialty of the rotation but not with whether it was required or elective. There was a positive correlation between the rotation evaluation scores and whether the student felt prepared for the rotation and whether they felt the rotation prepared them to be a PA.
This study adds to what is known about evaluations of clinical teaching and rotations. Research on assessments of clinical teaching in other educational settings has been previously published. Factorial validation of the SFDP core components among medical students resulted in the SFDP26, a questionnaire that consists of 25 items organized around these competencies and 1 item for overall effectiveness . The SFDP26 has been applied to resident education  and has been translated and validated in additional languages [23, 34, 35]. For example, a teaching evaluation form, which was based on the SFDP26 and our institution’s Faculty Resident Electronic Evaluation System, was developed and tested among residents . Research on clinical teaching of PA students found that characteristics of an effective and ineffective preceptor aligned with themes defined in the SFDP . Two other studies used allied health students (including some PA students) to evaluate effective clinical instruction and found similar results [37, 38]. The PACRE instrument described in this study is unique in that it was specifically designed for and tested in a PA student population and focused on the overall rotation experience, not just clinical teaching.
The current study builds on prior work regarding student and rotation factors associated with perceived quality of the rotation. We found that rotation evaluation scores were correlated with student gender. Although we are unaware of studies exploring relationships between gender and rotation evaluation scores, previous work has demonstrated an association between medical student gender and medical school performance [39, 40]. We found that clinical rotation evaluation scores were associated with rotation specialty, feeling prepared for the rotation, and a better perception of the value of the rotation. Studies of medical students and residents have demonstrated that rotation quality is related to rotation specialty [23, 41], clinic schedule design , learning climate , requirements for overnight call , quality of feedback [44, 45], caseload , continuity , and level of faculty involvement . In our study, associations between rotation quality and rotation specialty suggest that differences between specialties exist and that future studies could focus on elucidating these differences. The finding that rotation evaluation scores correlated with being prepared for the rotation is concordant with the current trend in medical schools to offer transition courses for early clinical experiences .
The PACRE questionnaire has compelling validity evidence. A common approach to validity in medical education research includes content, internal structure, and relations to other variables evidence . Content validity for the PACRE questionnaire is based on published assessments of clinical teaching [12, 14, 15, 20,21,22,23,24,25,26,27,28,29], iterative revision of instrument items, and pilot testing. Internal structure validity is supported by a single factor to measure student perceptions’ of clinical rotations and excellent internal consistency reliability. Relations to other variables validity evidence is demonstrated by associations between clinical rotation evaluation scores and gender, rotation specialty, feeling prepared for the rotation, and viewing the rotation as valuable. Future study should determine if associations between PACRE questionnaire scores and other rotation outcomes including knowledge assessments exist.
Limitations and strengths
First, all students did rotations through 2 programs, which could limit the generalizability of the findings. However, analyses of published medical education studies indicate that most are single institution studies . Second, the majority of responders in this study were female, which may limit generalizability. Third, while the response rate in this study was excellent, there could be differences between those that did and did not complete the survey. Fourth, responses from the PACRE instrument are reaction outcomes rather than the higher outcomes of learning, behavior, or results . Yet, reviews of medical education research suggest that reaction-level outcomes are commonly reported . Fifth, the utilization of 124 surveys from 33 students represents a relatively small number for factor analysis. Sixth, the SFDP questionnaire was originally developed for assessing only preceptors and our utilization of the SFDP framework includes application of items to both preceptors and the program. However, most of the items reflect students’ perceptions of their preceptors, the items that are applied to the program are applied in ways that are true to the item’s original intent (e.g., “The rotation goals were stated clearly”), and we note that an advantage is that ours is the first study to provide robust validity evidence for use of the SFDP framework for evaluation of students perceptions of a PA program. Seventh, PACRE evaluation scores should be considered in the context of other outcomes including faculty evaluations and knowledge assessments (e.g. rotation examinations, observed structured clinical examinations) to fully evaluation a rotation. Finally, certain statistically significant score differences in this study (e.g., general practice PACRE score = 4.6 versus surgical rotations PACRE score = 4.1; p = 0.006) may seem small; nonetheless, in many education settings the inflation and range restriction of assessment scores is very narrow, and such magnitudes of difference could potentially separate the best rotations from the rest. Strengths of this study include a rigorous survey development process, use of a dedicated survey research program, robust statistical methods to establish instrument validity, and high response rate.
This study has important implications for leaders of PA clinical rotation experiences. First, we believe that this is the first report of a valid method for assessing PA clinical rotation experiences. Application of the PACRE questionnaire could inform rotation directors about ways to improve clinical experiences. Given that the demand for PAs is rapidly increasing, the PACRE questionnaire could provide a consistent technique for ensuring that rotations provide meaningful teaching and clinical exposure. Second, the findings of this study suggest that PA students must be adequately prepared to have a successful experience on their rotations. PA programs should consider offering transition courses like those offered in many medical schools to prepare their students for clinical experiences. Third, variability exists among specialties regarding perceived quality of rotations. PA programs should work to provide standard requirements for rotations such as a curriculum, evaluation standards, competencies, and clinical exposure. Future research should explore whether additional rotation characteristics (e.g., didactic content, evaluation methods, call schedules) and educational outcomes (e.g., learning, behavior change) are associated with PACRE scores.
PA clinical rotation evaluation
Stanford Faculty Development Program
Dall T, West T, Chakrabarti R, Iacobucci W; Association of American Medical Colleges. The complexities of physician supply and demand: 2016 update: projections from 2014 to 2025 [Internet]. 2016 Apr 5 [cited 2017 Sept 18]; [51 p.]. Available from: https://www.aamc.org/download/458082/data/2016_complexities_of_supply_and_demand_projections.pdf
Green LV, Savin S, Lu Y. Primary care physician shortages could be eliminated through use of teams, nonphysicians, and electronic communication. Health Aff (Millwood). 2013;32(1):11–9.
Hofer AN, Abraham JM, Moscovice I. Expansion of coverage under the patient protection and affordable care act and primary care utilization. Milbank Q. 2011;89(1):69–89.
McDaniel MJ, Hildebrandt CA, Russell GB. Central application service for physician assistants ten-year data report, 2002 to 2011. J Physician Assist Educ. 2016;27(1):17–23.
Accreditation Review Commission on Education for the Physician Assistant. Accredited Programs [Internet]. 2017 [cited 2017 Sept 18]. Available from: http://www.arc-pa.org/accreditation/accredited-programs/
Erikson C, Hamann R, Levitan T, Pankow S, Stanley J, Whatley M; American Association of Colleges of Nursing; American Association of Colleges of Osteopathic Medicine; Association of American Medical Colleges; Physician Assistant Education Association. Recruiting and maintaining US clinical training sites: joint report of the 2013 multi-discipline clerkship/clinical training site survey [Internet]. [cited 2017 Sept 18]; [24 p.]. Available from: https://members.aamc.org/eweb/upload/13-225%20wc%20report%202%20update.pdf
Herrick A, Pearl J. Rotation shortages in physician assistant education [abstract]. J Am Acad Physician Assist. 2015;28(11):1.
Hudak NM, Enking PJ, Gorney C, Gonzalez-Colaso R. Tales from the trenches: physician assistants’ perspectives about precepting students. J Physician Assist Educ. 2014;25(1):12–9.
Porter S. Just-released clerkship study: growing shortage of clinical training sites challenges medical schools. Ann Fam Med. 2014;12(5):484–6.
Essary A, Berry-Lloyd C, Morgan C. A student-centered approach to physician assistant clinical rotations. J Physician Assist Educ. 2003;14(2):78–83.
Winter RJ, Jones RJ. Clerkship evaluation by students: a standardized electronic mechanism. Acad Med. 1997;72(5):418.
Bellini L, Shea JA, Asch DA. A new instrument for residency program evaluation. J Gen Intern Med. 1997;12(11):707–10.
Seelig CB. Changes in residents’ attitudes in response to residency program modifications: a prospective study. South Med J. 1992;85(10):972–5.
Seelig CB. Quantitating qualitative issues in residency training: development and testing of a scaled program evaluation questionnaire. J Gen Intern Med. 1993;8(11):610–3.
Seelig CB, DuPre CT, Adelman HM. Development and validation of a scaled questionnaire for evaluation of residency programs. South Med J. 1995;88(7):745–50.
Kirton OC, Antonetti M, Morejon O, Dobkin E, Angelica MD, Reilly PJ, et al. Measuring service-specific performance and educational value within a general surgery residency: the power of a prospective, anonymous, web-based rotation evaluation system in the optimization of resident satisfaction. Surgery. 2001;130(2):289–95.
Davisson L, Nuss M, Cottrell S. Women’s health curriculum for internal medicine residents: development, implementation, and evaluation. J Grad Med Educ. 2010;2(3):398–403.
Akl EA, Maroun N, Klocke RA, Schunemann HJ. A survey of internal medicine residents and faculty about the duration of attendings’ inpatient rotations. J Gen Intern Med. 2004;19(11):1133–9.
Tanaka P, Yanez D, Lemmens H, Djurdjulov A, Scotto L, Borg L, et al. Impact of an innovative classroom-based lecture series on residents’ evaluations of an anesthesiology rotation. Anesthesiol Res Pract. 2016;2016:8543809.
Beckman TJ, Lee MC, Rohren CH, Pankratz VS. Evaluating an instrument for the peer review of inpatient teaching. Med Teach. 2003;25(2):131–5.
Dolmans DH, Wolfhagen IH, Essed GG, Scherpbier AJ, van der Vleuten CP. The impacts of supervision, patient mix, and numbers of students on the effectiveness of clinical rotations. Acad Med. 2002;77(4):332–5.
Hunter AJ, Desai SS, Harrison RA, Chan BK. Medical student evaluation of the quality of hospitalist and nonhospitalist teaching faculty on inpatient medicine rotations. Acad Med. 2004;79(1):78–82.
Kihlberg P, Perzon M, Gedeborg R, Blomqvist P, Johansson J. Uniform evaluation of clinical teaching: an instrument for specific feedback and cross comparison between departments. Högre Utbildning. 2011;1(2):139–50.
Litzelman DK, Stratos GA, Marriott DJ, Skeff KM. Factorial validation of a widely disseminated educational framework for evaluating clinical teachers. Acad Med. 1998;73(6):688–95.
Owolabi MO. Development and psychometric characteristics of a new domain of the Stanford faculty development program instrument. J Contin Educ Heal Prof. 2014;34(1):13–24.
Skeff KM, Stratos GA, Berman J, Bergen MR. Improving clinical teaching: evaluation of a national dissemination program. Arch Intern Med. 1992;152(6):1156–61.
Sutkin G, Wagner E, Harris I, Schiffer R. What makes a good clinical teacher in medicine? A review of the literature. Acad Med. 2008;83(5):452–66.
Valdivia TD, Hartquist BA. An ACCESS-based academic attending physician and resident rotation evaluation database. Proc Annu Symp Comput Appl Med Care. 1994;1000.
Skeff KM. Enhancing teaching effectiveness and vitality in the ambulatory setting. J Gen Intern Med. 1988;3(2 Suppl):S26–33.
Post JA, Wittich CM, Thomas KG, Dupras DM, Halvorsen AJ, Mandrekar JN, et al. Rating the quality of entrustable professional activities: content validation and associations with the clinical context. J Gen Intern Med. 2016;31(5):518–23. Epub 2016 Feb 22
RF DV. Scale development: theory and applications. Newbury Park (CA): Sage. p. c1991.
Cohen J. Statistical power analysis for the behavioral sciences. Hillsdale (NJ): L. In: Erlbaum associates; 1988.
Litzelman DK, Westmoreland GR, Skeff KM, Stratos GA. Factorial validation of an educational framework using residents’ evaluations of clinician-educators. Acad Med. 1999;74(10 Suppl):S25–7.
Iblher P, Zupanic M, Hartel C, Heinze H, Schmucker P, Fischer MR. The questionnaire “SFDP26-German”: a reliable tool for evaluation of clinical teaching? GMS Z Med Ausbild 2011;28(2):Doc30.
Wong JG, Agisheva K. Developing teaching skills for medical educators in Russia: a cross-cultural faculty development project. Med Educ. 2007;41(3):318–24.
Zayas TT. Qualities of effective preceptors of physician assistant students. J Phycian Assist Educ. 1999;10(1):7–11.
Jarski RW, Kulig K, Olson RE. Allied health perceptions of effective clinical instruction. J Allied Health. 1989;18(5):469–78.
Rogers JL, Lautar CJ, Dunn LR. Allied health students’ perceptions of effective clinical instruction. Health Care Manag (Frederick). 2010;29(1):63–7.
Bienstock JL, Martin S, Tzou W, Fox HE. Medical students’ gender is a predictor of success in the obstetrics and gynecology basic clerkship. Teach Learn Med. 2002;14(4):240–3.
Haist SA, Wilson JF, Elam CL, Blue AV, Fosson SE. The effect of gender and age on medical school performance: an important interaction. Adv Health Sci Educ Theory Pract. 2000;5(3):197–205.
Lewisson N, Hellgren L, Johansson J. Quality improvement in clinical teaching through student evaluations of rotations and feedback to departments. Med Teach. 2013;35(10):820–5.
Wieland ML, Halvorsen AJ, Chaudhry R, Reed DA, McDonald FS, Thomas KG. An evaluation of internal medicine residency continuity clinic redesign to a 50/50 outpatient-inpatient model. J Gen Intern Med. 2013;28(8):1014–9.
Dyrbye LN, Thomas MR, Harper W, Massie FS Jr, Power DV, Eacker A, et al. The learning environment and medical student burnout: a multicentre study. Med Educ. 2009;43(3):274–82.
Remmen R, Denekens J, Scherpbier A, Hermann I, van der Vleuten C, Royen PV, et al. An evaluation study of the didactic quality of clerkships. Med Educ. 2000;34(6):460–4.
Torre DM, Sebastian JL, Simpson DE. Learning activities and high-quality teaching: perceptions of third-year IM clerkship students. Acad Med. 2003;78(8):812–4.
Schneider JR, Coyle JJ, Ryan ER, Bell RH Jr, DaRosa DA. Implementation and evaluation of a new surgical residency model. J Am Coll Surg. 2007;205(3):393–404.
Poncelet A, O’Brien B. Preparing medical students for clerkships: a descriptive analysis of transition courses. Acad Med. 2008;83(5):444–51.
Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119:166.e7–16.
Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM. Association between funding and quality of published medical education research. JAMA. 2007;298:1002–9.
Kirkpatrick D. Revisiting Kirkpatrick's four-level model. Training and. Development. 1996;50:54–9.
The authors report no external funding source for this study.
Availability of data and materials
The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.
Ethics approval and consent to participate
This study was deemed exempt by the Mayo Clinic institutional review board. Given that the research was conducted in an established educational setting involving normal educational practices, the Mayo Clinic institutional review board did not require consent from the participants.
The authors declare they have no competing interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Meverden, R.A., Szostek, J.H., Mahapatra, S. et al. Validation of a clinical rotation evaluation for physician assistant students. BMC Med Educ 18, 123 (2018) doi:10.1186/s12909-018-1242-y
- Clinical rotation
- Physician assistant education
- Validation study