Learning environment assessments of a single curriculum being taught at two medical schools 10,000 miles apart

Abstract

Background

Perdana University Graduate School of Medicine (PUGSOM), the first graduate-entry medical school in Malaysia, was established in 2011 in collaboration with Johns Hopkins University School of Medicine (JHUSOM), an American medical school. This study compared learning environments (LE) at these two schools, which shared the same overarching curriculum, along with a comparator Malaysian medical school, Cyberjaya University College of Medical Sciences (CUCMS). As a secondary aim, we compared 2 LE assessment tools - the widely-used Dundee Ready Educational Environment Measure (DREEM) and the newer Johns Hopkins Learning Environment Scale (JHLES).

Methods

Students responded anonymously at the end of their first year of medical school to surveys which included DREEM, JHLES, single-item global LE assessment variables, and demographics questions.

Results

Respondents included 24/24 (100 %) students at PUGSOM, 100/120 (83 %) at JHUSOM, and 79/83 (95 %) at CUCMS. PUGSOM had the highest overall LE ratings (p < 0.05) [DREEM 155.3 (SD 21.3); JHLES 116.5 (SD 12.2)], followed by JHUSOM [DREEM 143.3 (SD 22.5); JHLES 111.7 (SD 12.0)] and CUCMS [DREEM 138.5 (SD 22.4); JHLES 106.4 (SD 14.5)]. PUGSOM’s overall high LE ratings were driven by responses in “perception of teaching,” “meaningful engagement,” and “acceptance and safety” domains. JHLES detected significant differences across schools in 5/7 domains and had stronger correlations than DREEM to each global LE assessment variable.

Conclusions

The inaugural class of medical students at PUGSOM rated their LE exceptionally highly, providing evidence that transporting a medical school curriculum may be successful. The JHLES showed promise as a LE assessment tool for use in international settings.

Background

American academic health centers (AHC) are expanding their activities internationally to diversify and advance their missions in clinical care, research, and education [1, 2]. An ambitious example of this is for an American medical school to start a new medical school in another country. Weill Cornell Medical College in Qatar (WCMC-Q) was the first such venture in 2001 [3], followed by Duke University – National University of Singapore (NUS) Graduate Medical School in Singapore several years later [4]. On November 2, 2010, Johns Hopkins entered into an agreement to create an AHC in Malaysia, where it would work with partners to develop education, research, and clinical facilities in Malaysia’s first graduate-entry medical school: Perdana University Graduate School of Medicine (PUGSOM), which matriculated its first class less than 1 year later. While such enterprising international partnerships can reap strategic mutual benefits, previously reported experiences have described unanticipated challenges in medical education partnerships, particularly in sensitive content areas like cultural competence [5] and ethics [6].

The medical school learning environment, which “encompasses the physical, social, and psychological context in which students learn; all interactions with faculty, staff, and peers; and the formal, informal, and hidden curricula,” [7] is being increasingly recognized as crucial to medical student development and maturation [8–13]. When medical curricula are developed in one context and implemented in another, it is difficult to predict whether a positive learning environment will be created, making empiric learning environment assessment especially important in these situations.

Assessing the character and quality of the medical school learning environment remains challenging. There are at least 15 published instruments to assess undergraduate medical school learning environments, but none provide strong validity evidence [14]. The most widely-used learning environment assessment tool is the Dundee Ready Educational Environment Measure (DREEM) [14, 15]; however the DREEM was not initially grounded in learning theory, its five subscales have not been empirically validated [16], and it may be dated, having been created in the 1990s. Moreover, DREEM has not been used in American medical schools. A new learning environment assessment tool, called the Johns Hopkins Learning Environment Scale (JHLES), displayed promising psychometric properties in its validation study at JHUSOM [17] and may be useful if applied abroad.

The primary goal of this study was to determine whether a high quality learning environment was established during the first year of the implementation of the JHUSOM curriculum at PUGSOM. To do this, we compared students’ assessments of their learning environments at PUGSOM to those of students at JHUSOM, and those at Cyberjaya University College of Medical Science (CUCMS), a medical school established in the M.B.B.S. model in Malaysia. We used both DREEM and JHLES to enhance the validity of our learning environment assessments, and we compared the performance of JHLES and DREEM at the 3 medical schools.

Method

Study design and data collection

This was a cross-sectional survey given to all first year medical students at 3 medical schools at the end of the 2011–2012 academic year. Surveys were administered electronically at PUGSOM and JHUSOM and given on paper, using optical mark reader sheets, at CUCMS. Participation was voluntary, informed consent was obtained from each participant (all of whom were adults), responses were anonymous, and data were de-identified and analyzed by one author who had no role in teaching or evaluating medical students at any of the schools. IRB approval for the complete study protocol was obtained from Johns Hopkins Medicine (NA_00075019 and NA_00065786) and Perdana University (NA_00071425).

Subjects and setting

JHUSOM was founded in Baltimore, MD in 1893 and follows the graduate-entry model (i.e. all matriculating students must have a college degree). The most recent major curriculum revision created the Genes-to-Society curriculum [18], which was implemented for the 2009 entering class. Students experience a mix of educational methods, and peer-learning is emphasized. Students are very active in their learning communities, in student groups, and in other extracurricular activities. A new medical education building opened in 2009.

PUGSOM is a graduate-entry medical school established as a public-private partnership between Chase Perdana and Johns Hopkins. Its first cohort of students was selected in the spring of 2011 and matriculated in September 2011 to an interim campus near the site of the future medical school campus, about 70 km outside of Kuala Lumpur, Malaysia. Of the students in the first cohort at PUGSOM, all are Malaysian citizens, with the exception of one international student. The overarching Genes-to-Society curriculum was in place and taught by many of the same faculty who taught at JHUSOM in Baltimore.

CUCMS is a private medical school located in Cyberjaya, Malaysia. CUCMS was founded in 2005, and its first class matriculated in December 2005. It follows the school-leaver model of medical education (i.e. students enroll in a 5-year medical program after completing a one-year pre-medical matriculation program (post-high school), A-level certificate, or their equivalent. In the first year, students engage in “student-centered team-based learning,” intended to foster interaction and peer-learning. In this model, the class is subdivided into small groups of 10–12 students, groups which are consistent through all activities. Pedagogy is a mix of interactive lectures and problem-based learning.

Survey composition

The survey was developed by our study team from institutions in the U.S. and Malaysia, who have extensive experience in medical education and educational research. The survey included the DREEM [19], JHLES [17], several questions related to an overall assessment of the learning environment, and demographics questions.

The DREEM is a 50-item survey where students respond with their level of agreement across a 5-point scale. Items were grouped by its developers into 5 categories: (1) perception of teachers, (2) perception of teaching, (3) academic self-perception, (4) perception of atmosphere, and (5) social self-perception. Each item is scored 0–4 so that composite DREEM scores can range from 0–200, with higher scores indicating a more positive perception of the learning environment.

The JHLES was developed over a series of steps from 2010 to 2012 at JHUSOM [17]. It has 28 items each with 5-point response options. During development, exploratory factor analysis resulted in 7 domains: (1) community of peers, (2) faculty relationships, (3) academic climate, (4) meaningful engagement, (5) mentorship, (6) acceptance and safety, and (7) physical space. Each item is scored 1–5 so that JHLES totals can range from 28 to 140, with higher scores indicating a more positive perception of the learning environment.

We included 3 single-item global learning environment assessment variables. We asked students to rate (i) their overall perception of the learning environment as either (1) terrible, (2) poor, (3) fair, (4) good or (5) exceptional and to rate their agreement with 2 statements on a 1 (strongly disagree) to 5 (strongly agree) scale: (ii) “The overall quality of the educational experience at the School of Medicine is excellent,” and (iii) “Based on my sense of the learning environment at the School of Medicine, I would recommend it to a close friend as a great school to attend”.

Data analysis

Basic descriptive statistics were tabulated with tests for significant differences applied as appropriate. Overall DREEM and JHLES scores were computed by summing across survey items for each tool. Domain scores were computed by averaging item scores in each domain.

DREEM and JHLES total scores and domain scores were compared across schools, with one-way ANOVA and Kruskall-Wallis tests where appropriate. For institutional pairwise comparisons, we used t-tests with Bonferroni correction.

Pearson correlation coefficients were calculated to determine associations between each of the scales and their respective subcategories or domains. Spearman correlation coefficients were calculated for associations between learning environment scale totals and global LE assessment variables. Areas-under-the-curve were calculated for receiver-operating-characteristic curves created using JHLES and DREEM scale totals as a predictor of the most favorable response (e.g. “strongly agree”) for each global assessment variable.

Stata 13 (StataCorp. 2013. Stata Statistical Software: Release 13. College Station, TX: StataCorp LP.) was used for data analysis.

Results

We obtained complete surveys from 100/120 (83 %) students at JHUSOM, 24/24 (100 %) students at PUGSOM, and 79/83 (95 %) students at CUCMS (Table 1). Average ages were 23.8 years (SD 2.0) at JHUSOM, 25.3 years (SD 1.9) at PUGSOM, and 20.6 years (SD 1.2) at CUCMS, significantly different at p < 0.01 for all comparisons. Men numbered 50 (50 %) at JHUSOM, 7 (29 %) at PUGSOM, and 34 (43 %) at CUCMS (p = 0.196). Ethnic make-up varied considerably across sites: at JHUSOM most students were white (54 %), at PUGSOM half were Chinese (50 %), and at CUCMS most were Malay (83 %).

Table 1 Characteristics of medical schools and respondents at the 3 study sites

Assessment of learning environments across schools using DREEM and JHLES

DREEM and JHLES totals

On both DREEM and JHLES, the overall learning environment ratings were highest at PUGSOM [DREEM 155.3 (SD 21.3); JHLES 116.5 (SD 12.2)], followed by JHUSOM [DREEM 143.3 (SD 22.5); JHLES 111.7 (SD 12.0)] and then CUCMS [DREEM 138.5 (SD 22.4); JHLES 106.4 (SD 14.5)] (Table 2). There were no significant differences in DREEM or JHLES total scores by age, gender, or ethnicity, when controlling for medical school.

Table 2 DREEM and JHLES total and domain scores from the students at the 3 medical schools studied

DREEM and JHLES subscale and individual items

Ratings differed significantly in 2 of the 5 DREEM categories, for which PUGSOM gave the highest ratings for the “perception of teachers” and the “perception of teaching” categories (Table 2). Overall 15/50 (30 %) DREEM items detected statistically significant differences between schools (p < 0.05 after Bonferroni correction) (Table 3). PUGSOM students responded more favorably than JHUSOM and CUCMS on every item in DREEM’s “perception of teaching” category, with 4/12 (33 %) reaching statistical significance after adjusting for multiple comparisons.

Table 3 Percentage of students responding favorably for items from the DREEM and JHLES that were significantly different between schools

Ratings differed significantly in 5 of the 7 (71 %) JHLES domains (Table 2). CUCMS gave the lowest ratings for “faculty relationships” and “acceptance and safety.” PUGSOM gave the highest ratings for “acceptance and safety.” JHUSOM gave the highest ratings for “physical space.” Overall, 6/28 (21 %) JHLES items detected statistically significant differences between schools (p < 0.05 after Bonferroni correction) (Table 3). PUGSOM students responded more favorably than JHUSOM and CUCMS on every item in the JHLES “meaningful engagement” and “acceptance and safety” domains, with 1/4 (25 %) and 2/3 (67 %) items reaching statistical significance, respectively.

Responses to single-item global learning environment assessment variables

Responses to all global learning environment assessment variables showed large majorities reporting an appreciation of the climate at all schools (Table 4). PUGSOM had 100 % of its students rate the overall learning environment as “exceptional” or “good,” ratings which were significantly higher than those at JHUSOM (p = 0065). PUGSOM had a significantly greater proportion of students (100 %) than JHUSOM (85 %, p = .0001) and CUCMS (81 %, p = .0004) who agreed that the overall quality of their educational experience was excellent. Significantly greater proportions of students at JHUSOM (95 %) than at CUCMS (77 %), p = 0.001, agreed they would recommend their medical school to a friend based on their sense of the learning environment.

Table 4 Percentage of students responding favorably for single-item global assessment variables of the learning environment

Correlations of scales and associations with global learning environment assessment variables

DREEM and JHLES scores were highly correlated with one another overall (r = 0.73), with stronger correlations at PUGSOM (r = 0.80) and CUCMS (r = 0.80) compared to JHUSOM (r = 0.64).

JHLES total score was more highly correlated than DREEM total scores across all schools for each of the 3 global learning environment assessment variables. Correlation coefficients for the total study population for JHLES and DREEM were, respectively: 0.53 vs. 0.44 for overall learning environment quality, 0.64 vs. 0.54 for agreeing that the educational experience is excellent, and 0.63 vs. 0.51 for recommending the school to a friend. For analyses for each of these 3 global assessment variables at each school, in 8 of the 9 comparisons, JHLES was more strongly correlated with the global assessment item than was DREEM.

JHLES total score showed better discrimination than the DREEM for identifying students who were most satisfied with their school’s learning environment, with areas-under-the-curve for JHLES and DREEM respectively being: 0.81 vs. 0.73 for overall learning environment quality, 0.82 vs. 0.79 for agreeing the educational experience is excellent, and 0.84 vs. 0.77 for wanting to recommend the school to a friend.

Discussion

In comparing learning environment assessments by medical students at the end of their first year in medical school, PUGSOM learners rated the learning environment highest on most measures, with most variables being judged more favorably than its American counterpart and a comparison Malaysian school. We also found JHLES scores to be strongly correlated with DREEM scores and our single-item global learning environment assessment variables. JHLES domains successfully detected significant differences between the learning environments across institutions.

Overall ratings of medical school learning environments by students in this study were more favorable than those seen in other studies. Most often, DREEM scores average between 100 and 130 [15, 20], and we were unable to find any published DREEM score from medical schools higher than 143 [21]. Previous studies using DREEM in Malaysia have reported DREEM total mean scores of 117–133 [20, 22, 23]. Most studies report learning environment ratings from senior students or aggregated ratings across student cohorts, but there is some evidence to suggest that first year students may have higher ratings than senior students [20, 24]. It is also known that schools in developed countries, where multiple teaching methods are used (e.g. PBL) and teaching is more interactive, have learning environment ratings that are more positive [15, 25]. It is likely that assessing first year students and more interactive pedagogy can explain in part why all 3 schools scored higher than historical assessments.

To our knowledge, this is the first study to formally assess a medical school learning environment created after a Western academic medical center begins operations in a foreign country. PUGSOM’s exceptional learning environment scores appear to be driven by positive impressions of the teaching methods and faculty and feelings of engagement, acceptance, and safety. Much of this may be related to PUGSOM’s small first year class size, which allows all 24 students to attend all non-clinical teaching sessions together, permitting ample opportunity for interaction with PUGSOM’s small group of core faculty.

Notably, while the curricular content, instructional formats, and even specific faculty were nearly the same at PUGSOM and JHUSOM, PUGSOM responded more positively than JHUSOM on every item in the DREEM “perception of teaching” category, many of which relate to active and student-centered approaches to teaching. Nearly every PUGSOM student completed their pre-medical education in Malaysia, where learning is more teacher-centric and passive, whereas Western teaching styles emphasize learner autonomy and involvement through discussions. PUGSOM’s higher ratings for such elements may therefore represent a greater appreciation for these methods than that of JHUSOM students, who are accustomed to Western teaching styles from their pre-medical education. While this effect may also be related to small class size, the facts that JHUSOM’s students rated their advising higher than PUGSOM’s and the lack of differences in both JHLES’ “community of peers” domain and DREEM’s “social self-perception” category suggest that small class size is not the only factor at play.

The strong correlation between DREEM and JHLES suggests that both surveys may measure the same overall construct, the learning environment. Because many of the individual items from these scales address different areas, one could conjecture that these 2 scales may be complementary. For example DREEM’s questions in the “perception of teaching” and “perception of teachers” categories do not have obvious analogues in the JHLES, primarily because such questions were not found to be informative or were eliminated by factor analysis during JHLES development at JHUSOM. DREEM, on the other hand, does not contain questions that capture student perceptions of their level of engagement, mentorship, or impressions of physical space, which were found to be relevant at JHUSOM and appeared to be important in this study.

JHLES appears to be better able to tease out more differences in perceptions about the learning environment than DREEM. A greater proportion of JHLES domains than DREEM categories revealed differences, and JHLES was more strongly correlated with the 3 distinct global assessments of medical school learning environments. While one could argue that it may be ideal to administer both DREEM and JHLES, the combined 78-item questionnaire this would require may be burdensome. Moreover, a recent confirmatory factor analysis performed on the DREEM, at one medical school in Malaysia [26], suggested 33 DREEM items do not add meaningfully to its learning environment assessment. That the JHLES could discriminate among different factors in 28 items, suggests it may be the preferable instrument, provided it proves to be valid during confirmatory factor analysis on international populations.

Several limitations of this study should be considered. First, PUGSOM was a new school and Malaysia’s only graduate-entry medical school. It had created a new pathway for students to attend medical school, including those who may not have qualified for the traditional M.B.B.S. pathway. Students’ ratings may therefore be influenced by an appreciation for having the opportunity to attend medical school. While we were unable to adjust for this, one may expect that after enduring the rigor of the first year of medical school, the impact of the gratitude as a source of bias might begin to wane. Second, the small class size and the high faculty-student ratios at PUGSOM may play a role in their learning environment perception, although these students did not rate mentoring relationships higher than students at the other schools. Finally, we were not able to collect student grades or other objective educational outcomes, and thus it remains unknown whether learning environment perceptions correspond to meaningful academic success or more empathic patient care.

Conclusions

Ultimately, this study provides the first empiric evidence that a positive learning environment can be established when a curriculum and teaching style are implemented in a foreign country and within a different culture. Our findings suggest that the learning environment at the international school may even be more positive than its American partner and that of a domestic institution. In future work, it will be necessary to assess changes in the learning environment over time, to examine the factors most responsible for differences perceived in the learning environments, and to ensure that positive learning environments lead to improved professional development and clinical competence.