Bmc Medical Education Student Evaluation of a Primary Care Clerkship: Quality Assurance and Identification of Potential for Improvement

Background: In Germany, like many other countries, general practice clerkships have only recently become mandatory during medical education. The biggest challenges for the organisation of such clerkships are achieving a minimum level of standardisation, and developing and maintaining a system of quality assurance. The aim of this study is to assess the instructional quality in teaching practices using a benchmark system.


Background
There has always been a gap between the number of patients seen in primary care and how this is reflected in the curriculum of medical schools. To overcome this shortcoming, mandatory primary care clerkships are, to date, a component of most curricula. In the United States and the United Kingdom, for example, many schools already deliver a significant proportion of their curricula in the community [1][2][3]. However, in Germany -like many other countries -general practice is still not fully established in most medical schools, and clerkships in primary care/general practice were not mandatory until recently. In early 2004, the 8 th amendment of the Federal Regulations for Medical Education in Germany instituted mandatory practical training in primary care [4]. Clerkships in primary care in German medical schools range from 1 to 3 weeks. A 2-week clerkship in primary care in the 5 th year was introduced in 1999 at the University of Göttingen Medical School.
The core curriculum for this clerkship focuses both on general principles of teaching and on specific tasks of general practice. Some of the essentials are: Some of these tasks also reflect learning objectives, e.g. recognising and treating upper respiratory tract infection or communications skills, which fall within the curricular responsibility of family medicine in our faculty.
Primary care practices are usually rather loosely attached to medical schools. The instructional quality of undergraduate medical education in the community setting has been described as a "black box" [5] where the quality of teaching in some practices and the availability of time to teach and offer patient care are matters of concern [1]. This is due to variation in practice structure as well as the large number of decentralised sites (compared to a hospital setting) and the high number of tutors. The biggest challenges are creating and maintaining a minimum level of standardisation and a system of quality assurance [6,7]. Clerkship students are expected to learn from tutors, not usually trained in teaching, core concepts of primary care, such as "whole person" and "humanised health care" [8], along with strategies for differential diagnosis, patient management and comprehensive care for chronic illness [9]. Training programmes for general practitioners (GPs) with a special interest in medical education have been established elsewhere [10] but are either not available in Germany or restricted to doctors who are highly motivated to take part in teach-the-teacher training courses [11,12].
Although teaching general practices seem to perform better, for example, in prevention activities or prescribing indicators than non-teaching practices [13], the quality of community-based education cannot be taken for granted [6]. As early as 1999, Shipengrover and James [5] complained about a lack of valid measures to monitor the quality of teaching during clerkships. Meanwhile, there are some measures to survey the students' most valued practice and preceptor characteristics and to evaluate instructional quality in ambulatory teaching sites [6,14,15]. However, these measures lack a focus on characteristics unique to primary care.
A student assessment may provide exactly such data continuously, namely process data, which is needed to evaluate the quality of medical education. We therefore introduced a mandatory evaluation with a special focus on primary care characteristics that could form the basis of a subsequent benchmarking system. The aims of this study were to explore: (1) Which expectations or preconceptions students had of a primary care clerkship.
(2) Which areas of teaching students did and did not appreciate to identify GPs' areas of strength and weakness in teaching.
(3) Correlations between the students' evaluation of the clerkship and their desire to pursue a career in primary care.
(4) Whether the data helped to identify practices with excellent or only moderate teaching performance.

Teaching physicians/preceptors
Most primary care in Germany is delivered by GPs in private practices. The recruitment of teaching GPs started in 1999 from all GPs and some internal medicine primary care providers in the district surrounding the Göttingen University Medical School. All general practitioners in Germany have a board certified specialty training, which requires 3 to 5 years postgraduate training. They can apply for their practice to become a teaching site and become a member of the general practice teaching network if they agree to comply with organisational requirements as well as the learning content. These pre-requisites are displayed on the department's homepage [16]. The most important requirements are at least 3 years in practice, a second consultation room allowing students to see patients by themselves, and a focus in regular primary care. Practices providing for example mainly naturopathic medicine are excluded.
The GPs are kept informed of current requirements by email and an annual pamphlet. So far, we do not require teaching GPs to attend a mandatory course in medical education. GPs continue to see patients while teaching and they receive 25 € per day of student supervision to cover for the extra time students require to discuss patient care and examine patients Additionally the practice receives the title "academic teaching practice".
Compared with the demographic data of the doctors in the district of Göttingen, Lower Saxony, the average age of our teaching GPs did not differ significantly (51.3 vs. 52.3 years); the proportion of women was slightly smaller (24.1% vs. 20.4%). Nearly 60% of all teaching GPs were working single-handed and about 40% in group practices [17].

Medical students
In Germany, medical school training requires 6 years (2 years of basic science and four years of clinical coursework). Students are required to select a practice from the teaching network for a 2-week clerkship. It is recommended that the clerkship be taken at the beginning of the 5 th year. However, students are allowed to do it earlier if, for example, they plan a year abroad. A tutorial with casebased learning using the experience of students gained during the clerkship takes place at the end of the 5 th year.

Data collection
In April 2003, a mandatory web-based questionnaire was introduced both prior to (pre-evaluation) and after the clerkship (post-evaluation). This is a self-developed questionnaire based on the learning objectives of the core curriculum. The evaluation questionnaires, including a test version, can be accessed via the departmental website [18]. Students identify themselves with their student identification number over a secure connection. This number is used to match the pre-and post-questionnaires. Only completed questionnaires can be returned via the Internet. Personal data is stored separately from the questionnaire. The technical details have been described elsewhere [19].
The questionnaires allow the students to rate the importance of key aspects of the core curriculum of the primary care clerkship (pre-evaluation) as well as an assessment of the teaching practice and the tutor's performance (postevaluation). Relevant items can be seen in Table 1. Students are also asked after their clerkship if they feel encouraged to pursue a career in primary care.

Data analysis
We analysed all questionnaires that were returned via the To rank practices according to the students' evaluations, we assessed the proportion of items rated as excellent or good per practice. We only ranked practices that had been evaluated by at least 5 students. To determine the internal coherence of the post-evaluation form, we calculated Cronbach's alpha for each practice. The Wilcoxon rank test was used to compare the students' overall rating of the clerkship quality depending on whether they felt encour-  of medicine  43  50  7  22  51  20  7  preventive medicine  46  46  8  24  41  23  12  screening  48  45  7  28  36  23  13  vaccinations  25  52  23  31  37  21  11  family medicine  25  54  21  32  40  17  11  home visits  29  49  22  45  35  13  7  communication skills  65  30  5  50  37  10  3  patient expectations  31  59  10  35  49  13  3  caring for chronic patients  41  51  8  33  46  16  5  physical examination  61  30  9  29  44  18  9  collaboration with specialists  26  54  20  17  40  28  15 *based on n = 695 students aged to pursue a career in primary care or not. The software package SAS 9.1 was used for analysis.

Results
From a network of 125 teaching practices, 97 GPs had supervised at least one student during the study period (mean per practice: 7.2; median: 5; range: 1 to 35). Fiftyone practices had at least 5 students. At the time of the data analysis, a total of 695 students (44% female) had evaluated the primary care clerkship via the Internet. We had no missing data since a complete evaluation is a prerequisite for receipt of the clerkship certificate. Table 1 shows how the students assessed the importance of various aspects of the core curriculum before the clerkship and how well these same aspects had been addressed after the clerkship. "Recognition of frequent diseases" was considered the most important characteristic. "Physical examination" and "communication skills" were considered important by many students before the clerkship and received good scores in the evaluation after the clerkship. "Vaccinations", "family medicine", "collaboration with specialists" were considered to be less important and received a less positive evaluation, whereas "home visits", also not considered to be important by many students, was rated excellent. Table 2 shows how the students judged the tutors' didactic performance. Most students were (highly) satisfied with the total quality of the clerkship (mean score: 8.1 on a 10point-scale). They felt the doctors spent enough time (mean score: 8.3) and the demands were adequate (8.9). However, in the opinion of the students, the strategies adopted for "problem solving" in primary care were less well taught (7.1). Figure 1 shows the opportunities for students to learn and gain practical experience during the clerkship. Most of them had the opportunity to perform one or several independent physical examinations during the clerkship, but in more than half of the clerkships, the students were either not allowed or did not feel encouraged to take a history without the presence of their tutor.
Some GPs received excellent scores in all or most of the performance indicators. In a sample analysis, we identified the two best and the two worst practices (Table 3). Seventeen students did their clerkship in the two top practices. Of theses students, on average 94%, respectively 93% rated all teaching items as either good or excellent. For 8, respectively 7 items, all students in one practice agreed that the teaching was good or excellent. In contrast, on average only 31%, respectively 54% of the students who did their clerkships in the two worst practices found the teaching to be either good or excellent. Two items of practice 43 -one of the worst practices -were rated as good or excellent by 70% and 80% of the students. This percentage slightly exceeded the average rate for all practices. However, most items were rated far below average for both of these "bottom practices" and no items of the core curriculum were unanimously found to be good or excellent. Evaluation of the different students in one practice was consistent for most performance characteristics, with correlations (Cronbach's alpha) for all 51 practices, which were evaluated by at least 5 students, ranging from 0.75 to 0.94, with an average of 0.91.
Students who had a good experience during the clerkship and rated practices favourably were significantly more likely to consider pursuing training in primary care. These students (n = 374) rated the quality of the clerkship with an average score of 8.8 (CI 95 8.6-9) compared to an average score of 7.2 (CI 95 6.9-7.5) given by the remainder (n = 321).

Findings
Most students considered recognition of frequent diseases, communication skills and physical examination very important learning goals before starting their primary care clerkship. According to the students' assessment, teaching GPs did a good job conveying these central goals, but were less successful, for example, in conveying the importance of preventive medicine, screening or collaboration with specialists. With the exception of independent history taking, most students reported that they had had sufficient learning opportunities. On average, students Learning opportunities for students Figure 1 Learning opportunities for students. Yes: Grey shading. No: Black shading. were satisfied with the teaching performance, except when it came to problem solving strategies. However, we were able use this benchmarking system to identify teaching physicians who consistently received ratings far below average.

Strengths and limitations of the study
Our results are based on large samples of students and teaching GPs and may therefore be applicable to all academic teaching practices in Germany. We believe that the establishment of a benchmark system to evaluate teaching performance in ambulatory care is of interest outside Germany.
All student evaluations are based on the hypothesis that students are the best experts to assess their teachers [20]. Nevertheless; we did not study the adequacy of the assessments made by our medical undergraduates. We also appreciate that while students might be reluctant to award unfavourable ratings after two weeks of close contact, about 10% of the GP tutors did receive poor evaluations for their teaching performance. However, to be protected against highly subjective and undue opinions, we considered in our benchmark scenario only practice ratings from at least five students. Standardised medical student evaluations could represent an alternative to assess the quality of GP teaching [21] but this approach would have required more substantial funding.
It is unclear how many students are needed to obtain a valid assessment of a teaching physician; we arbitrarily used a cut off of five students [22]. This number might be too low since seasonal factors, the availability of patients and interpersonal factors could possibly influence the quality of teaching and thus the students' evaluations. Even so, the scores given by students who did their clerkship in the same practice did not vary greatly, so that the identification of poor practices does not seem to be the result of an arbitrary or random assessment.
The strong association between practice assessment and anticipated careers in primary care can, of course, not be interpreted as a strict causal path. Some of the students considering primary care as a career might have been positively biased towards the primary care clerkship whereas others interested in a different specialty might have felt bored by any experience in primary care, independent of their tutor's teaching performance. In a Slovenian longterm evaluation, high scores in student satisfaction with family medicine did not influence the career choice [23]. The association between practice assessment and career aspirations found in our study is in line with a recent systematic review of the positive outcomes of early exposure to clinical and community settings on career choices in primary care specialties [24].

Context
According to a US study [6] and a Canadian study [25], the following attributes are essential for educational quality in ambulatory teaching and valued by students: open communication with the teacher, feedback, integration of the student into the clinical setting, explaining clinical reasoning, improvement of communication skills and giving students an active role in patient care. Most of these attributes were also highly valued by our students, especially communication skills (considered important or very important by 95% of our sample).
Compared to the other measures of the students' most valued practice and teacher characteristics [14,26], our questionnaire also addressed core aspects of primary care. More than 20% of the medical undergraduates in our sample did not consider the following key elements of primary care and family medicine [27] to be important: vaccinations, family medicine, home visits and collaboration with specialists. Additionally, nearly two-thirds of the students did not have any chance to take "patient history independently" during the clerkship. It is exactly this opportunity that has been found to be an important factor for students' enthusiasm in the above-mentioned Canadian study [25].
Summarising the literature, Howe [2] names several important consequences of learning in a primary care setting, e.g. retention of patient-oriented values more effectively than in previous curricula and enabling early application of theory into clinical practice. However, without continuous evaluations, we cannot be sure as to whether these beneficial effects really occur in all primary care clerkships. Our methodology may therefore represent a fair and valid benchmarking system that identifies poor teaching practices.

Implications for practice and research
We need to improve teaching in some core items, namely preventive medicine, screening and problem solving strategies and to increase opportunities for independent history taking during the primary care clerkship. Qualitative studies may also help us to better understand why many students rate "family medicine" rather low. Once these reasons are uncovered, we will then be able to better encourage students to consider a career in primary care.
Since feedback and evaluation from students are usually highly valued by community-based preceptors [28], teaching GPs should receive a standardised feedback comparing their performance with the overall performance of all practices on each evaluated topic. According to the experience of a student feedback study from Turkey [29], students' ratings should be reported more than once a year to have impact on the instructors' teaching perform-ance. For identified areas of weakness, the distribution of a brief faculty development audiotape, as suggested by Willett [30] may be helpful, especially for a dispersed group of ambulatory preceptors who find it difficult to attend seminars. Practices that consistently receive excellent evaluations should receive public recognition.
To our knowledge, there is no agreed standard defining what proportion of dissatisfied students is acceptable. If more than 30% of students rate an item of the core curriculum as sufficient or poor, we assume that this is a call for action. We also believe that students' assessments (with coefficients of reliability between 0.75 and 0.94) are consistent enough to rank practices according to their teaching performance. Both assumptions should be validated by further research.
An important and challenging issue for further studies is how to address those practices that perform below average. To our knowledge, nothing has been published on this delicate issue. Current attempts focus typically on programmatic evaluation rather than improving quality through validated benchmarks. To study the consequences, if any, of practice benchmarking combined with a feedback process, could help continuously monitor and improve faculty development [31].

Conclusion
Benchmarking teaching practices based on student evaluations may help to identify areas of weakness in teaching and deficient teaching practices. These results could be used as a basis for tailored educational interventions and may improve the quality of primary care clerkships.
Publish with Bio Med Central and every scientist can read your work free of charge