A web-based simulation of a longitudinal clinic used in a 4-week ambulatory rotation: a cohort study
© Wong and Lochnan. 2009
Received: 09 September 2008
Accepted: 02 February 2009
Published: 02 February 2009
Skip to main content
© Wong and Lochnan. 2009
Received: 09 September 2008
Accepted: 02 February 2009
Published: 02 February 2009
Residency training takes place primarily on inpatient wards. In the absence of a resident continuity clinic, internal medicine residents rely on block rotations to learn about continuity of care. Alternate methods to introduce continuity of care are needed.
A web-based tool, Continuity of Care Online Simulations (COCOS), was designed for use in a one-month, postgraduate clinical rotation in endocrinology. It is an interactive tool that simulates the continuing care of any patient with a chronic endocrine disease. Twenty-three residents in internal medicine participated in a study to investigate the effects of using COCOS during a clinical rotation in endocrinology on pre-post knowledge test scores and self-assessment of confidence.
Compared to residents who did the rotation alone, residents who used COCOS during the rotation had significantly higher improvements in test scores (% increase in pre-post test scores +21.6 [standard deviation, SD, 8.0] vs. +5.9 [SD 6.8]; p < .001). Test score improvements were most pronounced for less commonly seen conditions. There were no significant differences in changes in confidence. Residents rated COCOS very highly, recommending its use as a standard part of the rotation and throughout residency.
A stand-alone web-based tool can be incorporated into an existing clinical rotation to help residents learn about continuity of care. It has the most potential to teach residents about topics that are less commonly seen during a clinical rotation. The adaptable, web-based format allows the creation of cases for most chronic medical conditions.
A shift of patient care from the inpatient to outpatient setting has taken place. Ideally, postgraduate training programs should ensure the curriculum matches this balance, yet internal medicine training programs in Canada continue to be dominated by clinical rotations on inpatient wards. Many obstacles exist in postgraduate medical education that limit the establishment of the most optimal experiences. First, block-format ambulatory rotations are often relied upon as settings to introduce continuity of care issues. Traditionally, ambulatory care rotations have been four – to six-week blocks of five to seven half-day clinics per week. Assigning residents to continuity or longitudinal clinics allow them to see patients over time, presenting opportunities to learn how to provide effective continuity of care. However, organizing high quality continuity experiences is often difficult especially when resident training is based at multiple sites. Second, residents are required to complete rotations in most, if not all, medical subspecialties; consequently the time spent per rotation is limited to one or two-month blocks which provide superficial exposure to clinical problems, and offer little continuity of care . Third, resident learning in the ambulatory setting is dictated largely by chance; the stimulus for patient-centered teaching and learning is determined by how patients are scheduled clinics rather than a standardized method. Faculty often ask residents to see "new patients" as opposed to patients booked for follow-up care, which may reduce the impetus to focus their learning about continuity of care. The types of medical problems that each resident encounters cannot be standardized.
Continuity of care is one of the expressed goals of an ambulatory care experience. Longitudinal or continuity experiences are characterized by at least one half-day per week of ambulatory care experience in the same location with either a single or multiple preceptors. The benefits of longitudinal clinics include providing opportunities to follow patients over time to develop ongoing patient-physician relationships and learn about the unique challenges of managing chronic disease [2, 3]. Due to the complexity of organizing continuity experiences for large numbers of residents most internal medicine training programs in Canada rely on block rotations to provide ambulatory experiences to its residents. Residents miss out on opportunities to learn about clinical issues that are specific to seeing patients longitudinally (for example, knowing how to adjust treatment, manage adverse events etc.) Alternate methods to enhance exposure to continuity of care issues within the constraints of existing residency training programs are needed.
Web-based teaching allows universal accessibility of the content, consistently delivers the same message to every learner and gives control of learning to the student without distracting them from their expected clinical responsibilities [4–8]. In the context of residency training programs that do not offer continuity clinics, the role of web-based simulations continuity experiences has not been defined. Numerous published studies evaluating the use of web-based teaching at both the undergraduate and postgraduate levels have noted increased learner satisfaction as well as improvements in pre-post test scores. However we were not able to identify any studies of the effect of web-based simulations of continuity of care as an adjunct to a block-style clinical rotation at the postgraduate level. The most applicable study compared web-based to traditional learning formats within an internal medicine resident continuity clinic. Residents preferred the web-based learning modules but no difference in pre- to postintervention test scores were noted . However, web-based learning was evaluated as an adjunct to a preexisting continuity experience, and the results may not be applicable to programs without this experience. Another pertinent study showed that the use of a 'stand-alone' online curriculum in ambulatory care was rated favorably by residents and program directors, and significantly improved test scores . In this study the degree to which the curriculum content focused on medical issues that arise during continuing or follow-up care was not explicitly mentioned.
In the absence of published reports demonstrating the use of web-based technology to simulate what occurs in a continuity experience, we designed Continuity of Care Online Simulations (COCOS), an online learning tool that simulates the longitudinal care of patients with chronic diseases. We report our experience in developing this web-based learning tool and its use by internal medicine residents whose only ambulatory experiences consist of short 4 to 8 week blocks rather than continuity or longitudinal experiences. We determined its effects (as an adjunct to a 4-week ambulatory rotation) on resident knowledge and confidence in providing continuing care of patients in the ambulatory setting.
Ambulatory care objectives used to structure COCOS
1. Identify pertinent features on history and physical examination
2. Describe diagnostic tests used to evaluate a patient, including their cost-effectiveness
3. List the differential diagnosis for a patient based on his/her clinical presentation
1. Classify a patient's current state in the natural history of a chronic condition
2. Describe, to a patient, the expected course of their condition without treatment
3. Order tests, if necessary, to monitor the course of a chronic condition
1. Describe, to a patient, and compare treatment options including: mechanism of action; benefits, including efficacy and time course; risks, including adverse effects
2. Order tests, if necessary, to monitor treatment efficacy and/or adverse effects
3. Analyze data to determine changes in treatment that is required
4. Discuss an appropriate duration of treatment for a chronic condition
5. Detect if treatment has failed, and select alternate treatment options
1. Provide prepregnancy counseling to a woman with a chronic condition, including: how her condition will affect fertility; how or if her condition will change during pregnancy; how the condition will affect her during pregnancy; how the condition will affect the fetus during pregnancy; the likelihood her baby will develop the same condition, management or treatment that should take place prior to conception
2. Formulate a management plan for a pregnant woman with a chronic medical condition
3. Manage a patient with a chronic medical problem during the peripartum period
4. Formulate a management plan for a woman with a chronic medical condition during the immediate and later post-partum period
1. Discuss the criteria by which requests for consultation should be prioritized as urgent
2. Discuss an appropriate time frame to arrange follow-up care in the case of: a patient in whom the diagnosis has not been made; a patient in whom you've initiated therapy; a patient on therapy who has not been stabilized; a patient on therapy who has been stabilized
3. Discuss factors that influence whether a patient can be followed his/her primary care physician for a medical condition
4. Describe key recommendations that should be made to a primary care physician who will be following a patient for a medical condition
Using the generic objectives as its foundation, we created a storyboard depicting sequential clinical appointments for a patient with a chronic medical condition. The storyboard was used as a template for all cases to ensure that each of the objectives (table 1) was discussed. Case authors were only required to add disease-specific information to the generic template to create a case. Case studies were created for: type 1 diabetes (commonly seen), Graves disease (moderately seen), hyperprolactinemia (uncommonly seen). These topics were specifically selected based on the frequency in which past residents reported seeing patients with those conditions during a typical 4-week endocrinology rotation. Each case averaged 7500 words and was reviewed by at least two endocrinologists to establish content validity.
A budget of $5000 Canadian was used to create the website for use in the pilot study. Only an Internet browser was required to utilize the website which was accessible using pre-assigned usernames and passwords. Each page of a case depicts a simulated appointment with a patient and the resident is asked to make one or more clinical decisions and/or answer questions posed by the simulated patient. Immediate feedback is shown in a pop-up message window or on the subsequent page depicting the subsequent clinic appointment. Residents could complete self-assessment quizzes before and after the case study.
Based on our ambulatory care curriculum, we developed short answer questions addressing the objectives for each disease. Questions were in the context of patient scenarios viewed from the perspective of a general internist. A panel of medical educators established content validity of the tests. Construct validity was established by the administration of the test to full-time division members in endocrinology. The final version of the pre- and post-tests contained an equal distribution of questions pertaining to topics specific to seeing a patient during an initial consultation, and topics relevant to seeing a patient in follow-up or during continuing care. Questions were also equally distributed by condition (type 1 diabetes, type 2 diabetes, hyperthyroidism, hyperprolactinemia, polycystic ovarian syndrome). To control for different levels of baseline knowledge among residents, we measured the difference between pre-rotation and post-rotation scores as the outcome variable.
Residents self-rated their confidence in managing patients with endocrine and other medical conditions on a five-point Likert scale. Questions were subdivided into ratings for two types of patient encounters: "initial consultation" and "follow-up appointment". To control for the different level of baseline confidence among residents, we measured the difference between pre-rotation and post-rotation scores as the outcome variable.
Residents were asked to indicate how often they saw a patient with each condition during their 4-week rotation. This allowed us to control for the amount and type of clinical content encountered during the rotation when analyzing the results of our outcomes, and allowed us to determine if there was a differential effect of COCOS depending on the type and commonality of a clinical topic. We did not review clinic records to measure the exact numbers of patients seen by each resident. Members of the intervention group were also asked to rate their level of agreement with statements about COCOS on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree) with respect to its contribution to their learning.
P values were calculated between pre and posttest results of the matched test results within groups using Students' t test (paired analysis), and between groups using a Student's t test (unpaired analysis). Median rankings of self-confidence pre- and post-rotation were calculated; to test for differences in the median values between the start and end of rotation, and between groups, a non-parametric one-way analysis of variance statistic was used. Survey Likert scores were analyzed using the Wilcoxon signed-ranked test with the median of the scale (three) as a reference. All analyses were performed using a two-sided alpha level of .05. To detect an improvement of knowledge score of 20 percentage points with an estimated standard deviation of 20 points, a 2-tailed alpha error of 0.05 and a power of 0.80, sample size was calculated at a minimum of 16 resident participants.
Characteristics of participating residents
Control (n = 10)
Intervention (n = 13)
Average age (years)
Bachelor of Science
Bachelor of Education
Level of training:
Prior clinical rotations in endocrinology
2 months or more
General internal medicine
Other medical subspecialty
Opinions regarding need for emphasis on continuity of care in residency:
More emphasis required
No change required
Less emphasis required
Self-reported use of internet
Full knowledge of internet use
Daily use of internet
Median estimates of the number of patients seen during the endocrine rotation:
Type 2 diabetes
Type 1 diabetes
Thyroid nodules and/or goiter
Polycystic ovarian syndrome
Cronbach's alpha for the test questions was .753 (pre-test) and .851 (post-test) suggesting acceptable reliability. There were no significant differences between the two groups' pre-rotation test scores. There was no difference in pre-rotation test scores between second and third year residents but residents who had completed one or more endocrinology rotations in the past (n = 8) had slightly higher pre-rotation scores (mean score 64.8% vs. 59.2%).
Mean total scores for pre-rotation and post-rotation tests (scores are expressed as a total score out of 100)
By type of patient encounter:
All COCOS cases†:
Type 1 diabetes
All non-COCOS cases††
Type 2 diabetes
Polycystic ovarian syndrome
Mean self-ratings of residents' confidence to manage different conditions (scores are expressed on a 5-pt Likert scale, 1 = no confidence, 5 = extremely confident).
Control (initial consultation)
Intervention (initial consultation)
Type 1 diabetes
Type 2 diabetes
Polycystic ovarian syndrome
Residents' level of agreement regarding the COCOS learning modules (1 = strongly disagree, 5 = strongly agree; * Wilcoxon Signed-rank test)
Mean Rating ± SD (p value)
The preparation material and instruction for using COCOS was adequate.
4.45 ± .52 (< .001)
The time available during this rotation was adequate to allow use of COCOS on my own time.
4.54 ± .52 (< .001)
COCOS allowed me to learn about continuity of care I would not otherwise have learned.
4.45 ± .69 (< .001)
COCOS provided me with more exposure to endocrinology I would not otherwise have had.
4.27 ± .79 (.003)
COCOS enhanced my self-directed learning.
4.54 ± .69 (< .001)
After this rotation, I plan to use the COCOS cases in endocrinology as a resource.
4.09 ± 1.04 (.010)
COCOS cases in more endocrinology topics would benefit residents' learning during this rotation.
4.54 ± .69 (< .001)
COCOS cases in other medical topics should be used on other rotations.
4.54 ± .52 (< .001)
COCOS increased my interest in endocrinology.
4.09 ± .94 (0.01)
COCOS should be included as a standard component of the endocrinology rotation.
4.36 ± .67 (.003)
We have described the successful implementation of an online resource used by residents in a one-month ambulatory experience that simulates the clinical decisions that needed to be made when seeing patients in a longitudinal clinic. Our use of simulations allowed the residents to practice patient care in a safe environment and provided immediate feedback on their patient-care decisions. The content and format was rated highly with the majority recommending its ongoing use not only in our rotation but also in others during their residency. Residents accessed COCOS on their own time without the need for direct involvement of faculty. It was not perceived to be labor or time intensive for the residents; each case required approximately 20 minutes to complete, consistent with other reports . Completing four to six cases in a four-week period appears to be an acceptable expectation of residents. Even though we applied the model to topics in endocrinology, its generic template facilitates writing cases in other medical subspecialties with a strong ambulatory component.
COCOS is unique in its emphasis on the competencies required to provide follow-up or continuing care to patients. To our knowledge this is the first study that has investigated the effect of web-based simulations of continuity of care as an adjunct to a clinical rotation at the postgraduate level. "Stand-alone" internet curricula in ambulatory have been rated highly by internal medicine residents and resulted in improved test scores [9, 11] but lack of control groups in these studies did not allow comparisons to 'traditional' teaching formats. Our use of a control group allowed us to make a comparison with our 'traditional' clinical rotation to help us determine whether it is useful as an adjunct rather than a replacement. Other interventions used at the postgraduate level in internal medicine ambulatory settings did not provide as much emphasis on continuity of care .
We did not measure the number of resident encounters with patients for each category (initial consult or follow-up appointment), but traditionally residents are asked to see new patients (i.e. initial consult) rather than patients returning to clinic. Students' learning in the clinical setting is more often opportunistic and thus more learning is likely to occur around topics that are common compared those that are less commonly seen. Even though the use of COCOS resulted in improved test scores for topics relevant to the "initial consult", a more pronounced improvement was for topics relevant to seeing patients during the "follow-up visit". Thus COCOS may be most beneficial to improve learning around types of patient encounters (i.e. follow-up appointments) that are less frequent during a short clinical rotation. Similarly, COCOS was more beneficial for diseases or conditions less commonly seen during the rotation. The use of COCOS cases for Graves disease and hyperprolactinemia resulted in a significant improvement in test scores, in contrast to a non-statistically significant improvement for questions pertaining to type 1 diabetes. Comments left by the residents included suggestions to create cases for other endocrine disorders uncommonly seen such as adrenal insufficiency and primary hyperparathyroidism. Thus the use of COCOS as an adjunct to a clinical rotation should emphasize more "follow-up" or "continuing care" for conditions like diabetes that are commonly seen, but include content for both "initial consult" and "follow-up" issues for uncommon conditions.
It is noteworthy that the use of COCOS resulted in greater test score improvements in topics for which there were no COCOS cases (thyroid nodules, PCOS). There are two possible explanations for this observation. Residents in the intervention group may have had better exam-taking abilities compared to those in the control group. Alternatively the use of COCOS results in qualitative changes in the learning process that may result in more efficient or improved learning, as has been noted in some studies outside of medical education [13, 14]. 91% of residents reported COCOS enhanced self-directed learning and our group of 6 full-time faculty members anecdotally reported residents asked more questions about follow-up care, lending support to this hypothesis. However for this to occur COCOS would have to be used early in the rotation and we did not determine when in the rotation it was used by each resident.
Our residents' confidence in managing different types of endocrine conditions is similar to what is reported in the literature . In contrast to the difference in knowledge gains with the use of COCOS, there were no significant differences between groups in the changes in confidence. Residents may overrate themselves at the start of the rotation or underrate themselves at the end of the rotation when compared with ratings of their supervisors [16, 17]. The latter may be especially true for residents who used COCOS if they perceived they had done poorly during the case simulations. A retrospective self-analysis of how residents thought their confidence changed over the course of the rotation would be insightful.
The use of COCOS during the rotation proved to be advantageous compared to the rotation alone that included the provision of printed material. One of the key reasons for conducting this study relates to how there is a lack of reading material that adequately covers many of the "continuity of care" learning objectives. The reading material and printed guidelines provided to the control group were primarily review articles that only briefly address the COCOS learning objectives. We did not provide the control group with a paper-based version of the COCOS content. We feel that COCOS has many features that provide advantages over a stand-alone, paper-based source of information. Simulated cases provide residents with opportunities to build on knowledge gained from their clinical experience thus promoting a deeper learning. While we did not include multimedia content for the purpose of this pilot study, images, sounds and video clips can be easily incorporated into any case to add realism to the simulations. The option to complete online pre- and post-case self-assessment quizzes captures their attention and capitalizes on the learner's motivation . The interactive format including the ability to record resident answers to questions, was adopted not only to better engage the learners , but also to assist course administrators identify gaps in knowledge and in the curriculum as part of a needs analysis.
Manually authoring unique case scenarios with potentially different sequences, content and variables, is an extremely labor-intensive undertaking that limits their availability or reproducibility . By using a generic template design for COCOS, disease-specific information was easily added to allow the creation of cases. In COCOS residents are asked to answer questions or make clinical decisions, and regardless of the type of response, the storyline continues in its preset course. A switch to a more probabilistic type of simulation where an action could result in a number of theoretic outcomes would improve the level of realism.
The text-based format of the cases allows for easy modification in light of changes in clinical evidence or to adjust the case sequence for disease in which certain topics may not be applicable. For example, the page devoted to "Medical management during pregnancy" could be omitted or interchanged with other special situations such as "Preparation of surgery" with relative ease. Opportunity exists for a wide application of COCOS beyond the postgraduate level, including both undergraduate and continuing education.
This study has several limitations. It was conducted at a single training program and the structure and administration of our rotation may not be representative. We only focused on "continuity of care" from the perspective of the CanMEDS role as a Medical Expert. There are elements of longitudinal clinical practice that are difficult to simulate (e.g. reviewing the chart, refreshing oneself on the patient's problems, retrieving missing information, understanding patient-doctor relationships, family dynamics and community resources)  that require competencies in other CanMEDS roles such as Manager, Communicator and Professional. Only hands-on experience may be adequate to teach our trainees about these elements. We did not randomize residents within each rotation to avoid contamination. We are somewhat reassured that the structure and administration of the rotation did not differ between the two groups, and the self-reports of the numbers of types of patients seen by the two groups were similar. Even though residents may have been enrolled at different times, there was no difference in baseline test or confidence scores. 50% of the control group and 31% of the intervention group were in their third-year of residency. If clinical experience influenced outcomes, one would anticipate that the group with more experience would perform better on the knowledge assessment. Alternatively, more senior residents may have better-established learning habits and thus may be less likely to benefit from an online intervention compared to more impressionable junior residents. Regardless, despite being at a relatively earlier stage of training, the intervention group had greater improvements in test scores; if randomization resulted in an equal distribution by postgraduate year, the differences between groups may have been more pronounced. Similarly, if participants were randomized and contamination had occurred, the control group's test scores could be artificially raised, and thus the results would still be significant if the intervention group demonstrated greater improvements in knowledge. Alternating the control with the intervention group from month to month is a potential way to improve validity and avoid contamination.
When interpreting the results, it is not known whether the improvements in test scores seen in the intervention group can be attributed to a computer-dependent feature in COCOS. Future studies comparing paper-based to a web-based presentation of COCOS content, or comparing two different versions of COCOS (to measure the impact of change in a single feature while holding others constant) would be useful. The net worth of COCOS on the learning gain, relative to the resource costs to develop and maintain it, remains unknown. We did not measure the amount of time residents spent completing the online modules nor did we survey residents on whether they felt the use of COCOS resulted in a more efficient use of time to learn about its specific topics, thereby allowing them to allocate the saved time for learning in other curriculum areas. Thus it is theoretically possible that the time allocated to the use of COCOS could have taken away from other learning opportunities. We also did not measure whether the implementation of COCOS resulted in staff physicians being able to save time and/or observe improvements in their other academic responsibilities. While resident opinions may not be the best measure of worth, the overwhelming response of residents who used COCOS was very positive and did allow them to better learn about topics that they would not have otherwise been able to do.
Forty-four percent of our total participants did not complete the post-test, and their pre-test scores were not included in analysis. We did not systematically measure reasons for non-completion. It was explicit that participation in the study was voluntary and we suspect that these residents gave less priority to completing the post-test in favor of other aspects of the rotation. This could have biased the results, in two ways. First, if these residents had a lower baseline level of knowledge, our actual results would have ultimately underestimated the changes in pre-post test scores. In contrast, if these residents had higher pre-test scores, our results would have overestimated the changes. However, when analyzing the pre-test scores of who did not complete the study, there was no difference between groups and we feel the dropout rate did not influence our interpretation of the data. Second, not completing the post-test could be an indicator of relatively weaker skills or level of interest in self-directed learning and thus, our final results would not be applicable to them. It is unclear as to why one resident assigned to the intervention group did not end up using the COCOS program, but there were no reports however of technical difficulties with accessing the website.
The objective was to determine the effect of COCOS as an adjunct learning tool within an existing clinical rotation, and we can not decipher whether the technology itself or the content (using technology as its vehicle for presentation) contributed most to the benefits. Specific studies are needed to determine whether different types of such presentation elements that we included (e.g. popup windows, interactivity) provide enough learning benefit to justify their higher costs. A comparison of the effects of case-based content and non-case-based content would be an informative comparison of two different instructional methods to present the topics in our curriculum. It would also be interesting to compare the effects of the use of COCOS in a short, block rotation to the longitudinal clinical experience. We are currently studying whether the knowledge gained from using COCOS leads to long term retention and improvements in clinical performance.
In summary, we have described and evaluated an innovative way of teaching continuity care objectives using a web-based module in endocrinology in which cases are presented through several episodes across virtual time. As an adjunct, not replacement, to clinical experiences it can significantly improve residents' learning of ambulatory care endocrinology. Cases can be written for any chronic medical condition using a generic template based on preset objectives. The most pronounced benefits are seen for topics that are uncommonly seen during their clinical rotation or issues that arise in follow-up care. Our results have significant relevance to postgraduate medical training, where traditionally short, block-style rotations limit exposure to the wide range of patient problems and logistic difficulties and demands for increasing faculty productivity challenge traditional teaching efforts. Interested parties may sample the website by directing their browser to http://www.med.uottawa.ca/endocrinologie/index.shtml and use the guest username : endocrino and password : cocos .
The authors would like to acknowledge the support of a Banting and Best Diabetes Centre Pilot Grant for Innovative Activities Related to Diabetes Education, Management and Care, Maryse Grignon of Pixelopia® Design (funded through a AMS/Wilson Student Fellowship from Associated Medical Services Incorporated) for design and development of the COCOS website, and Nadine Booth (funded through a University of Ottawa Division of Postgraduate Medical Education Fund for Educational Initiatives devoted to Residency Education) for her assistance in data collection.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.