This article has Open Peer Review reports available.
A mixed methods approach to developing and evaluating oncology trainee education around minimization of adverse events and improved patient quality and safety
© Janssen et al. 2016
Received: 27 November 2015
Accepted: 4 March 2016
Published: 12 March 2016
Adverse events are a significant quality and safety issue in the hospital setting due to their direct impact on patients. Additionally, such events are often handled by junior doctors due to their direct involvement with patients. As such, it is important for health care organizations to prioritize education and training for junior doctors on identifying adverse events and handling them when they occur. The Cancer Cup Challenge is an educational program focuses on quality improvement and adverse event awareness targeting for junior oncology doctors across three international sites.
A mixed methodology was used to develop and evaluate the program. The Qstream spaced learning platform was used to disseminate information to participants, as it has been demonstrated to impact on both knowledge and behavior. Eight short case based scenarios with expert feedback were developed by a multidisciplinary advisory committee containing representatives from the international sites. At the conclusion of the course impact on participant knowledge was evaluated using analysis of the metrics collected by the Qstream platform. Additionally, an online survey and semi-structured interviews were used to evaluate engagement and perceived value by participants.
A total of 35 junior doctors registered to undertake the Qstream program, with 31 (88.57 %) successfully completing it. Analysis of the Qstream metrics revealed 76.57 % of cases were answered correctly on first attempt. The post-program survey received 17 responses, with 76.47 % indicating cases for the course were interesting and 82.35 % feeling cases were relevant. Finally, 14 participants consented to participate in semi-structured interviews about the program, with feedback towards the course being generally very positive.
Our study demonstrates that an online game is well accepted by junior doctors as a method to increase their quality improvement awareness. Developing effective and sustainable training for doctors is important to ensure positive patient outcomes are maintained in the hospital setting. This is particularly important for junior doctors as they are working closely with patients and learning skills and behaviors, which will influence their practice throughout their careers.
The rate of adverse event reports in hospitals globally has remained consistent over the last 20 years [1, 2], which is particularly disappointing given their direct impact on patient quality care. Adverse events are usually defined as incidents that result in harm to a patient–for example, either a short-term or permanent disability or in extreme cases patient death. Not only do adverse events have a severe impact on the patient, but the literature suggests that as many as one third of these events are preventable . Hospital-based adverse events cover a wide range or of incidents, many of which can be classified as due to either systemic or human factors [4, 5], and negatively impact on patient quality of care.
Doctors in training represent a group of health professionals that maybe particularly vulnerable to involvement in adverse events due to their high patient contact, workload and relative inexperience . This is evident in a number of studies that have reported on a phenomena referred to as the ‘July Effect’ where there have been reported increases in adverse events early in the academic training year .
Effective education and rigorous training has been recognized as an important component of developing safe and efficient health professionals. This is reflected by organizations such as the Accreditation Council for Graduate Medical Education (ACGME) mandating that residency programs provide education for residents and fellows in safety and quality . Quality improvement has emerged as an increasingly important focus of activity in health services and it is essential that clinicians understand the effectiveness of such interventions . In addition, current international training curricula include key competencies in safety and quality . This highlights the need for guidance on how to design and research quality improvement and safety programs. In spite of this focus, the delivery of effective education to health professionals in training remains a key challenge and under-researched area.
Physicians in training are required to demonstrate skills and knowledge across an increasingly large number of competencies while developing skills in their chosen discipline [11, 12]. Even though adult learning principles have long identified that education and training must be contextually relevant and directly linked to practice , many safety and quality programs in hospitals are delivered via didactic lectures or workshops that are not specific to the training environment or practice . While online learning is increasingly employed for the scalable delivery of education to doctors in training, there is little evidence in the literature regarding whether computer-based online educational methodologies are effective in actually changing behavior. Furthermore, if online programs are to include education and training in safety and quality to meet mandatory training requirements then it is essential that researchers can demonstrate evidence of their impact.
Qstream is a novel, evidence-based form of online education that has been demonstrated to improve knowledge acquisition and recall . It has also been repeatedly demonstrated to change participant behavior and maintain that change over time [16, 17]. Participants in Qstream courses receive repeating, short, case-based multiple choice questions with feedback via email in a reinforcing pattern over a number of weeks. The methodology is based on two core psychological research findings: the spacing and testing effects. The spacing effect refers to the finding that educational encounters that are repeated over time increase the acquisition and retention of knowledge. The testing effect refers to the finding that the process of testing does not only measure knowledge, but also improves retention [18, 19]. It has further been demonstrated that participants find repetition of cases via email effective for completing the course due to the ease of completing cases in existing schedules .
In a randomized trial this Qstream platform was found to increase learning efficiency by over 35 % over a non Qstream program with identical content. In addition, Qstream includes a gamification element whereby participants can compete as both individuals and teams and review their progress against de-identified leader boards . Gamification refers to the use of individual game mechanics such as leader boards in order to enhance another online tool such as an online course. The use of gamification can be beneficial in online education as it has the ability to both engage and motivate learners, strengthen social ties and preserve the original focus of the learning activity . The use of gamification elements in e-learning has also been demonstrated to result in more accurate results on quizzes due to the use of competition, which stimulates learners to interact more with their peers and the content of the course .
This study evaluated the feasibility and acceptability of using Qstream to deliver safety and quality education in an oncology-specific context to doctors in training in the United States, Australia and Denmark. In particular, the program focused on frequently encountered urgent clinical scenarios, adverse events from cancer treatments, and event reporting of errors and near misses to demonstrate key learning points. The study also evaluated the impact of competition in motivating participation through promoting the program as a friendly competition between the countries titled the ‘Cancer Cup Challenge’.
The aim of the Cancer Cup Challenge was to develop a program targeting key safety and quality issues that are encountered by physicians training in medical oncology. An expert Advisory Committee was convened in March 2014 to oversee the development of the program. This Committee consisted of senior medical oncologists, quality improvement staff, educational designers, senior oncology registrars and research fellows across participating sites.
The program was developed by representatives from international sites in Australia, Denmark and The United States of America (USA). This included individuals from the Sydney West Translational Cancer Research Centre in Sydney, Australia; Aarhus University Hospital in Denmark; and the Partners Program in Boston, the United States of America.
Once the take home messages had been identified, the Advisory Committee worked collaboratively to write short clinical case scenarios that reinforced the specific messages. The correct answer was identified and input as a multiple choice response. This was augmented by three distractor options which could be equally plausible to a participant. The final stage of building a case was developing the expert feedback. This expanded on the take home message in more detail and provided additional resources for further reading.
Junior oncology trainees were recruited to participate in the study from the Basic Science in Oncology Course (BSOC) in Australia, Aarhus University Hospital in Denmark and the Partners Program in the United States of America in October 2014. The research team was provided with contact details for a recruitment coordinator at each site. This coordinator was provided with generic text to create an expression of interest email for participants at their site. This was emailed to all medical oncology trainees at the site. Once recruitment had been completed participants were allocated to teams based on their country of origin and then were emailed a link to access the Qstream spaced learning system on which the Qstream: Cancer Cup Challenge was running by a member of the research team. The research team did not give participants access to any patient safety or dverse events training prior to undertaking the Qstream: Cancer Cup Challenge.
Each participant received an email every 2 days containing at least 2 cases
If they answered a case incorrectly, it was re-sent 5 days later
Once a case was answered correctly it was retired
The course was completed once all questions had been retired.
Each Qstream case consists of an evaluative component (a clinically-relevant multiple-choice question) and an educational component (the correct answer and a detailed explanation of the answer). Participants submit an answer, receive immediate feedback, and compare their performance with peers. To harness the educational benefits of the spacing effect, the case is repeated until the participant answers the case correctly. The decision to repeat only incorrect cases was made based on data from previous Qstream programs at the clinical sites. This data indicated that repeating correct cases increased the likelihood participants would not complete the program.
During the Cancer Cup Challenge participants were allocated to teams to evaluate the impact of the gamification element. Upon answering a question, participants were presented with a de-identified league table that indicated how they were performing compared to their peers. Participants were sent periodic emails containing team scores.
Mixed methods were used to evaluate the impact of the Cancer Cup Challenge on participant knowledge, skills and team engagement. Quantitative data collected by the Qstream system was analyzed for participant response accuracy. Qualitative data was collected through semi-structured interview and online survey to assess participant experience of the program regarding content and format. Qualitative data from interviews was transcribed and thematically analyzed.
Upon completion of the Cancer Cup Challenge participants were emailed a link to the online survey. This survey asked them to rank aspects of the course format and course content using a Likert ranking scale. They were also given the opportunity to leave free-text comments on their views of the course. A reminder to complete the survey was emailed out 1 week after the initial survey link was disseminated.
Participants were also invited to participate in a brief interview within 4 weeks of completing the course to explore the perceived impact of the Qstream education program on their knowledge and behavior in relation to adverse events in cancer care. Interviews were conducted over the phone or in person and took between 10 and 20 min to complete. The interviews were semi-structured without set questions. However, an interview guide was developed to maintain a consistent structure for all the interviews. Interviews first explored registrar and fellow needs in regards to adverse events training and how the Qstream program engaged participants and impacted on their educational experience.
Permission to conduct this study has been received from the Western Sydney Local Health District Human Research and Ethics Committee.
Participation and case performance
Details of the number of participants who enrolled in the Qstream: Cancer Cup Challenge compared to the number who completed
The United States of America
Demographic breakdown of the number of participants enrolled in the Qstream: Cancer Cup Challenge based on nationality
The United States of America
Thematic overview of cases compared to percentage of participants who answered correctly on first attempt
Case take home message
Total % correct responses on first attempt combined
Average number of attempts required to retire case
Minimum number of attempts to retire case
Maximum number of attempts to retire case
For patients at home, toxicities are serious and must be monitored and managed as closely as any reactions observed in the clinic or hospital. Ipilimumab can result in an autoimmune phenomenon that requires treatment with steroids.
Fertility preservation should be discussed with all patients of child bearing age, including sperm banking which should be offered whenever possible before a young man begins chemotherapy.
Spinal cord compression is an emergency and requires neurosurgical or radiation intervention.
Patients may not volunteer that they are not taking their medications as prescribed. Physicians should work with ancillary and nursing services to address these issues as low adherence can cause poorer clinical outcomes.
It is important for trainees to report errors that do not lead to harm as well as near misses so that we can collectively improve the safety of our systems of care. Medical errors should be disclosed to patients in a thoughtful way that involves appropriate risk management staff.
Drug/drug interactions are vital to review when prescribing oral chemotherapy. Capecitabine can potentiate warfarin levels and result in life-threatening bleeding.
Recognize the inappropriate, appropriate and required indications for use of growth factor support.
Due to the risk of life-threatening infection, it is vital to immediately treat neutropenic fever. Steroids can mask a fever so clinical judgment must be used to watch for signs/symptoms.
76.57 % of cases were answered correctly on the first attempt. However, only one case was answered correctly by all participants on the initial try. All the others had at least two participants answer incorrectly on the first attempt. One case had significantly more incorrect answers than all others with seven participants requiring two or more attempts to retire it. The case explored the challenges of monitoring toxicities that cancer patients experience at home.
Case performance was also reviewed across the three international teams. All teams had three cases which all participants answered correctly on the first attempt. However, only one case was answered correctly by all teams on the first attempt, the case dealt with treating neutropenic fever. In Addition, the Australian team scored 100 % on the first attempt for two cases, one on spinal cord compression, and one on oral chemotherapy drug interactions. The Danish team scored 100 % on the first attempt for a case on error reporting for trainees, and one on appropriate use of growth factor support. The US team scored 100 % of on the first attempt for a case on poor patient medication compliance, as well as a case on managing toxicities in patients as closely as would be done if observed in the hospital setting. Refer to Table 3 for an overview of the percentage of participants who answered each case correctly on first attempt compared to case take home message.
Four participants who begun undertaking the Qstream course, but did not complete it. Of this group one participant answered three cases before discontinuing, two participants completed four cases before discontinuing, and one participant answered seven cases before discontinuing. One participant did not answer a single case incorrectly before discontinuing the course, with the other three only answering one case incorrectly. Only one participant discontinued the course directly after answering a case incorrectly, the remaining three discontinued after answering a case correctly. Two participants that discontinued the course answered the same case incorrectly, a case which related to oral chemotherapy interactions.
A representative of the research team contacted each of the four participants who did not complete the course to find out why. Unfortunately, none of the non-completers responded to this contact.
Participants were also given the opportunity to leave free-text comments regarding strengths and weakness, along with any general comments about the Cancer Cup Challenge. There were 32 free-text comments about the Cancer Cup Challenge, with some respondents leaving more than one comment. Of those, 21 comments indicated the course was a generally positive experience. Reasons respondents enjoyed the course included the expert feedback on cases, the competitive aspect of the course, and the flexibility and manageability of the course format. One respondent indicated they appreciated that the cases covered scenarios that were less commonly considered issues in education and training.
Of the participants who did not find the course beneficial, the most commonly cited reason was that the cases were too easy. This was closely followed by respondents stating they would prefer receiving more cases.
Interview thematic review
A total of 14 participants volunteered to participate in semi-structured interviews about the Cancer Cup Challenge. Of the 14 interviewees a total of five were from Australia, two were from Denmark and 12 were from the United States of America. Once interviews had been conducted they were transcribed, de-identified and then content reviewed to identify key themes. A total of five broad themes emerged from the analysis: Impact of course on knowledge and confidence, enjoyment and engagement, motivation for course completion, online learning/Qstream format, and course content in general.
“It was engaging. It didn’t take very long to do so it wasn’t a burden.”
A further two participants made comments stating their engagement and enjoyment of the course was linked to the content. One participant specifically noted that the Cancer Cup Challenge could have been more engaging with revised content.
The topic of motivation to complete the Cancer Cup Challenge was another theme that emerged from the analysis. Interestingly, respondents were quite divided when it came to their motivation to continue completing cases for the course. Five respondents commented that their primary driver was knowledge acquisition, or interest in the topics covered in the course. Other respondents indicated that the leaderboards and individual competition was a motivator to continue, though not necessarily the primary one.
“I think even though we didn’t know exactly who was in our team, it was still a good motivation to actually provide a little bit of excitement to the thing rather than just turning it into another exercise we had to do. I think that was certainly a good motivation to complete at least the next case. I quite enjoyed that.”
“Primarily it was to do you a favour in a way because the med said you wanted to develop some e-learning and I think that’s very important because if you can develop some good e-earning which can be part of our specialist training that would be excellent.”
“It certainly prompted me to think about some of the alternative pathways that I could’ve taken in each situation and why they might be incorrect or less correct and putting my own best answer into context. I found it useful from that end even though most of the questions I felt I knew exactly which answer I wanted to put down.”
Of the respondents who felt the course impacted on their knowledge, the majority suggested this was because of the applicability of the cases to their clinical practice. Several respondents indicated they had dealt with situations during their training that were covered by the cases. One also added that they could easily see themselves encountering the case scenarios in practice.
“It was helpful to know that there was a shared terminology globally (across institutions). Everyone is dealing with the same problems internationally and people are working in the same framework. It was nice to see this is what everyone, everywhere is dealing with.”
Interview participants frequently commented on the Qstream format, with all but one finding it appealing. Eleven respondents indicated that flexibility and the ability to answer cases at their convenience was an advantage of the platform. Four respondents made particular note of the fact they completed the course on their smart phones in between other activities. Seven participants specifically commented on the user friendly interface.
“I thought it was very good that the prompt came through and it would come through again if you didn’t answer the question. That kind of reminded me that the course was there because you can forget these things.”
The final theme that emerged from analysis of the interviews was generalised comments about course content. Seven participants gave responses regarding content that could be included in future courses, three of whom specifically asked for more cases on chemotherapy errors, drug interactions and other issues. In addition, three respondents made general comments about the length of the course, expressing a desire for there to be more cases.
This study demonstrated the feasibility of Qstream as a delivery vehicle for specialty-specific content in the area of safety and quality. The completion rates for the program, which was entirely voluntary, were encouraging and indicate a high level of engagement by participants in each country. This finding aligns with other studies using Qstream in the context of residency training . Once again, the case-based and bite-sized nature of the program was found to be appealing to busy doctors in training. Feedback from trainees indicates that the use of oncology specific scenarios motivated participation. While this finding is perhaps not surprising, it does demonstrate that junior doctors are interested in using non-traditional training delivery methods such as online course games.
The ability of the gamification element of the program to motivate participation supports the use of friendly competition in health professional training programs and is consistent with findings in other contexts . Howerver, further research is needed to explore using this innovative approach in medical education in the clinical setting as there is no other research on the topic. This is a significant gap due to the busy schedules of health professionals and the difficulty of engaging this group in online education. Incorporation of game based approaches can be used to increase learner motivation to complete an online program, particularly if it is asynchronous, as was demonstrated in this study. The use of international competition clearly motivated a number of participants and the use of high-profile organizations in each country likely enhanced this. However, it was interesting to note that the individual competition appeared a more significant motivating factor than the international team-based competition. This indicates the value in establishing competition just at a program level to stimulate participation.
The use of doctors in training to develop the core cases in this program also provides a mechanism to meet training recommendations from organisations such as the ACGME which mandate direct engagement of residents and fellows in educational activities to enhance high quality patient care . Literature shows that residents and junior doctors may perceive events and actions in unique ways due to their frequent interactions with patients. In particular they may be well suited to identifying near misses and other incidents not normally captured in current reporting structures . Case-based programs in safety and quality such as Qstream engage trainees and serve as effective tools to teach critical topics such as patient safety and quality improvement.
Strengths of the study include that over 50 % of participants completed the feedback survey and 14 participants took part in semi-structured interviews allowing for further valuable assessment of the value of the program. Limitations of the study include the lack of available measures to judge impact on practice and relatively low total number of participants in the study. Although the sample size was small comparative to the number of junior doctors in the clinical setting, medical oncology junior doctors represent a sub-set of trainees which is much smaller. In this study all medical oncology junior doctors at the three sites were invited to participate in this study and there was a high participation of oncology trainees at each site. In future studies it may be beneficial to use a pre and post test approach in order to precisely measure knowledge change amongst participants. Additionally, the inclusion of more hospital sites may be a benefit in increasing the number or individuals eligible to participate in the course.
Qstream represents a tool that can be used by individuals and organizations to engage junior doctors in patient safety and quality improvement training. The ability of Qstream to motivate participation, combined with its previously demonstrated impact on behaviour, has implications for directors of training programs. Our findings also illustrate the benefits of tailoring content to the specialty-specific context of the trainee, as well as introducing a gamification element to enhance participation.
Ethics approval and consent to participate
Permission to conduct this study has been received from the Western Sydney Local Health District Human Research and Ethics Committee.
Availability of data and materials
Data will not be shared due to restrictions stipulated by the ethics committee that approved this study.
The authors wish to thank the program coordinators across the international sites who greatly assisted with the promotion of this program to potential study participants. Additionally, thank you to all participants in the study for volunteer ring their time to assist with this research project.
This study was financed through the Sydney West Translational Cancer Research Centre grant from the Cancer Institute New South Wales.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Wilson RM, Runciman WB, Gibberd RW, et al. The quality in Australian health care study. Med J Aust. 1995;163:458–71.Google Scholar
- Neale G, Woloshynowych M, Vincent C. Exploring the causes of adverse events in NHS hospital practice. J R Soc Med. 2001;947:322–30.Google Scholar
- Scott IA. Errors in clinical reasoning: causes and remedial strategies. BMJ. 2009;338.Google Scholar
- Vincent C. Understanding and responding to adverse events. N Engl J Med. 2003;348:1051–6.View ArticleGoogle Scholar
- de Vries EN, Ramrattan MA, Smorenburg SM, et al. The incidence and nature of in-hospital adverse events: a systematic review. Qual Saf Health Care. 2008;17:216–23.View ArticleGoogle Scholar
- Jagsi R, Kitch BT, Weinstein DF, et al. Residents report on adverse events and their causes. Arch Intern Med. 2005;165:2607–13.View ArticleGoogle Scholar
- Haller G, Myles PS, Taffé P, et al. Rate of undesirable events at beginning of academic year: retrospective cohort study. BMJ. 2009;339.Google Scholar
- Weiss KB, Bagian JP, Wagner R. CLER Pathways to excellence: expectations for an optimal clinical learning environment (executive summary). J Grad Med Educ. 2014;6:610–3.View ArticleGoogle Scholar
- Reed JE, McNicholas C, Woodcock T, et al. Designing quality improvement initiatives: the action effect method, a structured approach to identifying and articulating programme theory. BMJ Qual Saf. 2014;23:1040–8.View ArticleGoogle Scholar
- Graham IS, Gleason AJ, Keogh GW, et al. Australian curriculum framework for junior doctors. Med J Aust. 2007;186:S14.Google Scholar
- Benstead K. What is valuable for specialist registrars to learn in order to become good consultant clinical oncologists? Clin Oncol. 2006;18:549–54.View ArticleGoogle Scholar
- Van Der Weyden MB. Challenges and change in medical training: the Australian Curriculum Framework for Junior Doctors. Med J Aust. 2007;186:332.Google Scholar
- Shaw T, Barnet S, Mcgregor D, et al. Using the Knowledge, Process, Practice (KPP) model for driving the design and development of online postgraduate medical education. Med Teach. 2015;37:53–8.View ArticleGoogle Scholar
- Fischer MA, Mazor KM, Baril J, et al. Learning from mistakes. J Gen Intern Med. 2006;21:419–23.View ArticleGoogle Scholar
- Kerfoot BP, Kearney MC, Connelly D, et al. Interactive spaced education to assess and improve knowledge of clinical practice guidelines: a randomized controlled trial. Ann Surg. 2009;249:744–9.View ArticleGoogle Scholar
- Robinson T, Janssen A, Kirk J, DeFazio A, Godwin A, Tucker K, et al. New aproaches to continuing medical education: a Qstream (spaced education) program for research translation in ovarian cancer. J Cancer Educ. 2015. doi:10.1007/213187-015-0944-7.Google Scholar
- Bruckel J, Carballo V, Kalibatas O, Soule M, Wynne KE, Ryan MP, Shaw T, Co JPT. Use of spaced education to deliver a curriculum in quality, safety and value for postgraduate medical trainees: trainee satisfaction and knowledge. Postgrad Med J. postgradmedj; 2015; 0:1-8.Google Scholar
- Kerfoot BP, DeWolf WC, Masser BA, Church PA, Federman DD. Spaced education improves the retention of clinical knowledge by medical students: a randomised controlled trial. Med Educ. 2007;41(1):23–31.View ArticleGoogle Scholar
- Vlach HA, Sandhofer CM. Distributing learning over time: the spacing effect in children’s acquisition and generalization of science concepts. Child Dev. 2012;83(4):1137–44.View ArticleGoogle Scholar
- Kerfoot BP, Baker H. An online spaced-education game to teach and assess residents: a multi-institutional prospective trial. J Am Coll Surg. 2012;214:367–73.View ArticleGoogle Scholar
- Knaving K, Björk S. Designing for fun and play: exploring possibilities in design for gamification. Proceedings of the First international Conference on Gameful Design, Research and Applications. 2013. p. 131–34.Google Scholar
- Worm BS, Buch SV. Does competition work as a motivating factor in E-learning? A randomized controlled trial. PloS one. 2014;9, e85434.View ArticleGoogle Scholar
- Papastergiou M. Exploring the potential of computer and video games for health and physical education: a literature review. Comput Educ. 2009;53:603–22.View ArticleGoogle Scholar