Gamification versus Face-to-Face Learning Approach in Point-of-Care Ultrasound Training: A Randomized Controlled Trial

Background: Although gamification increases user engagement, its effectiveness in point-of-care ultrasound training may yet to be fully established. This study was conducted with the primary outcome of evaluating its effectiveness in point-of-care ultrasound training workshop as compared to conventional face-to-face learning. Methods: Participants (who were junior doctors between 2 – 4 years of clinical experience) were randomized into either the (1) gamification or the (2) face-to-face learning arms. Similar educational intervention was implemented to participants in both arms but the in-gamification arm, the theory assessment was administered in the form of live quizzes with real time leaderboards, and the practical assessment was administered in the form of 3 games, i.e., ultrasound minefield, ultrasound pong and ultrasound game. Pre-test, post-test and 2 months post-training theory and practical assessments were conducted. Results: A total 32 junior doctors participated in this study (16 participants in each arm). For theory assessment, paired student’s t-test showed significant improvement in both face-to-face learning (pre-test score: 12.38 vs post-test score: 19.88; 95% CI [5.35, 9.65] p < 0.001) and gamification arms (pre-test score: 13.38 vs post-test score: 20.81; 95% CI [5.93, 8.94] p < 0.001). For the practical assessment, paired student’s t-test showed significant improvement in gamification arm (pre-test score: 12.56 vs post-test score: 18.13; 95% CI [2.44, 8.69] p<0.001) but not the in the face-to-face learning arm (pre-test score: 16.00 vs post-test score: 18.38; 95% CI [-0.17, 4.92] p<0.065). When re-tested 2 months post-training, both face-to-face learning and gamification arms showed significant improvement for both theory and practical assessment. Conclusions: Gamification approach could be an effective alternative or more effective than face-to-face learning in point-of-care ultrasound training.


Introduction
Point-of-care ultrasound (POCUS) is a vital diagnostic and therapeutic intervention in acute patient care [1]. The Rapid Ultrasound for Shock and Hypotension (RUSH) protocol, for example, can be used to determine the appropriate interventions for patients with undifferentiated shock [2]. POCUS training has since evolved to be a core competency curriculum as outlined in the American College of Emergency Physicians (ACEP) Ultrasound Guidelines [1]. Nonetheless, the most appropriate pedagogy to achieve the learning outcomes for POCUS training may remain unclear [3].
Previous POCUS pedagogy using conventional apprentice-based model ('see one, do one, teach one approach') is gradually being phased out as the default pedagogy [4]. Such conventional strategy usually comprises of a mix of instructor-centered didactic classroom lectures with hands-on training sessions [4]. This approach, however, is rather passive as it does not truly engage the learners. It may preclude critical thinking in decision making, which essentially, is a key component as clinician needs to be able to integrate real-time ultrasound findings in patient management.
Gamification is defined as "the use of game design elements in non-game context" [5]. Gamified pedagogy injects the element of fun through the use of game components such as the points, badges, leader boards [6]. An example of innovative gamified pedagogy in emergency medicine (EM) is known as the SonoGames initiatives, introduced in 2012 in the annual Society for Academic Emergency Medicine (SAEM) conference. This was an annual 4-hour game-style platform aimed to assess the EM residents' knowledge, skills and decision-making while promoting the development of teamwork and peer communication [7]. Another example of POCUS competition was called the "Sono-Witcher Wild Hunt" [8]. It was organized by the Emergency Medicine Association of Turkey Ultrasound Working Group (EMATUS) during the 9th Asian Conference on Emergency Medicine (ACEM) in November 2017 [8].
Although gamification has been shown to improve user engagement, focus, motivation as well as productivity and knowledge retention [9,10], there is a paucity of comparison done with a control group to objectively determine its effectiveness in the teaching and learning of POCUS. Hence, the effectiveness of gamification in POCUS training may not be fully explored. To fill this gap, we embarked on a study with the primary outcome of evaluating the effectiveness of the gamification approach in POCUS training for junior doctors compared to the conventional face-to-face learning (F2FL). It is hypothesized that the gamification approach is as effective as or more effective than the conventional F2FL approach. The secondary outcomes of this study are 1) to assess the overall retention of junior doctors in POCUS education 2 months post-training and 2) to assess the participants' perception of the overall gamification experience.

Method Study Design And Setting
This study was a prospective randomised controlled trial conducted from May to November 2018 in the Emergency and Trauma Department (ETD) of Sarawak General Hospital (SGH), Malaysia to compare the gamification approach versus F2FL approach for POCUS training using the RUSH protocol. In this study, we also included an assessment of the participants' perception towards the gamification approach in POCUS training. This study was approved by the Medical Research and Ethics Committee (MREC) Malaysia and was registered with the National Medical Research Register (NMRR-18-444-40348). As this study was conducted as an actual POCUS training workshop, the number of participants was limited by resource availability. Hence, a convenient sampling of 32 participants (16 participants in each arm) were recruited.

Study Population
The study population comprised of junior doctors working in the ETD SGH, Sarawak Heart Center and the Internal Medicine Department of SGH. We defined a 'junior doctor' as a doctor with at least 2 years of but no more than four years' experience of clinical service. The reason for selecting doctors with 2 to 4 years of clinical experience was because this group of doctors would have completed their compulsory two-year internship program and would likely have developed sufficient clinical exposure to be able to critically understand the course content of POCUS training and the utilization of POCUS in clinical decision making.
Any junior doctor who had participated before in any formal POCUS training was excluded. Informed consent was obtained from all participants before commencing this study. All participants joined this free POCUS training on a voluntary basis without any payment or monetary compensation.

Materials
The topics for this POCUS training workshop were based on the requirements of the original RUSH protocol and adopted from the World Interactive Network

Study Procedure
This study was divided into two stages: 1) identifying learning materials and development of assessment questions using the modified Delphi method and 2) recruitment, randomization and implementation of interventions. In the first stage, the discussions were carried out with the experts in three rounds. Most of the discussion was carried out via e-mail and online group dialogues as the authors were based in different locations in Malaysia. The first round of discussions focused on identifying the main objectives and probable contents of the workshop. In the second round of discussion, the list of probable topics was distributed by email to the experts for review and to reach a consensus on the suitability of the topics. In the final round of discussions, the shortlisted topics were divided and assigned to the specific panel of experts for teaching as well as for preparation for the assessment questions. This assessment consists of two sections: (1) 30 one-best answer (OBA) type of multiple-choice questions (for theory assessment), and (2) one objective structured clinical examination (OSCE) question (for practical assessment). The questions were then vetted, revised, finalized and agreed upon by the experts.
In the second stage of study, participant recruitment, randomization and implementation of educational intervention were conducted. Thirty-two participants were randomized to either the gamification or F2FL. The participants in the gamification arm were further randomized into groups of 4 participants. Randomization processes were performed using an online random number generator.
On the first day of the course, all participants are required to complete a pre-test knowledge (30 onebest answer type of multiple-choice questions) and practical skills assessment test. The practical skills assessment was conducted in the format of one objective structured clinical examination (OSCE) question and assessed by three independent emergency physicians who were blinded to the participants' study arms. This process was repeated during the end of the course on the following day as the post-test knowledge and skills assessment. To assess the participants' retention of knowledge and skills, the theory and practical assessment was repeated two months after completion of the course. We chose a time gap of 2 months based on a previous study which shows that knowledge retention after an educational intervention was approximately 55 days or less [11].
In addition, participants from the gamification arm also completed a gamification experience survey (adapted from Lobo et al, 2017) [4] aimed to assess the participant's perception of the different components of gamification using a Likert scale. The schedule for POCUS training workshop using RUSH protocol for both gamification and F2FL groups is given in Supplementary Table 1.
With regards to theory assessment, in the F2FL arm, a written OBA questions paper was distributed for students to answer. Answers were discussed at the end of the day. The questions were marked individually. In the gamification arm, instead of an individual written theory paper (OBA), live quizzes were implemented. There were 4 teams with 4 participants in each team. Gamification mechanics and elements used were the points, badges, leader boards, teamwork, competition, immediate real-time feedback, challenges with set objectives, chance, turns, rules, scaffolded learning with optional increasing challenges, a sense of progression and presence of social dynamics and interaction [12]. In this study, each team began at level one with zero experience points (XP) and required XP to progress to the next level. XP points are gained through live quizzes interspersed between lecture modules by answering correctly the questions. Real-time leader boards were used to monitor the progression of teams (Fig. 1A). Virtual badges were rewarded based on progression. Each team had a chance to answer the quiz and discussed amongst the team members. The quiz was presented using a Jeopardy-style game show format using FlipQuiz™ technology (Fig. 1B) whereby teams are able to pick the level of difficulty of questions with different points allocated. If any team were unable to answer the question, another team was given a chance to answer but would only obtain half of the awarded points if answered correctly. Feedback for the correct answers was given immediately to all teams. There was no negative marking in all quizzes.
For the practical assessment in the F2FL arm, case-based assessment (in OSCE format) was carried out individually to every participant. In the gamification arm, the ultrasound games were played as team activities with similar marks awarded to the all team members. The psychomotor skill objectives covered in the hands-on practical sessions and the corresponding gamified versions for participants in the gamification arm are given in Table 1. At the end of the course, the team with the highest score won a reward. All the quantitative data was analyzed using IBM Statistical Package for the Social Sciences (SPSS) v23 for Windows.

Results
A total of 32 junior doctors participated in this study with 16 participants randomized to each arm.
One participant was subsequently dropped out from the study during the 2-month post-retention assessments due to work commitment. The median age was almost similar at 27 and 27.5 years old in the gamification arm and F2FL arm respectively. The detailed demographics and characteristics of the participants are detailed in Table 2.
Parametric test was used for analyses as the assumptions of normality were satisfied. Analysis with paired student's t-test demonstrated that there was statistically significant improvement in the posttest (as compared to the pre-test) in the knowledge score in both gamification and F2FL arms. With regards to the practical skill component, the post-test score was statistically higher compared to the pre-test score in the gamification arm only, but not for the F2FL group (Table 3). When re-tested 2 months later, all participants showed significant improvement in both knowledge and skills irrespective of whether they were from the F2FL group or the gamification group (see Table 4 for details).
A subgroup analysis of the participants' perception towards the gamification experience survey was also performed using a Likert scale of 1 to 5. This subgroup analysis was divided into three components related to gamification, i.e., "engagement", "perceived knowledge and learning benefit" and "game elements and mechanics". Overall, the responders demonstrated positive perceptions to the various aspects of gamifications like better engagement and motivation, increased in selfperceived knowledge and learning benefits and enjoyable gaming elements and mechanics (teamwork, competition, points, badges, leader board, immediate feedback, rewards) introduced throughout the course. One particular game which stood out compared to other types of games was the ultrasound hands-on games. All the participants in the gamification arm (n = 16, 100%) concordantly agreed that the hands-on ultrasound games were enjoyable during the course, and probably the most likable game mechanic introduced in the gamification group. The detailed responses of the participants gamification experience are shown in Table 5. Table 1 Psychomotor objective Gamified version 1. To select and adjust for appropriate settings such as the gain and depth on the ultrasound machine ⇒ Ultrasound minefield: The main aim of the game was to be able to acquire the image and correctly calculate the number of water balloons hidden in a gelatinfilled container. The fastest team and closest to the correct number of water balloons would earn the highest points. This tested the participant's psychomotor skills, selection of ultrasound probes and settings and teamwork (see Fig. 1C and 1D) 2. To perform and obtain correct sonographic images of different body parts and organ systems; ⇒ Ultrasound pong: Labelled ping pong balls with written anatomy are placed in a bowl. Participants from each team are then required to randomly pick one and demonstrate the anatomy of the simulated patient. The main aim of the game is to be able to demonstrate the image of the anatomy being randomly chosen by chance. Each team member in the team of four would take turns, with points gained for every correct answer. 3. To perform a bedside POCUS based on a case scenario using simulated patients ⇒ Ultrasound game: Each team was given different simulated case scenarios and required to correctly give their assessment for the patient using the RUSH protocol. After assessing the simulated patient, participants were required to interpret the video clip leading to the diagnosis and management of the critically ill patient. Points are given for adequate view, correct ultrasound interpretation, diagnosis and management for each team. Table 2 Baseline characteristic of participants in the two study groups  Table 3 Paired t-test to study the difference between pre-and post-test knowledge, skills and total scores for face-to-face and gamification groups.  Note: Items 1 to 3 represents perception of the engagement component. Items 4 to 8 represents perception of perceived knowledge and learning benefit in the gamification approach. Items 9 to 17 represents perception of game elements and game mechanics.

Discussion
The results from this study suggest that gamification approach in POCUS training may be as good as or even better than F2FL. This is evidenced by a statistically significant improvement in the practical skill component from pre-to post-test in the gamification arm but not the F2FL arm (besides the significant improvement in knowledge and total scores for both groups). We also found that the retention of POCUS knowledge and skills in the junior doctors two months after completion of the initial course in both arms was good.
Participants from the gamification arm responded favorably towards the game elements (points, leader boards, fun, teamwork, competition and immediate real-time feedback) and game mechanics (quiz-based format and level up progression via points, the added sense of enjoyability) in the gamified approach towards POCUS training. More importantly, all participants in the gamification arm either "agreed" (12.5%) or "strongly agreed" (87.5%) that they felt motivated when participating in the gamified approach. The motivation, interaction and fun elements were important to fully engage the participants in their learning processes.
Flow is a concept describing the experience people enjoys so much to the extent that their attention and focus is fully vested into the task at hand [13]. It is called "flow" because metaphorically, this is akin to someone being drifted effortlessly in the flow of water current without being aware of the passage of time or the fatigue feelings that he or she may have. In this regard, all participants either agreed (12.5%) or strongly agreed (87.5%) "…. that time passed by quickly during this course". The state of flow can be seen in the pursuit of varied types of games [14,15]. According to Csikszentmihalyi et al (2014) [13], three key conditions need to be fulfilled in order to achieve the engaging flow experience. First, there should be a clear set of goals to provide the trajectory and purposes of the activities. In this regard, the three ultrasound games have clear goals for the participants to achieve, i.e., acquiring the image and correctly calculate the number of water balloons hidden in a gelatin-filled container (in "ultrasound minefield"), demonstrating the image of the anatomy required as dictated in the ping-pong ball (in "ultrasound pong") and competently performing POCUS assessment using the RUSH protocol in a simulated case scenario (in "ultrasound game"). Second, there should be a balance between the perceived challenges and perceived skills.
When the perceived challenges and perceived skills are well matched, the attention is completely absorbed. When the perceived challenges begin to exceed the perceived skills, anxiety may set in. On the other hand, when the perceived skills exceed the perceived challenges, one may sink into boredom. In this regard, the game mechanics (the level up progression of the three ultrasound games tailored with increasing difficulty levels) sets to ensure that the participants have obtained the skills necessary in one level before progresses to the more challenging level (see Fig. 2 for an illustration of this concept). Third, clear and immediate feedback should be given, including both positive performance feedback and negative performance feedback. In this regard, the game elements (such as the points, leader board and immediate comments from examiners) provide the immediate feedback to the participants. In fact, 18.8% of participants agreed, and another 81.3% strongly agreed with the statement "I believe that immediate feedback from the quiz was helpful".
Besides that, 75% of the participants strongly agreed that they enjoyed working with a teammate and 87.5% of them believed that the group competition helped to better acquaint themselves with their fellow colleagues. These 2 traits, i.e., teamwork and effective communication are essential in clinical management in the emergency department to deliver optimal health care. In fact, as alluded by Bandura (2001) in his social cognitive theory, optimal learning takes place in a social context, when participants learn and imitate from one another through the various social interactions, teamworking and communication [16]. The World Health Organization (WHO) has accordingly also stressed on the importance in effective teamwork in the health care environment, as it is closely linked to reducing adverse events due to miscommunication and misunderstanding among teams caring for the patients [17].
This study, however, is subjected to several limitations. First, the sample size is relatively small. This decreases the power of the study and increases the risk of Type II errors. Furthermore, this study only included junior doctors from a single center. Inviting doctors of different competency level and from different centers in Malaysia to participate in such study would be more representative of a wider population and, therefore, bolster a stronger basis for consequent generalization of findings. Next, we could not exclude the possibility of the Hawthorne effect among the participants, as they were well aware that they were under evaluation in the research study to assess the effectiveness of gamification, thus may lead to a biased outcome. Besides that, the way the assessment was conducted also differed between the two arms. In the gamification arm, the scoring done was basically a team scoring, whereas in the F2FL arm, the scoring was first done individually before the scores being summed up as a group.

Conclusion
This study shows that gamification approach could be an effective alternative or more effective than F2FL in POCUS training, particular in the skill training. The incorporation of gamification into POCUS training with the use of the RUSH protocol can potentially serve as an engaging and enjoyable platform to deliver ultrasound education for junior doctors, which we hope, would translate into better patient care and safety.

Declarations
Ethics approval and consent to participate: Written approval from the National Medical Research Registry (NMRR-18-444-40348) was obtained prior to commencement of the study. All participants in this study consented to participate voluntary and that their data might be used in any research or presentation without in any way revealing their personal identities. Participants were assured of the confidentiality of their responses. No monetary compensation was involved in the recruitment of the participants.
Consent for publication: Permissions were also obtained from the participants to publish their data anonymously without revealing their names and identities.
Availability of data and material: The *.sav data that support the findings of this study are available in https://tinyurl.com/wytfjjb or from the corresponding author (KSC) upon reasonable request (should the link no longer works) Competing interests: All authors declare that they do not have any competing or conflict of interest in this study Funding: Not applicable.