Implementation of a Smartphone application in medical education: a randomised trial (iSTART)

Smartphones are recent technologies that combine the capabilities of telephone communications and informatics in small portable devices that allow communications and information processing even at the patient’s bedside [1, 2]. As noted in the general public, these devices have shown significant growth in the international medical community [3], niche where they perform functions that range from undergraduate education to health resource management [4, 5]. Several studies have shown that smartphones are frequently used among physicians, medical students and interns, with overall use rates reaching 80%. iOs®-based systems, such as the iPhone®, seem to be the most commonly used platforms [1, 6–8]. The popularity of smartphones is likely to stem from their versatility. Current devices have a wide variety of functions, which can assist in medical decision making, information searches and educational applications, among other uses [8]. Use in clinical practice seems to be more common among women, people with an interest in new technologies and those with prior experiences with these platforms [7].

Despite this popularity, there is limited evidence regarding the effectiveness of smartphone use in improving academic performance among medical students [9]. While there is a wide availability of applications and resources available for these platforms, only a few randomised trials have addressed their effectiveness in improving academic performance. In 2011, Low and coworkers published one of these studies using objective clinical competence scores as a primary endpoint [10]. The latter trial reported a statistically significant improvement of roughly 15% in the academic performance of students allocated to receive the application. Similar findings were seen in a second, before & after, study that was conducted among Obstetrics & Gynecology residents [11].

Since 2003, a national examination for undergraduate medical students that have completed their internships is carried out in Chile. This exam (Examen Unico Nacional de Conocimientos en Medicina - EUNACOM) is designed to assess the overall knowledge and practical skills that any medical student should attain before practising medicine in the country. Its confection and administration are regulated by law, and its oversight has been delegated to the Association of Faculties of Medicine of Chile (ASOFAMECH). EUNACOM is made of two sections, theoretical and practical, and is considered qualifying to practise medicine in Chile. The contents of both sections are of public knowledge and include 1543 items distributed according to the curricular time spent training in different areas of medicine, with special emphasis on internal medicine and its subspecialties [12]. The theoretical component is evaluated using 180 multiple-choice questions delivered in two 90-min sessions. Additionally, EUNACOM provides professional title validation or equivalencies for foreign physicians who wish to practice medicine in Chile. Given the importance of this exam, several medical schools have implemented preparation courses for their students. However, the methodologies used in the latter courses are heterogeneous, and uncertainty exists regarding the best way in which contents should be delivered.

This study aims to determine whether the implementation of a smartphone application designed to assist in delivering key concepts relevant to internal medicine might improve academic performance in EUNACOM.

Smartphones are commonly used devices among medical trainees. In this study, every eligible student had at least one of these gadgets at their disposal, and most reported considerable experience using them in their everyday lives. This popularity makes these platforms attractive targets to design and develop interventions for medical training. However, only a handful of randomised trials that address smartphone applications with educational purposes are available in the literature. Most of them have been conducted among postgraduate students undergoing specialty training, and used smartphones as cognitive aids concomitant to assessments of very specific competences [10, 16, 17].

We found that the provision of key concepts in internal medicine using smartphones was a feasible option that also translated in significant improvements in academic performance among medical interns. The observed progress was significant even for a relatively brief intervention that was also self-administered by our students, which adds to the relevance of our findings. Our results are in concordance with the ones observed in similar experiences. In 2011, Low and coworkers [10] published a randomised trial assessing iResus®, an iOs®-based application aimed at improving performance of an advanced life support provider in an emulated medical emergency. The application was designed to provide a quick reference to algorithms and drug dosages to assist in the management of resuscitation efforts. Thirty-one physicians who had already completed an advanced life support course within the previous 4 years were randomised to receive iResus® as a cognitive aid or no additional support during a simulated cardiac emergency. Performance was measured using a validated scoring system. Participants allocated to iResus® showed median scores that were 12.5 (14%) points higher than those seem among students without any further cognitive aids (p = 0.02). Similar findings were seen in a larger study by Hand an coworkers [16], in which 111 residents were randomised to a smartphone-based decision support tool aimed at improving adherence to the American Heart Association Guidelines on Perioperative Cardiac Evaluation. Use of the decision support tool resulted in a 25% improvement in adherence to guidelines (p < 0.001), and participants made 77% fewer incorrect responses in two standardised tests.

Although our findings are similar to the ones seen in the aforementioned trials, our estimates are far more conservative than the ones observed by Low [10] and Hand [16]. This might be explained by the fact that our intervention was not devised to be used concomitantly to assessments as a cognitive aid, but rather as a complementary resource to facilitate study of internal medicine as a discipline. In addition, it should be considered that the scope of contents established as key by the designers of EUNACOM is broader than the ones required by guidelines aimed at aiding clinicians in the management of specific healthcare issues; thus resulting in an apparent reduction of the intervention’s benefits.

Intention-to-treat analyses also showed a nonsignificant trend towards a reduction in total test times and mean time spent per inquiry. A post-hoc power calculation showed that the estimated power for this contrast was of only 45%, thus making insufficient power a reasonable possibility to explain this observed lack of statistical significance. Nonetheless, the observed reduction of 8.5 min is relevant for interns planning to undertake EUNACOM, and is likely to be the result of practice in answering multiple-choice questions. Clinical vignettes are constructed using certain features that are typical of certain conditions, thus leading to patterns that students exposed to the application might have been able to recognise faster than those allocated to the no-intervention group. It could also be argued that students allocated to the intervention also had more experience answering questions on an electronic platform, thus resulting in familiarity with the interface that might have explained these findings. However, this explanation seems rather unlikely considering the vast experience with smartphone applications that participants had in this study.

Given that the intervention was devised to be self-administered by students, adherence was a key aspect to assess while conducting our study. Thirty-four out of 40 participants (85%) used the application’s study mode to review internal medicine in this trial, which was very satisfactory. Furthermore, the median number of questions and questionnaires completed was more than adequate considering the relatively brief timeframe in which this study was conducted. Only a minority of students allocated to the intervention (12 students, 30%) used the applications’ training mode, the sole feature within the application in which a time restraint to respond clinical vignettes was applied. This obvious contrast in use rates reached statistical significance (p < 0.001), and might be explained by performance pressure. It is possible that interns felt discouraged to undertake activities that recorded results in a manner similar than the one used in the actual EUNACOM. Participants could have associated underperforming in these exercises with a potential for poor results in the exam, thus leading to the observed use rates. Feedback provided by this mode did not include a revision of the key concept in internal medicine that was being assessed, thus possibly making pressure for delivering high scores more tangible. Furthermore, interns were warned that time-limited exercises were accessible only once during our trial, which might have led to lesser use rates in order to save this component of the application after the reviewable contents (study mode) were completed. Given these explanations and the fact that EUNACOM applies a time limit of 60 s per question, future interventions aimed at improving performance in this and/or similar tests should not disregard applying time restraints as part of their strategies. Exploring motivations to use these types of applications should be considered in future qualitative research.

In summary, this randomised trial showed that the provision of a smartphone application designed to emulate EUNACOM was successfully implemented amongst medical interns. Its use was associated with significant improvements in academic performance, and is likely to be chiefly explained by the direct provision of concepts in Internal Medicine in form of clinical vignettes. The observed high adherence rates amongst interns make these kinds of interventions a promising field to develop in the future of medical education.