iSTART is a double-masked randomised trial that was held among medical students at the School of Medicine of the Universidad de Valparaíso, Chile. The study protocol has been drafted in compliance with the Consolidated Standards of Reporting Trials (CONSORT) statement as in its version adapted for trials evaluating non-pharmacological interventions [13, 14]. The complete protocol was registered in March 2016 at
clinicaltrials.gov
(NCT02723136) and can be reviewed at https://clinicaltrials.gov/ct2/show/NCT02723136?term=NCT02723136&rank=1. A flowchart describing participant recruitment and overall study design is shown in Fig. 1.
Participants
Eligible participants were medical interns coursing their last year of training at the School of Medicine of the Universidad de Valparaiso, who had a personal Smartphone with an iOs®- or Android®-based operating system. Only those that did not wish to participate were excluded from this study. Informed consent was obtained from all participants.
Every student sat a baseline 90-question test aimed to resemble EUNACOM (see below) and were randomised to receive a smartphone-based application training afterwards. Randomisation was carried out using permuted blocks by a statistician that was unaware of treatment allocation. Allocation sequences were concealed from other researchers participating in this study. All participants were asked to complete an entry form with basic demographic data, including age, sex, year of training and prior experiences with smartphones or similar platforms (i.e. tablets). Data regarding academic performance was obtained from the University, including qualifications relevant to the area of Internal Medicine.
Interventions
Students allocated to receive the active intervention received a downloadable application that was installed in their smartphones. Those allocated to the control group did not receive any additional training for EUNACOM. The mobile application was devised by a team of informatic engineers and physicians and made available for free at the App Store® and PlayStore® for both iOs® and Android® operating systems. In order to monitor adherence, the application required an active internet connection for operation. Students also received a brief (5-min) description on functionality that was also made available in text form as a part of the software.
Contents were primarily directed at the area of Internal Medicine, which is the most important specialty within EUNACOM. It included a series of questions in the form of brief clinical vignettes constructed in a format similar to the one described in EUNACOM’s website [15]. In short, these vignettes correspond to clinical scenarios against which the student must answer a key aspect relevant to the diagnosis, management or monitoring of several diseases. These multiple-choice questions must be answered from five possible options, with only one being the correct answer. The depth of knowledge required to answer was established using the provisions of the EUNACOM agenda [12]. All contents of the application were designed by two internists with 5 years experience in developing questions for the exam. Examples of these vignettes are provided in the Additional file 1.
The application had two modes to provide the aforementioned inquiries. In the first, study mode, students were not given time constraints to answer the clinical vignettes. Whenever an answer was provided, instant feedback was delivered alongside a brief explanation of the key concept that was being assessed by the inquiry. In the second, training mode, participants had a restricted time window to provide answers. This mode was designed because of a perceived difficulty amongst interns in managing time in answering questions in previous simulations of the exam. A default of 60 s was established, but the application allowed the user to modify this timeframe to 30 or 90 s. Students had knowledge regarding their individual performance in both modes, but no additional feedback in terms of concept review was provided in training mode.
Outcomes
The primary outcome is the mean change in overall scores in a 90-question practise test designed to resemble EUNACOM between groups. The final test did not repeat any of the questions used within the application that was delivered to students and was held 4 weeks after randomisation. This timeframe was selected in order to allow students to practise and study internal medicine with the application given the extent of contents required by EUNACOM. Simulation tests were used because of the impossibility to use the actual exam as part of this study, since it is managed independently from universities and kept in strict reserve by ASOFAMECH. However, previous data has shown that both simulation exams (baseline and final) have good correlation with overall EUNACOM scores (r > 0.7, p < 0.001), as well as an excellent diagnostic accuracy for detecting students at risk of failing the exam(area under the ROC curve 0.95, 95% CI 0.90 to 0.99) and identifying students that will obtain high scores in the review (AUC 0.80, 95% CI 0.71 to 0.88, unpublished data). The correction of both practice tests was undertaken by reseachers that were kept unaware of allocation.
A secondary endpoint was to establish differences in the average time required to answer clinical vignettes. In order to allow reliable comparisons to be made, exams were conducted electronically and under supervision by the research team, thus allowing an objective assessment of the total time required to complete the review. Data regarding adherence was also collected.
Statistical analyses
Sample size
Sample size was calculated using data regarding overall perfomance in prior experiences with practise exams and estimates from a randomised trial [10]. It was calculated that a sample size of 64 participants (32 per group) would be required to obtain 80% power to detect an absolute difference of 5 points between groups, assuming a standard deviation of 7 points for both groups at standard significance levels (two-tailed α of 5%). In order to correct for up to 20% losses of follow-up, it was sought to randomise 75 participants. All estimates were calculated using nQuery Advisor® 3.0 for windows.
Analysis plan
Basic descriptive statistics (means, medians, proportions, interquartile ranges -IQR-, etc) were performed to assess the characteristics of the study sample. Fisher’s exact test was used to evaluate univariate association of categorical variables. Quantitative variables were compared using Mann-Whitney or Student’s T tests according to data distribution and variances. Ninety-five percent confidence intervals were constructed whenever appropriate. Missing data relevant to the primary and secondary outcomes were handled using multiple imputation techniques. In order to reduce sampling variability due to the imputation process, 20 datasets were generated for every variable with missing data. Predictor variables were included in this procedure using linear regression for data showing normal distributions. Predictive mean matchings were preferred to impute data for variables with skewed distributions. All analyses were undertaken by a statistician who was unaware of participant allocation using Stata v12.0® (StataCorp LP, 1996–2016) under the intention-to-treat principle, but complementary complete-case analyses were conducted as part of multiple imputation techniques.