- Research article
- Open Access
- Open Peer Review
An evaluation of UK foundation trainee doctors’ learning behaviours in a technology-enhanced learning environment
© Brooks et al. 2016
- Received: 24 March 2015
- Accepted: 23 April 2016
- Published: 3 May 2016
Technology-Enhanced Learning (TEL) can be used to educate Foundation Programme trainee (F1 and F2) doctors. Despite the advantages of TEL, learning behaviours may be exhibited that are not desired by system developers or educators. The aim of this evaluation was to investigate how learner behaviours (e.g. time spent on task) were affected by temporal (e.g. time of year), module (e.g. word count), and individual (e.g. knowledge) factors for 16 mandatory TEL modules related to prescribing and therapeutics.
Data were extracted from the SCRIPT e-Learning platform for first year Foundation trainee (F1) doctors in the Health Education England’s West Midland region from 1st August 2013 to 5th August 2014. Generalised Estimating Equation models were used to examine the relationship between time taken to complete modules, date modules were completed, pre- and post-test scores, and module factors.
Over the time period examined, 688 F1 doctors interacted with the 16 compulsory modules 10,255 times. The geometric mean time taken to complete a module was 28.9 min (95 % Confidence Interval: 28.4–29.5) and 1,075 (10.5 %) modules were completed in less than 10 min. In February and June (prior to F1 progression reviews) peaks occurred in the number of modules completed and troughs in the time taken. Most modules were completed, and the greatest amount of time was spent on the learning on a Sunday. More time was taken by those doctors with greater pre-test scores and those with larger improvements in test scores.
Foundation trainees are exhibiting unintended learning behaviours in this TEL environment, which may be attributed to several factors. These findings can help guide future developments of this TEL programme and the integration of other TEL programmes into curricula by raising awareness of potential behavioural issues that may arise.
- Foundation trainees
- Technology enhanced learning
- Learning behaviours
Foundation trainee doctors (F1 and F2; those in their first two years after qualifying with a medical degree from university) in the UK are currently allocated protected learning time during each working week , of which some activities can be mandated. Given that prescribing is one of the activities most frequently undertaken by Foundation trainees, and that trainees often feel underprepared to prescribe following their undergraduate education [2–6], it has been recommended that some time should be dedicated to prescribing education .
Keogh et al.  found that the most frequent educational activity voluntarily undertaken by Foundation trainees was via Technology-Enhanced Learning (TEL) packages and it likely that the use of technology in medical education will continue to increase . There are many advantages associated with TEL, including ease of access and the permitted flexibility in the time and place of learning, [10–13]. Such ‘just-in-time’ learning allows learners to obtain the specific information needed to achieve a goal, as and when they require it [14–16]. This may be beneficial for busy trainees, where time is often a limiting factor in undertaking formal education [8, 17]. Indeed, TEL has been widely advocated for the education of medical professionals in a variety of settings and circumstances [18–21].
Despite the advantages of TEL, such flexibility may allow for busy Foundation trainees to exhibit inappropriate learning behaviours within the learning environment. Learners may not complete the TEL as recommended (e.g. completing content only directly prior to an assessment) or use system workarounds (e.g. skipping over content), thereby facilitating superficial as opposed to deep learning, if any learning at all. It is important to determine specifically what may occur in order to aid with future developments of TEL programmes, which has not been discussed in previous research. In Temporal Motivation Theory  (TMT), it is suggested that motivation to complete a task is affected by the individual’s perception of their probability of achieving success; the salience of the reward or punishment associated with completing or not completing the task, and the salience of a potential reward or punishment in relation to its temporal proximity. Therefore, Foundation trainees may only be motivated to complete TEL activities if they will be strongly rewarded (e.g. gaining additional credits) or punished (e.g. failing a course) as a result of an imminent task (e.g. an exam they must take tomorrow). By allowing little time to complete work, other learning behaviours (e.g. time spent on the task) may be compromised. TMT may serve to account for some of the learning behaviours that may be observed in a mandated ‘just-in-time’ learning context.
It is important to evaluate learning behaviours in order to identify those that may be suboptimal. This can help to improve education, since a lack of engagement may be associated with poor performance. This may be especially pertinent in unmonitored TEL environments, to ensure that the appropriate learning is undertaken and to inform future instructional design. Owing to ad hoc monitoring of users’ interactions with the system, we identified a need to evaluate exactly this. As such, we investigated how Foundation trainee doctors’ learning behaviours (e.g. time spent on task; frequency of task completion) were affected by temporal (e.g. time of year), module (e.g. the quantity of learning material), and individual (e.g. knowledge) factors for a range of mandatory TEL modules relating to prescribing and therapeutics.
Online learning for Foundation trainee doctors is commonly used across the UK to help prepare them for practice in various areas of healthcare practice. SCRIPT is an e-learning resource used by Foundation trainee doctors across various regions in England (www.safeprescriber.org). Within SCRIPT, there are currently 45 available modules, designed to educate Foundation trainee doctors about prescribing in a variety of patient populations and situations. Modules include topics such as Prescription Documentation, Prescribing in Renal Dysfunction, and Adverse Drug Reactions. The modules were developed by a multidisciplinary team of healthcare professionals and are updated on a regular basis. Each module comprises a set of multiple choice questions (‘pre-test’), the main body of learning content, a repeat set of identical questions presented in a random order (‘post-test’), and some suggested further reading. Following completion of the post-test, a certificate of completion can be viewed and downloaded by the user for their online learning portfolio (ePortfolio), which they are required to complete during the Foundation training programme.
In August 2011, it became mandatory for Foundation Year 1 (F1) doctors in the West Midlands to complete 16 specified modules within this first year of training. No other instruction about the order of module completion or time frame for completing modules was specified. Bi-annual progression review meetings are held between Foundation trainees and their clinical tutor in early March and early June, for which the trainee must have prepared specific items in their ePortfolio for submission and evaluation.
Through the SCRIPT e-Learning platform, it is possible to monitor individual user activity. It is possible to identify the date on which individual modules are completed (pre-test and post-test completion are monitored independently), the length of time spent engaging with the learning content, and both pre-test and post-test scores. Clinical tutors (for their tutees only) and administrators (for all users) are able to access this data.
Foundation Year 1 (F1) Mandatory SCRIPT Module Titles
Adverse Drug Reactions
Prescribing in Renal Dysfunction
Prescribing in Older Adults
Dosing and Calculation
Drug Allergy and Anaphylaxis
Prescribing in Infection
Rational Drug Choice
Taking a Safe and Effective Drug History
It is important to note that all medical graduates must achieve a specified standard in order to commence the Foundation training programme. Although it is expected that all users would have a similar level of competence prior to completing the modules, there may be some between-trainee variability. Our data capture therefore examined individual progression from pre- to post-test scores.
Owing to the level of skew in the distribution of the times taken to complete modules, the variable was log10 transformed prior to analysis. Since each doctor took multiple modules, it was not reasonable to assume that all instances in the data were independent. For this reason, the data were analysed using Generalised Estimating Equations (GEE), with an exchangeable correlation structure, in order to account for the within-doctor correlations in the times taken to complete modules. A range of temporal- and module-related factors were included as independent variables in this model, to test for associations with module completion time. The coefficients from this model were then anti-logged, and converted into estimates of the geometric mean module completion times, after accounting for the effects of the other factors in the model as well as the within-doctor correlations.
Factors relating to module content were then collated, and compared to the estimated averages from the GEE model using Spearman’s correlation coefficients. Variables with significant correlations were then entered into a linear regression model.
All analyses were performed using IBM SPSS 19 (IBM Corp. Armonk, NY), with p < 0.05 deemed to be indicative of statistical significance.
Distribution of modules over time
Figure 1b shows how the modules taken were distributed over the days of the week. The numbers were similar for Monday through Thursday, at around 1,500 modules on each day. Friday and Saturday were the days where least modules were taken, with totals of 878 and 1,133 respectively. However, Sunday was found to be the day that F1 doctors most commonly completed modules, with a total of 2,107.
Figure 1c shows the clustering of modules within the same calendar week. For each module completed, the total number of other modules completed in the same week was calculated. There were a number of instances where a large number of modules were completed in a batch during the same week. There were 49 instances of more than 10 modules being completed in the same calendar week, of which nine represented doctors completing all 16 of their modules within the same week. Of these, one trainee completed their modules in August 2013, one in October 2013, two in January 2014, one in February 2014 and four in May 2014.
Time taken to complete modules
The log-transformed times to complete modules closely followed a normal distribution, however there were more modules than expected that were completed quickly. The geometric mean time taken to complete modules was 28.9 min (95 % Confidence Interval (CI): 28.4–29.5), with 1,075 (10.5 %) modules being completed in under 10 min. The shortest time to complete a module was 38 s. These short times also resulted in an asymmetrical distribution, which contravened the assumptions of parametric analytical techniques. For this reason, modules that were completed in less than five minutes (N = 497, 4.8 %) were excluded from the analysis of module completion times. After excluding these cases, the geometric mean completion time of the remaining modules increased to 32.9 min (95 % CI: 32.5–33.4).
The GEE model demonstrated a moderate level of within-doctor correlation (coefficient = 0.484). After accounting for this, all of the factors considered (module; pre-module score; change in score from pre-test to post-test; weekday; number of modules completed per week; time period) were found to be significant at p < 0.001.
Figure 2b illustrates that, whilst the average time taken to complete modules was relatively constant across each week, there was a small increase over the weekend, with geometric means of 35 min for weekdays, 37.6 min on Saturday and 38.1 min on Sunday.
The correlation coefficients between the content-related factors, and the geometric mean times for each module from the GEE model were also examined. The only factor found to be significant was the module word count, with a Spearman’s correlation coefficient of 0.756 (p < 0.001). A linear regression model found that the average time taken to complete a module increased by 2.8 min per 1,000 words (95 % CI: 1.4–4.3, p = 0.001).
Using data from a content management system, we investigated Foundation trainee doctors’ learning behaviours when completing TEL modules for prescribing education. A number of results indicate that suboptimal learning behaviours were being exhibited by trainees, demonstrated by changes to the frequency of module completion and the time spent on learning over time, across modules, and individuals.
Regarding the frequency of module completion, peaks existed in February and June, which appeared to occur around the time of trainees’ biannual progression reviews with their clinical tutor. This suggests a large number of modules were being completed in response to an upcoming or recently completed review. As Trusts’ review periods may differ slightly depending on the number of trainees in the hospital and availability of both trainees and Clinical Tutors, this may also account for a prolonged peak in the increased frequency of completion. Furthermore, in the month prior to the second review period there was a rapid increase in the number of trainees who had completed all 16 of their mandatory modules. These findings support Temporal Motivation Theory , as the review would act as a salient and temporally proximal event leading to reward or punishment, meaning that trainees would experience increasing motivation to complete the modules leading up to the review. The reviews may also provide some explanation as to the behaviour of the nine trainees who completed all 16 mandatory modules within one week – of these cases, two trainees completed their modules in January, one completed them in February, and four completed them in May, which would have been just prior to their review meeting. This finding highlights the drawbacks of the flexibility of TEL, as it is not possible to ensure that modules are completed as they were designed and intended, or distributed throughout the year appropriately.
Of the 688 F1 trainees, 105 did not complete all of the mandatory modules. The modules must be completed before progressing into the Foundation Year 2 (F2) training; therefore it is unlikely that this was due to trainees simply not completing modules without good reason. For example, some trainees may have completed some of the modules at their undergraduate institution and therefore were exempt from completing these again in their F1 year. Furthermore, some trainees may have taken extended leave or left the Foundation training programme and therefore not completed all of the mandatory modules within the study period. It would be interesting to investigate these cases in more detail.
Despite the findings that many modules may have been completed in response to tutor reviews, some modules were more likely to be completed early in the year (e.g. Prescription Documentation, Taking a Safe and Effective Drug History). This may have been in response to trainees’ perceived educational needs, or based on individual Trusts’ requirements to complete these specific modules (which may be considered introductory) at the start of the year. Furthermore, modules are grouped together into seven sections (e.g. Principles of Prescribing; Prescribing in Special Circumstances) within the online programme. The two modules completed earlier in the year are located in the first section (Principles of Prescribing), which encompasses what may be considered the most basic material and is located at the top of the navigation pane. Therefore, they may have been completed earlier in the year based on their salience in the online learning environment, the relative ease of the content, or the necessity of the knowledge for trainees, which are factors we did not consider in our analyses.
It was also found that there was an increased frequency of module completion on Sundays compared to other days of the week, and that more time was spent on modules on Saturdays and Sundays. Trainees were utilising their free time during weekends, when they were presumably less busy, to dedicate time to TEL. This suggests a degree of intrinsic motivation for completing the learning, but also suggests that modules were not being completed during allocated study time during the working week.
Over the study period, the average time to complete a module declined. This may indicate that trainees are becoming more familiar with using the TEL programme, that they are becoming more confident and competent in their prescribing knowledge, and/or that the time and effort that they put in declines over the year. However, given that the average time for completing a module was 28.9 min and that 10.5 % of all modules were being completed in less than 10 min, it could be suggested that at least some of these cases can be attributed to a lack of time and effort rather than greater system or content expertise. The mean time is likely to be representative of the expected length of each module and significantly shorter times would seem to suggest a lack of engagement with the activities. The shortest time taken to complete a module (38 s) was surprising; there are no modules that are substantially short enough to warrant the interactions observed by some trainees. However it was found that it is physically possible to complete the modules in this time by rapidly navigating the learning system and avoiding interacting with optional module content. Clearly this does not represent doctors correctly using the educational tool. Furthermore, two troughs were evident in the average time taken to complete modules in late February and late May, which also correspond with the progression review dates. This provides evidence for a systematic decrease in the time taken to complete modules, which may be somewhat attributed to trainees rushing to complete numerous modules prior to their reviews. However, a greater word count was associated with more time being spent on the module, which provides some evidence that modules are in fact being utilised as intended.
Importantly, the results also highlighted that those doctors who spend little time completing one module tended to complete other modules quickly, and vice versa (within-doctor correlation from the GEE model of 0.484). Those trainees who spent less time completing modules had, on average, lower pre-test scores and showed smaller improvements in their knowledge from pre-test to post-test, compared to trainees who spent more time completing modules. The first of these findings may indicate that it is the more competent doctors who are more highly motivated to spend time on the TEL modules, or that those highly competent doctors also have better time management skills and are therefore able to spend more time working through each module. The second finding highlights that spending inadequate time completing the modules negates the purpose of the TEL programme, as prescribing knowledge (as measured by improvements on the multiple choice question test scores) is not improved to its full capacity. Those trainees spending more time completing modules are experiencing greater educational benefits, which may translate to the prescribing behaviours that are exhibited on the ward.
Based on these findings, it is possible to suggest that there is a need to regulate more closely when modules are completed throughout the year, so that modules are not being rapidly completed as a response to an upcoming review. Nonetheless, there is a need for the learning to remain flexible and based upon trainees’ needs at a particular time of year during the first year of the Foundation Programme, meaning it may be too challenging to mandate exactly when modules must be completed. Regarding the time taken to complete modules, there have already been some changes made to this system. For example, from June 2013, it became mandatory for users to view every page of a module, and users who take less than 10 min to complete a module are flagged on the e-learning platform, which their clinical tutor has the ability to view, Nonetheless, it is impossible to control how long a user actually spends on the learning, given that they are not required to physically attend a classroom-based session to complete the TEL. In the future, it may be possible to introduce a ‘code of conduct’ to make users aware of the expectations associated with SCRIPT, which may encourage Foundation trainees to utilise the TEL programme as intended.
Implications for educators
The findings from this study highlight the need for educators to devise methods to encourage optimal behaviours during the implementation of any TEL programme. Careful consideration should be given to the integration of such learning into a curriculum and the recommendations given as to how and when the learning should be completed.
If the learning is mandatory but is completed during the learner’s free time, educators should consider providing instruction as to how long the learning should take and when it should be completed, in a specified order (if appropriate) over a specified time frame (e.g. per week or per year). Alternatively, time could be allocated for the learning to be completed. It is also important to regularly monitor individual learners’ progress within the learning environment (i.e. whether they are completing learning at an appropriate time point and engaging with the TEL environment for a suitable length of time), from which feedback can be provided to the learner. In line with Temporal Motivation Theory , regular deadlines and active monitoring of progress may help to motivate learners to complete the learning in a timely manner.
As a result of our findings, we have implemented clearer guidance for postgraduate centre managers and trainers about integrating TEL with other training opportunities. We have also provided Clinical Tutors and Foundation trainee doctors with clearer guidance on the use of the programme, and we have suggested that tutors regularly review completeness of the e-Learning modules and incorporate discussion of prescribing competence in their progress reviews, thus discouraging adverse temporal completion.
Limitations and future research
In this study we only investigated mandatory TEL modules, and did not consider additional optional modules, which could provide additional or alternative explanations of learner behaviours. Second, it is only possible to hypothesise about Foundation trainees’ rationale for completing modules quickly. Furthermore, it is not possible to identify whether trainees are attending to the module content, even if they are technically engaged in the programme, or if they are completing the module alone. Therefore, the recorded time spent on a module can only be considered a surrogate measure of actual time spent engaged with the learning material. It is inevitable that work commitments will also influence trainee engagement with the programme. Whilst we make reference to temporal factors over the calendar period, due to the anonymity of trainees, we were unable to link specific doctors to rotations and therefore frequency of on calls. Lastly, learners’ engagement with TEL, particularly on screen activities and audio visual material, will vary according to their preferential learning styles. However, given the mandatory nature of the modules and lack of flexibility of the learning environment, it is difficult to take account of this variation.
In future research, to understand why trainees exhibit such learning behaviours, it would be interesting to further investigate whether there are any factors (e.g. personality, medical training background, clinical rotation and on-call commitment) shared among trainees who spend more or less time completing modules. In order to understand if there is a rationale for the patterns of behaviour observed in this study, it would also be useful to explore user attitudes towards the TEL programme, their interactions with the programme (e.g. completing modules with multiple or single interactions), and their motivation for its use. It may be important to gain an understanding of whether the mandatory nature of the learning had any effect on the observed trends in time taken, or the time of year that modules were completed.
Finally, it would be interesting to investigate whether the prescribing information provided to trainees in the TEL programme was retained and utilised in clinical practice, and if this had any effects on a larger scale (e.g. an overall reduction in medication errors within a hospital).
These results lead to the suggestion that some Foundation trainees were exhibiting unintended learning behaviours in this TEL environment. Such behaviours may be attributed to several factors, including a lack of time during the working week, upcoming review meetings, prior knowledge, and system factors such as module length. The findings from this evaluation can be used to guide future developments of this TEL programme and the rules and regulations surrounding its use. This may help to ensure that Foundation trainees are undertaking appropriate learning in a timely manner, which will ultimately help to improve their knowledge of prescribing and subsequent prescribing behaviours. These findings will also hopefully raise awareness of potential behavioural issues that may be encountered when designing mandated TEL programmes for continued medical education.
Ethics approval and consent to participate
Availability of data and materials
This article presents independent research funded by the Health Education England's West Midland team (HE-WM). The views expressed are those of the author(s) and not necessarily those of Health Education England, NHS England, or HE-WM.
This study was funded by Health Education England's West Midland team.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- UK Foundation Programme. The Foundation Programme: FAQs. 2014. http://www.foundationprogramme.nhs.uk/pages/home. Accessed 09 October 2014.
- Han WH, Maxwell SRJ. Are medical students adequately trained to prescribe at the point of graduation? Views of first year foundation doctors. Scottish Med J. 2006;51(4):27–32.View ArticleGoogle Scholar
- Watmough S, O'Sullivan H, Taylor D. Graduates from a traditional medical curriculum evaluate the effectiveness of their medical curriculum through interviews. BMC Med Educ. 2009;9:64.View ArticleGoogle Scholar
- Rothwell C, Burford B, Morrison J, Morrow G, Allen M, Davies C, et al. Junior doctors prescribing: enhancing their learning practice. Br J Clin Pharmacol. 2012;73(2):194–202.View ArticleGoogle Scholar
- Goldacre MJ, Lambert TW, Svirko E. Foundation doctors’ views on whether their medical school prepared them well for work: UK graduates of 2008 and 2009. Postgrad Med J. 2014;90(1060):63–8.View ArticleGoogle Scholar
- Van Hamel C, Jenner LE. Prepared for practice? A national survey of UK foundation doctors and their supervisors. Medical Teacher. 2015; 37(2):181–188.View ArticleGoogle Scholar
- Dornan T, Ashcroft D, Heathfield H, Lewis P, Miles J, Taylor D et al. An in depth investigation into causes of prescribing errors by foundation trainees in relation to their medical education: EQUIP study. London: General Medical Council; 2009.Google Scholar
- Keogh MJ, Findlay JM, Sithamparanathan S, Matheson D. Education in the Foundation Programme: what doctors are doing and why. British J Med Pract. 2010;3(10):306–10.Google Scholar
- Walsh K. The future of postgraduate training. Pan Afr Med J. 2014;19:333. doi:10.11604/pamj.2014.19.333.5555.View ArticleGoogle Scholar
- Zehry K, Halder N, Theodosiou L. E-Learning in medical education in the United Kingdom. Procedia Soc Behav Sci. 2011;15:3163–7. doi:10.1016/j.sbspro.2011.04.265.View ArticleGoogle Scholar
- Zhang D, Zhao JL, Zhou L, Nunamaker JF. Can e-learning replace classroom learning? Commun ACM. 2004;47(5):75–9. doi:10.1145/986213.986216.View ArticleGoogle Scholar
- Zhang D, Nunamaker JF. Powering e-learning in the new millenium: An overview of e-learning and enabling technology. Inf Syst Front. 2003;5(2):207–18.View ArticleGoogle Scholar
- Welsh ET, Wanberg CR, Brown KG, Simmering MJ. E-learning: emerging uses, empirical results and future directions. J Train Dev. 2003;7(4):245–58.View ArticleGoogle Scholar
- Baruah HK. Just-in-time learning. IOSR J Humanit Soc Sci. 2013;12(4):53–7.View ArticleGoogle Scholar
- Changiz T, Haghani F, Nowroozi N. Are postgraduate students in distance medical education program ready for e-learning? A survey in Iran. J Educ Health Promot. 2013;2:61. doi:10.4103/2277-9531.120862.View ArticleGoogle Scholar
- Harden RM. A new vision for distance learning and continuing medical education. J Contin Educ Health Prof. 2005;25(1):43–51. doi:10.1002/chp.8.View ArticleGoogle Scholar
- Scallan S. Education and the working patterns of junior doctors in the UK: a review of the literature. Med Educ. 2003;37:907–12.View ArticleGoogle Scholar
- Wutoh R, Boren SA, Balas EA. eLearning: a review of internet-based continuing medical education. J Contin Educ Heal Prof. 2004;24:20–30.View ArticleGoogle Scholar
- Maxwell S, Mucklow J. e-Learning initiatives to support prescribing. Br J Clin Pharmacol. 2012;74(4):621–31.View ArticleGoogle Scholar
- Lam-Antoniades M, Ratnapalan S, Tait G. Electronic continuing education in the health professions: an update on evidence from RCTs. J Contin Educ Health Prof. 2009;29(1):44–51.View ArticleGoogle Scholar
- Feng Y-J, Chang Y-T, Chang H-Y, Erdley WS, Lin C-H, Chang Y-J. Systematic review of effectiveness of situation e-learning on medical and nursing education. Worldviews Evid Based Nurs. 2013;10(3):174–83.View ArticleGoogle Scholar
- Steel P, König CJ. Integrating theories of motivation. Acad Manag Rev. 2006;31(4):889–913.View ArticleGoogle Scholar