Skip to main content
Download PDF

Quality specifications in postgraduate medical e-learning: an integrative literature review leading to a postgraduate medical e-learning model

BMC Medical Education volume 16, Article number: 168 (2016) Cite this article

Abstract

Background

E-learning is driving major shifts in medical education. Prioritizing learning theories and quality models improves the success of e-learning programs. Although many e-learning quality standards are available, few are focused on postgraduate medical education.

Methods

We conducted an integrative review of the current postgraduate medical e-learning literature to identify quality specifications. The literature was thematically organized into a working model.

Results

Unique quality specifications (n = 72) were consolidated and re-organized into a six-domain model that we called the Postgraduate Medical E-learning Model (Postgraduate ME Model). This model was partially based on the ISO-19796 standard, and drew on cognitive load multimedia principles. The domains of the model are preparation, software design and system specifications, communication, content, assessment, and maintenance.

Conclusion

This review clarified the current state of postgraduate medical e-learning standards and specifications. It also synthesized these specifications into a single working model. To validate our findings, the next-steps include testing the Postgraduate ME Model in controlled e-learning settings.

Peer Review reports

Background

E-learning plays a prominent role in conventional education, adult education, and medical training because of its flexibility, broad resource-sharing capacity, and cost-effective scalability [1]. E-learning has become central to medical education, and web technologies offer valuable new opportunities for both under- and postgraduate medical education [2]. E-learning also offers participants’ an advantage in that they can choose a comfortable and accessible place and time to study, which is important in postgraduate and continuous medical education [3].

There are many studies comparing e-learning methods. However, one of the problems is that the results of studies directly comparing technology-assisted education with traditional teaching often conflict and often do not demonstrate or propose best practices [4, 5]. Critical evaluation of the quality and efficiency of e-learning is warranted [6]. Therefore, there is a need to develop a consensus-based quality assurance standard for postgraduate medical e-learning [7, 8].

It is known that e-learning should be targeted to the needs of the specific audience (in this case postgraduates) [9, 10]. The success of e-learning programs has also been linked to the use of a theoretical framework or a learning theory [11]. Standards for e-learning exist and have been evaluated [12].

Although learning theories are broad and diverse, there is progressive agreement about the psychological fundamentals [13]. Previous research suggests that the constructivist approach (founded by Jean Piaget, among others) is compatible and appropriately designed for e-learning. According to this theory, humans are active learners and construct new knowledge based on prior experiences and interactions [14]. An example is problem-based learning, which has been shown to be effective in medical education [15, 16]. Another theory based on the constructivist approach is the cognitive load theory (CLT) developed by Sweller [17]. The constructivist approach also forms the foundation of the cognitive theory of multimedia learning, a well-evaluated learning theory developed by Mayer [18]. This theory has been specifically adjusted for e-learning, and is believed to provide a good basis for an e-learning standard [18]. These theories are not elaborated here. However, it is important to remember that learning theory, not technology, should guide the design and content of e-learning.

Standards for e-learning exist [12], but are often isolated to a specific sector of the industry for which they were developed [19]. There are many industrial standards in education, often published by organizations such as the American Council on Education and the European Association of Distance Teaching Universities [20]. Several organizations have made progress in developing international industrial standards for universities, including the Open e-Quality Learning Standards and the Leonardo DaVinci program [21, 22].

The most common formal standard in use is the ISO/IEC 19796-1 [23]. This standard was issued by the International Standardization Organization (ISO) in 2005, and contains the Reference Framework for the Description of Quality Approaches (RFDQ), a framework supported by the European Quality Observatory [24]. Stracke implemented this standard and concluded that it was not only the first quality standard for learning, education, and training, but was a valuable instrument for sustained quality development in e-learning [23]. Little tested two higher education quality standards or rubrics, in professional continuous nursing education in 2009 [25]. The College of Public Health Online Course Standards and the Quality Matters Peer Course Review Rubric were evaluated, and although they look hopeful, little is known about their actual effectiveness. The Quality Matters rubric is not publicly available, making evaluation even less accessible.

Existing literature fails to clarify the methods and strategies used to evaluate the quality of postgraduate medical e-learning. As Clark noted in his well-respected book E-Learning and the Science of Instruction, the target audience should be the first thing considered when designing an e-learning program [9, 10]. Although undergraduates and postgraduates may learn in the same way, adult learning theories suggest their learning strategies and capacities might be dissimilar. Therefore, we focused our review on our target audience: postgraduates and physicians bound to continuous medical education. Ellaway et al. began this process in 2008 by describing a two-part guideline for e-learning in medical education. Although the guideline was not specifically aimed at postgraduate medical education, it served as a foundation for further research [16, 26]. The previously mentioned standards and the Quality Matters Rubric look promising, but are all aimed at different target audiences [25]. The ISO standard (ISO/IEC 19796) covers a lot of domains, but lacks detail regarding their application [27].

It is difficult to establish quality control practices for e-learning materials [28], yet this is an important problem in postgraduate medical education where the quality of training directly influences patient care [29]. Therefore, as technological innovations reshape medical education institutions, the question of quality assurance is at the forefront of university leadership concerns worldwide [30]. In 2010, Cook et al. reiterated the primary importance of defining quality standards in medical education. In their Second Flexner Report, they identified standardization as a key goal for improving medical education [31]. We believe that it is important to establish a testable quality assurance model to improve the uptake of e-learning and motivate continuous medical learning [5]. This review will add to existing literature by providing an integrative literature review and a working model of quality assurance standards in postgraduate medical e-learning.

Methods

We performed an integrative review to identify and critically evaluate qualitative and quantitative literature associated with current postgraduate medical e-learning quality assurance. We used the updated integrative review methodology developed by Whittemore et al. [32]. This method consists of three steps: 1) a systematic search combined with at least one nonsystematic search method; 2) data evaluation comparing study models and quality scores; and 3) data analysis. During data analysis, we compared individual quality items, clustered them, searched for contrasts and comparisons, discerned patterns, and built an overview of the domains.

Definitions

E-learning

An approach to teaching and learning, representing all or part of the educational model applied, based on the use of electronic media and devices as tools for improving access to training, communication, and interaction and that facilitates the adoption of new ways of understanding and developing learning [33]. In practice, the definition includes any digital content made to teach and distributed physically or online.

Quality

To date, there is no consensus definition of quality. However, a high-quality product is generally defined as one that meets consumer-defined specifications, delights the consumer, consistently meets the standard that the producer has set for itself, and leads to customer satisfaction. Producers should be able to assure this quality [30, 33].

Postgraduate (and continuous) medical education

Any form of learning aimed at medical professionals who have graduated from formal training and residency, or used by medical professionals a part of continuous learning to maintain their competency and develop new knowledge [34].

Standards and specifications

A standard is a set of specifications that guide an e-learning author in developing an e-learning program. A specification is a specific item that is addressed within the standard [17].

Step 1: Systematic literature search

The primary search terms were distance learning (and all synonyms) [35] and quality (and all synonyms). We conducted the search on May 2, 2015. We searched ISI/Web of Knowledge, PubMed, EBSCO/Cinahl, EBSCO/PsycInfo, and EBSCo/ERIC (Table 1). Google Scholar was also searched, despite debate on its added value [36].

Table 1 Databases searched and corresponding search strings
Full size table

Inclusion criteria

Included articles were peer-reviewed and published in the English language between 1970 and May 2015. The articles had to describe and evaluate specific e-learning characteristics in postgraduate or continuous medical education. The search was kept broad and selection was made on postgraduate medical education manually after reading the titles, abstracts, and the full text (if necessary).

Exclusion criteria

We excluded dissertations, conference abstracts, and articles comparing e-learning with other forms of learning without describing the quality specifications used.

After selecting the titles and abstracts, we sampled 40 random abstracts for independent evaluation by a second (MW) and third author (FS). If there was no consensus, three authors participated (RL, MW, FS) in a discussion until consensus was reached on whether or not to include the article.

Step 2: Data evaluation

We aggregated the selected studies that met the inclusion criteria, flagging those with an unclear method. The aggregate was graded according to criteria adapted from the levels of evidence model of the Oxford Centre for Evidence-Based Medicine (Table 2). Our adaptation consisted of the addition of grade 6, indicating an unclear method. We flagged a study as unclear if it did not fall into any of the five existing categories.

Table 2 Adapted Oxford Centre for Evidence-Based Medicine levels of evidence
Full size table

We used the two-tier data evaluation strategy developed by Wittemore et al. to grade the quality of each item. Levels 1, 2, and 3 were identified as high quality (tier 1), and 4 and 5 as low quality (tier 2) [32]. We removed grade 6 studies from the analysis.

Step 3: Data analysis

After aggregating the studies and grading the papers, we analysed each item for common themes and contradictory findings. If items conflicted, we rejected the lower quality study in favor of the higher quality study. We categorized the items according to the ISO/IEC 19796-1 domains, and then generated new themes for items that did not fit the established domains.

Results

Primary selection and analysis

The literature search identified 10,732 articles. Searching Google Scholar gave us access to unanticipated databases such as the Emerald, IEXEE, and Editlib, as well as journals that were not registered with the other databases (International Journal of Information and Educational Technology, Journal of E-Learning and Knowledge Society, Applied Soft Computing, International Educational E-Journal, and Journal of Theoretical and Applied Information Technology). We then manually reviewed the identified studies.

After applying inclusion and exclusion criteria, 884 eligible titles remained. Eligible abstracts were reviewed and final selection criteria were refined as described in the Methods section.

In total, 36 articles met the final selection criteria (see Fig. 1), representing quality items in postgraduate medical education literature. Of these, 15 were original case reports and expert opinions; 13 were reviews of previous case reports and expert opinions; five were randomized controlled trials; and two were meta-analyses (Table 3). Collectively, these articles represented 16 high tier articles (Oxford Centre for Evidence-Based Medicine levels 1, 2, and 3) and 20 low tier articles (levels 4 and 5). The publication dates ranged from 1995 to 2015.

Fig. 1
figure 1

Flowchart of the literature search results

Full size image
Table 3 Articles used for the postgraduate medical education model identified in the literature search
Full size table

Sub-selection and analysis

After consolidating duplicates for quality specifications (440 in total), there were 72 unique quality specifications remaining. We categorized these specifications according to the seven ISO 19796 domains, combined them, and renamed them if necessary. This produced a model with six final domains, which we called the Postgraduate Medical E-learning Model (Postgraduate ME Model; Fig. 2). In the following paragraphs, we have defined the domains and provided examples from the literature.

Fig. 2
figure 2

Postgraduate medical e-learning specifications model

Full size image
  1. 1.

    Preparation: This step should be performed before designing and building the e-learning platform. Twelve articles described the importance of knowing the end users [23, 36–46]. Preparation is a two-step process. First, one must define the platforms used by different types of end users and their expectations of the e-learning platform. If the end users’ level of knowledge is known, the e-learning strategy can be designed appropriately. Special functionalities for individuals with learning disabilities can also be incorporated if applicable. Cook et al. described this step as a part of the needs analysis [10]. If possible, learning should be adapted to the audience’s motivational level, skills, and expectations. Curry et al. [47] and Olson et al. [20] emphasized the importance of selecting e-learning authors based on subject matter expertise. They suggested that academically qualified people must actively participate in development and training to improve e-learning. Four articles advised readers to allocate time to accurately budget and plan for the expected costs [23, 41, 44, 48]. Budgets and policy statements should reflect commitment to the program. Seven authors agreed on the positive effect of planning, describing objectives and expectations, and placing these in a timeline to maximize efficiency [7, 23, 36, 37, 39, 40, 49]. Cook et al. [50] identified potential barriers to implementation, and Olson et al. [20] advised readers to develop a marketing plan for reaching end users. Curry [47], Wong [51], and Sowan [39] advocated efficient e-learning that saves the user time, compared with other forms of learning.

  2. 2.

    Design: This depends on a series of difficult choices that include the types of hardware and software that best fit end-users’ needs. Although, hardware and software are very different, most articles described these in one section. In several articles (n = 5), authors stressed the importance of reliability and emphasized that the combined hardware and software system should be tested and iterated to meet end-user design requirements [7, 23, 39, 42, 49]. Testing should include different browsers and different monitors with varying resolutions, as well as different hand-held devices if a mobile-based platform is developed.

    • Most (n = 14) articles discussing software design focused on ready access to the e-learning platform. Access should be fast, easy to find, and always available; the platform should have a short loading time, and should provide reliable online access to all external links [7, 36-40, 45, 46, 48, 49, 52, 53, 70]. A secure connection is also important to support the privacy and legal requirements, copyright rules, and intellectual property issues [23, 39, 44, 49]. Lewis et al. [52] urged readers to be mindful of the basics such as grammar and spelling. Cook et al. [6] advised piloting e-learning websites before publishing them, and Bangert [46] advised the use of a variety of learning environments. Yavner et al. [53] wanted to give users maximum control over navigation, an approach that Mayer later challenged [54]. Navigation and layout are important design elements. Navigation should be user friendly and intuitive with a “less is more” design strategy. The design should be visually pleasing, adhere to the principles of excellent website design, and use reusable learning objects for a standard look and feel [37, 45]. Four articles provided a minimal set of pages/functionalities: glossary of terms, frequently asked questions, concept map, references cited, abbreviation key, and labelled diagrams [36, 45, 55, 56].

    • Olson et al. [20] and Lau [42] suggested choosing appropriate, intuitive, and user-friendly hardware devices that advise the learner of the skills and technology required for success. Desired learning outcomes should drive the choice of technology. Khoiny [57], Garde et al. [43], and Mhouti et al. [58] described the importance of the environment where the e-learning is used. The physical setting should be a stimulating and motivational place to learn.

  3. 3.

    Communication: This includes all forms of internal user-oriented communication and external expert-oriented communication. Articles included in our review provided descriptions of several forms of communication, including: (i) communication about the program that informs the user about learning objectives, costs, and support options; (ii) communication that allows users to contact the faculty/course authors; and (iii) communication between users as they collaborate on coursework. Several articles (n = 8) emphasized the second form of communication, recommending opportunities for live interaction with experts/authors, possibly in a group context [38, 41, 43, 45, 55, 59]. The credibility of the authors should be well established, and should include the authors’ credentials and disclosure of sponsors and conflicts of interest.

  4. 4.

    Content: This is the aggregate material used to build the lessons (e.g., words, images, videos). Content was a central theme in the articles we reviewed. Problem-based learning was favoured by 13 articles as the best way to incorporate motivation and better understanding [7, 15, 40, 43, 46–48, 52, 56–60]. Introducing interactivity to encourage higher-order thinking was also important (n = 16 articles) [23, 36, 39–41, 45–47, 51, 52, 56–59, 61]. The user should be provided with time and impetus to learn with as little stress as possible and made responsible for the learning process to create a feeling of belonging, and the platform should provide learning exercises [7, 41, 46, 48, 61]. Learning modules should end with summaries, consist of short paragraphs, state a timeline, and use milestones [23, 36, 39, 41, 43, 44, 46, 47, 49, 50, 59]. Lewis et al. [52] emphasized the use of educational standards, and most other articles suggested using cognitive load principles. Cognitive load principles are described in the Discussion. Bluestone et al. [61] advocated for reminders in e-learning systems, and Friedlander et al. [62] and Cook [59] suggested incorporating rewards and reinforcements to maintain motivation.

  5. 5.

    Assessment: This refers to all of the possible ways to test end users and formalize their knowledge gain. Almost all of the articles prioritized assessment and feedback on performance. Seventeen articles described assessment as most effective when used in a direct, continuous, and personalized way [7, 15, 23, 36–38, 40, 41, 45–47, 50–52, 56, 58, 63]. Self-assessment was also an important part of learning [7, 23, 36, 39, 40, 43–45, 47, 54, 57, 59]. Additionally, Lau [42], Short et al. [64], and Wong et al. [51] stressed the importance of continuing to provide the e-learning software and associated tools after the end of the course. Cook et al. focused on the importance of assessing user experience and satisfaction [50]. Assessment is also a way of evaluating learning outcomes [10].

  6. 6.

    Maintenance: This includes the steps taken to avoid the loss of knowledge after a user finishes the e-learning process. Maintenance also includes reliable long-term access to the platform to allow an end user to return to the platform. Maintenance-related articles (n = 10) described the importance of evaluating a platform’s user experience, effectiveness, usability, and cost [7, 23, 36–38, 40, 44–46, 59]. Technical maintenance included protecting and verifying hyperlinks. Several articles (n = 6) emphasized that modifications to e-learning programs should be enabled and updated, and those proven to be unsuccessful removed [7, 23, 37, 38, 44, 45]. E-learning developers should also estimate the reusability and sustainability of new platforms, as these factors are important in the platform’s maintainability [10].

Discussion

We identified 72 features as important in postgraduate medical e-learning, and grouped these into six domains. The domains also provided a model framework for educators involved in drafting e-learning strategies or evaluating e-learning initiatives. The content domain is the most widely described and discussed domain of the model, and we discuss this in detail below followed by the limitations of our review.

The content domain

Perhaps the most important part of e-learning is the content, which was emphasized in all articles reviewed. Content is the heart of e-learning, and the design merely delivers content. A common pitfall is developing e-learning simply for the sake of using a new technology. Instead of making e-learning technology-centred, developers should subscribe to a learning theory to ensure the design is guided by pedagogical principles [14]. Mayer described the cognitive theory of multimedia learning, based the CLT [17]. CLT aims to develop instructional design guidelines based on a model of human cognitive architecture [65]. CLT states that working memory is limited in its capacity to selectively attend to and process incoming sensory data. This theory is concerned with the way in which a learner focuses and uses cognitive resources during learning and problem solving. It suggests that for an instruction to be effective, it must be designed in a way that does not overload the mind’s capacity for processing information [40]. Based on this theory, Mayer defined a set of principles, which describe the effect of different design techniques on learning. These principles form guidelines for using multimedia in a learning environment. In the content domain, we found 20 of 36 articles tested or recommended one or more of Mayer’s principles. Because the current literature pays so much attention to these principles, we considered them to be the foundation of the content domain [18].

Not all authors agreed with all of these principles. For example, Yavner et al. [53] proposed giving the learner maximum control, in contradiction to Mayer’s assertion that giving control to learners yields no benefit because learners may have too many options [17]. Curry et al. [47], Mhouti et al. [58], and Posel et al. [66] also highlighted the importance of learner-centred e-learning and supported individualized, rather than standards-based e-learning. Therefore, consensus on the principles of the content of e-learning is lacking.

Limitations

The major limitations of this review pertain to the methods. We performed an integrative review instead of a systematic review or meta-analysis, because the majority of published studies related to postgraduate medical education did not meet the parameters required for a systematic review. The major limitation of integrative reviews is the potential for bias from the inclusion of non-peer-reviewed information or lower-quality studies. Although Cook et al. [50] conducted a meta-analysis, it was limited to a few domains in e-learning. However, the literature from which the authors drew their conclusions was largely consensus-based.

The second limitation was the fast-changing technology that threatens to render our results obsolete. E-learning is rapidly changing the landscape of medical education and is developing faster than research can evaluate it [67]. This pace of change is a limitation of our review because research is always one step behind technology [67]. Examples of these fast changes are two disruptive innovations in medical education: Massive Open Online Courses and social media on mobile devices [68]. Both impact on e-learning and might dramatically change the education landscape [69]. In this landscape, it is almost impossible to evaluate an innovation properly before it is already outdated. In addition, social media is expected to become important to the collaboration domain of e-learning [70].

Despite these limitations, we believe that our six-domain Postgraduate ME Model will generate discussion and increase the quality of new e-learning courses. Our e-learning model could be interpreted as a general framework rather than postgraduate-specific, although we have not provided evidence to extend it to other settings, due to the limitation in our search strategy. We have limited the search, to target our audience as good the literature allowed us. Clark et al. clearly states that the target audience should be as specific as possible [9, 10]. The articles represented in this review were selected with focus on postgraduate learners; further analysis will be required to determine the applicability of our Postgraduate ME Model to other audiences. Even if e-learning developers reject our model, we feel that it is better to have reasons for not using a model than to have no model at all [71].

Conclusion

In summary, our Postgraduate ME Model aimed to provide a practical framework that can be used to build postgraduate medical e-learning programs that are learner centred, interactive, well planned and designed, based on cognitive load theory, and easy to maintain. E-learning should be about the learner, not the technology. Our proposed model may guide e-learning designers who are developing quality e-learning targeted to postgraduates in medicine. Our six-domain model is unique in that it combines the technical requirements from industry standards with the critical aspects of content and interaction from learning theories.

The next step is for research to validate these domains with international experts to determine if they are beneficial to postgraduate real-world e-learning. It would be interesting to know if postgraduate e-learning experts agree with our description of qualitative e-learning, based on their experience. Another important question is whether a model such as this could actually be used in practice when developing e-learning platforms. Ultimately, we would like to know whether e-learning based on our Postgraduate ME Model will reproducibly improve learner motivation.

Abbreviations

CLT, Cognitive load theory; ISO, International standardization organization; Postgraduate ME Model, Postgraduate Medical Education Model

References

  1. Allen IE, Seaman J. Going the distance: Online education in the United States, 2011. The Sloan Consortium: Organizations Committed to Quality Online Education; 2011.

  2. Sandars J, Schroter S. Web 2.0 technologies for undergraduate and postgraduate medical education: an online survey. Postgrad Med J. 2007;83(986):759–62.

    Article  Google Scholar 

  3. Curran VR, Fleet L. A review of evaluation outcomes of web-based continuing medical education. Med Educ. 2005;39(6):561–7.

    Article  Google Scholar 

  4. Jwayyed S, Stiffler KA, Wilber ST, Southern A, Weigand J, Bare R, Gerson LW. Technology-assisted education in graduate medical education: a review of the literature. Int J Emerg Med. 2011;4(1):51-64.

    Article  Google Scholar 

  5. Childs S, Blenkinsopp E, Hall A, Walton G. Effective e-learning for health professionals and students-barriers and their solutions. A systematic review of the literature-findings from the HeXL project. Health Info Libr J. 2005;22 Suppl 2:20–32.

    Article  Google Scholar 

  6. Cook DA, Dupras DM. A practical guide to developing effective web-based learning. J Gen Intern Med. 2004;19(6):698–707.

    Article  Google Scholar 

  7. Maggio LA, Tannery NH, Chen HC, Cate O, O’Brien B. Evidence-Based Medicine Training in Undergraduate Medical Education. Acad Med. 2013;88(7):1022–8.

    Article  Google Scholar 

  8. Ruiz JG, Mintzer MJ, Leipzig RM. The impact of E-learning in medical education. Acad Med. 2006;81(3):207–12.

    Article  Google Scholar 

  9. Clark RC, Mayer R.E. E-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning. 3rd ed. San Francisco: Pfeiffer; 2011.

  10. Cook DA, Ellaway RH. Evaluating technology-enhanced learning: A comprehensive framework. Med Teach. 2015;37(10):961–70.

    Article  Google Scholar 

  11. Hung DWL, Chen D-T. Situated Cognition, Vygotskian Thought and Learning from the Communities of Practice Perspective: Implications for the Design of Web-Based E-Learning. EMI Educ Media Int. 2001;38(1):3–12.

    Article  Google Scholar 

  12. Marshall S. E-learning standards: Open enablers of learning or compliance strait jackets. Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference. 2004:596-605.

  13. Jonassen DH. Learning to solve problems: An instructional design guide. 1st ed. San Francisco: Pfeiffer; 2004. p. 1-10.

  14. Koohang A, Harman K. Open source: A metaphor for e-learning. Inf Science J. 2005;8:75–86.

    Google Scholar 

  15. Schmidt HG, Van der Molen HT. Constructivist, Problem-Based Learning Does Work: A Meta-Analysis of Curricular Comparisons Involving a Single Medical School. Educ Psychol. 2009;44(4):227–49.

    Article  Google Scholar 

  16. Ellaway R, Masters K. AMEE Guide 32: e-Learning in medical education Part 1: Learning, teaching and assessment. Med Teach. 2008;30(5):455–73.

    Article  Google Scholar 

  17. Sweller J. Cognitive load during problem solving: Effects on learning. Cogn Sci. 1988;12(2):257–85.

    Article  Google Scholar 

  18. Mayer RE. The Cambridge handbook of multimedia learning. 2nd ed. Cambridge: Cambridge University Press; 2014.

  19. Singh H, Reed C. Demystifying e-learning standards. Ind Commer Train. 2002;34(2):62–5.

    Article  Google Scholar 

  20. Olson CA, Shershneva MB. Setting quality standards for web-based continuing medical education. J Contin Educ Health Prof. 2004;24(2):100–11.

    Article  Google Scholar 

  21. Marques CG, Noivo J, Verissimo M. e-QUAL: e-Learning with Quality. Proposal for an Evaluation Model on the Quality of e-Learning Courses. In Mendes AJ, Pereira I, Costa R, Editors. Computers and Education. Towards Educational Change and Innovation. London: Springer; 2008. p. 83-90.

  22. Caramihai M, Severin I. eLearning Tools Evaluation based on Quality Concept Distance Computing. A Case Study. Int J Innov Res Sci Eng Technol. 2009;3(5):438–42.

    Google Scholar 

  23. Stracke CM. Quality Development and Standards in e-Learning: Benefits and Guidelines for Implementations. Proceedings of the ASEM Lifelong Learning Conference: e-Learning and Workplace Learning Bangkok (Thailand). 2009.

  24. Ehlers UD, Goertz L, Hildebrandt B, Pawlowski JM. Quality in e-learning: use and dissemination of quality approaches in European e-learning: a study by the European Quality Observatory. Cedefop Panorama series. 2006 (116).

  25. Little BB. The Use of Standards for Peer Review of Online Nursing Courses: A Pilot Study. J Nurs Educ. 2009;48(7):411–5.

    Article  Google Scholar 

  26. Masters K, Ellaway R. e-Learning in medical education Guide 32 Part 2: Technology, management and design. Med Teach. 2008;30(5):474–89.

    Article  Google Scholar 

  27. Pawlowski JM. The quality adaptation model: adaptation and adoption of the quality standard ISO/IEC 19796-1 for learning, education, and training. J Educ Techno Soc. 2007;10(2):3–16.

    Google Scholar 

  28. Zhang W, Cheng YL. Quality assurance in e-learning: PDPP evaluation model and its application. Int Rev Res Open Dis. 2012;13(3):66–82.

    Google Scholar 

  29. Chan CH, Robbins LI. E-Learning Systems: Promises and Pitfalls. Acad Psychiatry. 2006;30(6):491–7.

    Article  Google Scholar 

  30. Abdous M. E‐learning quality assurance: a process‐oriented lifecycle model. Qual Assur Educ. 2009;17(3):281–95.

    Article  Google Scholar 

  31. Cooke M, Irby DM, O’Brien BC. Educating physicians: a call for reform of medical school and residency. J1st ed.  San Francisco: Jossey-Bass Imprint; 2010.

  32. Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs. 2005;52(5):546–53.

    Article  Google Scholar 

  33. Harvey L, Green D. Defining Quality. Assess Eval High Educ. 2006;18(1):9–34.

    Article  Google Scholar 

  34. Editor. ACCME about us. 2013. http://www.accme.org/about-us. Accessed 11 May 2014.

  35. Al-Jewair TS, Azarpazhooh A, Suri S, Shah PS. Computer-assisted learning in orthodontic education: a systematic review and meta-analysis. J Dent Educ. 2009;73(6):730–9.

    Google Scholar 

  36. Mastrangelo G, Fadda E, Rossi CR, Zamprogno E, Buja A, Cegolon L. Literature search on risk factors for sarcoma: PubMed and Google Scholar may be complementary sources. BMC Res Notes. 2010;3:131.

    Article  Google Scholar 

  37. Alexander LK, Horney JA, Markiewicz M, MacDonald PDM. 10 Guiding principles of a comprehensive Internet-based public health preparedness training and education program. Public Health Rep. 2010;125 Suppl 5:51–60.

    Google Scholar 

  38. Gordon M, Chandratilake M, Baker P. Low fidelity, high quality: a model for e‐learning. Clin Teach. 2013;10(4):258–63.

    Article  Google Scholar 

  39. Sowan AK, Jenkins LS. Use of the seven principles of effective teaching to design and deliver an interactive hybrid nursing research course. Nurs Educ Perspect. 2013;34(5):315–22.

    Article  Google Scholar 

  40. Sorden SD. A cognitive approach to instructional design for multimedia learning. Informing Sci J. 2005;8:263–79.

    Google Scholar 

  41. Raymond M, Iliffe S, Pickett J. Technology-enhanced learning. Educ Prim Care. 2012;23(5):458–9.

    Article  Google Scholar 

  42. Lau KHV. Computer-based teaching module design: principles derived from learning theories. Med Educ. 2014;48(3):247–54.

    Article  Google Scholar 

  43. Garde S, Heid J, Haag M, Bauch M, Weires T, Leven FJ. Can design principles of traditional learning theories be fulfilled by computer-based training systems in medicine: The example of CAMPUS. Int J Med Inform. 2007;76(2-3):124–9.

    Article  Google Scholar 

  44. Casebeer L, Kristofco RE, Strasser S, Reilly M, Krishnamoorthy P, Rabin A, Zheng S, Karp S, Myers L. Standardizing evaluation of on-line continuing medical education: physician knowledge, attitudes, and reflection on practice. J Contin Educ Health Prof. 2004;24(2):68–75.

    Article  Google Scholar 

  45. Maor D, Volet S. Interactivity in professional online learning: A review of research based studies. Int J Educ Techno. 2007;23(2):269–90.

    Google Scholar 

  46. Bangert AW. The seven principles of effective teaching: a framework for designing, delivering, and evaluating an internet-based assessment course for nurse educators. Nurse Educ. 2005;30(5):221–5.

    Article  Google Scholar 

  47. Curry M, Smith L. Twelve tips for authoring on-line distance learning medical post-registration programmes. Med Teach. 2005;27(4):316–21.

    Article  Google Scholar 

  48. Knight CL, Sakowski HA, Houghton BL, Laya MB, DeWitt DE. Developing a peer review process for web-based curricula: minting a new coin of the realm. J Gen Intern Med. 2004;19(5):594–8.

    Article  Google Scholar 

  49. Chang H-C, Liu C-F, Hwang H-G. Exploring Nursing E-Learning Systems Success Based on Information System Success Model. Comput Inform Nurs. 2011;29(12):741–7.

    Article  Google Scholar 

  50. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Instructional design variations in internet-based learning for health professions education: a systematic review and meta-analysis. Acad Med. 2010;85(5):909–22.

    Article  Google Scholar 

  51. Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ. 2010;10(1):12.

    Article  Google Scholar 

  52. Lewis KO, Cidon MJ, Seto TL, Chen H, Mahan JD. Leveraging e-Learning in Medical Education. Curr Probl Pediatr Adolesc Health Care. 2014;44(6):150–63.

    Article  Google Scholar 

  53. Yavner SD, Pusic MV, Kalet AL, Song HS, Hopkins MA, Nick MW, Ellaway RH. Twelve tips for improving the effectiveness of web-based multimedia instruction for clinical learners. Med Teach. 2015;37(3):239–44.

    Article  Google Scholar 

  54. Mayer RE. Applying the science of learning to medical education. Med Educ. 2010;44(6):543–9.

    Article  Google Scholar 

  55. Minasian-Batmanian LC. Guidelines for developing an online learning strategy for your subject. Med Teach. 2002;24(6):645–7.

    Article  Google Scholar 

  56. Davids MR, Chikte UME, Halperin ML. Effect of improving the usability of an e-learning resource: a randomized trial. Adv Physiol Educ. 2014;38:155–60.

    Article  Google Scholar 

  57. Khoiny FE. Factors that contribute to computer-assisted instruction effectiveness. Comput Nurs. 1995;13(4):165–8.

    Google Scholar 

  58. El Mhouti ANA, Erradi M. Development of a Tool for quality assessment of digital learning resources. I J Comp Appl. 2013;64:27–31.

    Google Scholar 

  59. Cook DA. Revisiting Cognitive and Learning Styles in Computer-Assisted Instruction. Acad Med. 2012;87(6):778–84.

    Article  Google Scholar 

  60. Mounsey A, Reid A. A randomized controlled trial of two different types of web-based instructional methods: One with case-based scenarios and one without. Med Teach. 2012;34(9):654–8.

    Article  Google Scholar 

  61. Bluestone J, Johnson P, Fullerton J, Carr C, Alderman J, BonTempo J. Effective in-service training design and delivery: evidence from an integrative literature review. Hum Resour Health. 2013;11:51.

    Article  Google Scholar 

  62. Friedlander MJ, Andrews L, Armstrong EG, Aschenbrenner C, Kass JS, Ogden P, Schwartzstein R, Viggiano TR. What can medical education learn from the neurobiology of learning? Acad Med. 2011;86(4):415–20.

    Article  Google Scholar 

  63. Issa N, Schuller M, Santacaterina S, Shapiro M, Wang E, Mayer RE, DaRosa DA. Applying multimedia design principles enhances learning in medical education. Med Educ. 2011;45:818–26.

    Article  Google Scholar 

  64. Short SED, Guillemette J-M, Duncan AM, Kirby F. Defining quality criteria for online continuing medical education modules using modified nominal group technique. J Contin Educ Health Prof. 2010;30(4):246–50.

    Article  Google Scholar 

  65. Van Merrienboer JJG, Sweller J. Cognitive load theory in health professional education: design principles and strategies. Med Educ. 2010;44(1):85–93.

    Article  Google Scholar 

  66. Posel N, Fleiszer D, Shore BM. 12 Tips: Guidelines for authoring virtual patient cases. Med Teach. 2009;31(8):701–8.

    Article  Google Scholar 

  67. Triola MM, Huwendiek S, Levinson AJ, Cook DA. New directions in e-learning research in health professions education: Report of two symposia. Med Teach. 2012;34(1):15–20.

    Article  Google Scholar 

  68. Mehta NB, Hull AL, Young JB, Stoller JK. Just Imagine. Acad Med. 2013;88(10):1418–23.

    Article  Google Scholar 

  69. Liyanagunawardena TR, Williams SA. Massive open online courses on health and medicine: review. J Med Internet Res. 2014;16(8):e191.

    Article  Google Scholar 

  70. Hillman T, Sherbino J. Social media in medical education: a new pedagogical paradigm? Postgrad Med J. 2015;91(1080):544–5.

    Article  Google Scholar 

  71. Casimiro L, MacDonald CJ, Thompson TL, Stodel EJ. Grounding theories of W(e)Learn: A framework for online interprofessional education. J Interprof Care. 2009;23(4):390–400.

    Article  Google Scholar 

  72. Kim P, Eng TR, Deering MJ, Maxfield A. Published criteria for evaluating health related web sites: review. BMJ. 1999;318:647–9.

    Article  Google Scholar 

  73. Sekikawa A, Aaron DJ, Acosta B, Sa E, LaPorte RE. Does the perception of downloading speed influence the evaluation of web-based lectures? Public Health. 2001;115:152–6.

    Article  Google Scholar 

  74. Doyle DJ. Web page evaluation in medical education. Can J Anaesth. 2002;49:997.

    Article  Google Scholar 

  75. Jha V, Duffy S. “Ten golden rules” for designing software in medical education: results from a formative evaluation of DIALOG. Med Teach. 2002;24:417–21.

    Article  Google Scholar 

  76. Shaw T, Barnet S, Mcgregor D, Avery J. Using the Knowledge, Process, Practice (KPP) model for driving the design and development of online postgraduate medical education. Med Teach. 2015;37(1):53–8.

    Article  Google Scholar 

Download references

Acknowledgements

We wish to thank all authors of the articles used for our study, for their great and inspiring work.

Funding

There was no funding for this study.

Availability of data and materials

The dataset supporting the conclusions of this article is available in the Postgraduate ME Model repository, http://ideact.nl/memodel.

Authors’ contributions

RL participated in the study design, search, data analysis, and the learning model design. MW was the second reviewer of the search results, drafted the manuscript, and participated in the data evaluation. EN revised the manuscript, added the background data, and participated in the interpretation of results. JK designed the search strings and coordinated the search itself. FS participated in the study design, coordinated the study, was the third reviewer of the data, and revised the manuscript. All authors read and approved the final manuscript.

Authors’ information

R.A. de Leeuw, MD, is a PhD student at the Athena Institute for Transdisciplinary Research of the VU University in Amsterdam, the Netherlands, and senior house officer in obstetrics and gynecology. His primary interest is digital education innovations and minimally invasive surgery.

M. Westerman, MD, PhD, is a senior medical education researcher at the VUMC School of Medical Sciences in Amsterdam, the Netherlands, and resident in internal medicine. His main research interests are transitions within medical education, professional identity development, generic competencies, and postgraduate medical education.

E. Nelson, MD, PhD, is a physician-scientist at Stanford University School of Medicine. Together with R.A. de Leeuw and collaborators, he co-developed the Cholera Outbreak Training and Shigellosis Program. He is now working on digital innovations for e-learning and novel android-based methods to combat infectious disease outbreaks.

J.C.F. Ket is a medical information specialist at the Medical Library at VU University in Amsterdam, the Netherlands. He specializes in search strategies and methodology variations for reviews and literature questions.

F. Scheele, MD, PhD, is Professor in Health Systems, Innovation, and Education at the Athena Institute for Transdisciplinary Research at the VU University in Amsterdam, the Netherlands. His expertise is in postgraduate education, curriculum development, and social responsiveness.

Competing interests

The authors report no competing interests, financially or otherwise.

Consent for publication

Not applicable.

Ethical approval and consent

Not applicable.

Author information

Authors and Affiliations

  1. Athena Institute for Trans-disciplinary Research, VU University Amsterdam, De Boelelaan 1118, Amsterdam, 1081 HZ, The Netherlands

    R. A. De Leeuw & F. Scheele

  2. VUmc, School of Medical Sciences, Amsterdam, The Netherlands

    M. Westerman & F. Scheele

  3. Division of Pediatric Infectious Diseases, Department of Pediatrics, Stanford University School of Medicine, Stanford, USA

    E. Nelson

  4. VU University Amsterdam, University Library, Amsterdam, The Netherlands

    J. C. F. Ket

Authors
  1. R. A. De Leeuw
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. Westerman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. E. Nelson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. J. C. F. Ket
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. F. Scheele
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to R. A. De Leeuw.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

De Leeuw, R.A., Westerman, M., Nelson, E. et al. Quality specifications in postgraduate medical e-learning: an integrative literature review leading to a postgraduate medical e-learning model. BMC Med Educ 16, 168 (2016). https://doi.org/10.1186/s12909-016-0700-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12909-016-0700-7

Keywords

BMC Medical Education

ISSN: 1472-6920

Contact us