Integrated online formative assessments in the biomedical sciences for medical students: benefits for learning

Assessment has sufficiently powerful effects on learning to be the de facto curriculum [1]. This includes not only what is learnt, but also students' approaches to learning [2–4]. Formative assessments are designed for the purpose of giving feedback on performance and suggestions for improvement, and are intended to promote students' learning [5, 6]. Formative assessments that provide timely, relevant and supportive feedback (not just grades) can contribute to improved learning outcomes [7]. In contrast, summative assessments are predominantly utilized for grading and certification at the end of a period of study, often without providing feedback to students on their performance. Indeed, one of the major weaknesses of most modern higher education programs, as evidenced by course evaluation surveys, is failure to provide adequate feedback to students on their learning [8]. It should be noted that the provision of diagnostic and remedial feedback has been found to be one of the most potent influences on student achievement [9]. If the purpose of assessment is to foster better learning outcomes, it could be argued that formative assessment is the most important assessment practice [10].

Paper-based formative assessments have a number of limitations [11]: students must be gathered together and invigilated; individualized feedback is time-consuming, and might not be feasible with large class sizes [12]; and analysis of question reliability and validity can be tedious. In contrast, Web-based formative assessments offer clear advantages for students: immediacy of feedback; flexibility in time and place of undertaking the assessment; feedback can provide links to learning resources, thereby providing motivation to study; opportunity for repetition; and interactivity [11]. Furthermore, a comparative study has reported that online formative assessments might be of greater benefit for learning than paper-based equivalents [13]. These are persuasive arguments for moving from paper-based to online formative assessments.

Web-based formative assessments also support equity and inclusiveness by allowing students to attempt each assessment anonymously on multiple occasions, at any time, and from virtually anywhere. This permits students with family responsibilities and work commitments to access the assessments at times that are most convenient for them. Many students, particularly those from culturally and linguistically diverse backgrounds, fear embarrassment if found to be in error, which might inhibit their propensity to clarify misconceptions directly with a member of academic staff. Online formative assessments provide a safe environment, where trial and error is permitted.

Although online formative assessments such as quizzes and practice examinations are becoming increasingly common in higher education, there is little formal evidence of their educational effectiveness and available data are both contradictory and inconclusive [12]. Nevertheless, such assessments have proved to be popular with pre-clinical medical students [11, 14], medical students in clinical attachments [15], students of dentistry [16, 17], as well as medical specialist trainees [18]. Thus in modern self-directed medical curricula, formative assessments may be perceived as "a safety net in a self-directed learning course" [5]. However, the potential of these assessments to assist learning by postgraduate trainees has not yet been fully exploited.

We evaluated the impact of online formative assessments on learning by the cohort of students enrolled in the initial 2 years of our undergraduate Medicine program [19] in 2006 and 2007. This employs vertically integrated scenario-based learning for first and second year students during Phase 1, which is comprised of a sequence of nine 8-week courses. Following an introductory Foundations course, these are based on domains related to the life cycle: Beginnings Growth and Development; Health Maintenance; Ageing and Endings; as well as Society and Health. All courses are interdisciplinary: biomedical sciences are integrated with one another; with the social and psychological sciences; and with early clinical experience. Learning is assessed in cross-disciplinary end of course (EOC) written and practical examinations, which are aligned with desired graduate capabilities and support integrated learning [20]. For the purposes of this study, courses in 2006 and 2007 (excluding Foundations, which has no summative assessment) were numbered 1 to 7 in chronological order.

Our data indicate that the online formative assessments in Phase 1 of our undergraduate Medicine program have been effective in promoting learning by students. Students who participated in formative assessments were likely to achieve higher marks in EOC examinations. Better performance in each of the formative assessments was also consistently associated with higher marks in the respective EOC examinations. There was a trend, although not consistent across all courses, for first year students to derive greater benefit from the formative assessments at the commencement of their program, consistent with the notion that such assessments provide a "safety net" for novices in student-centred learning [5].

We believe this report provides much-needed evidence of a quantifiable effect of online formative assessments on learning. Our findings are in contrast to two recent studies in related settings. The first study demonstrated the value of online formative quizzes in improving preparation and participation in classes [23], but reported no effect on summative examination results. The second study was a randomized control trial of online formative assessments for medical students in clinical clerkships, which found no positive effect on learning [24]. It should be noted that the latter investigation, in contrast to our study, failed to gain significant numbers of participating students.

Our findings also contrast with reports suggesting that online formative assessments utilising objective items such as multiple choice questions have no effect on student learning outcomes [23, 25], or even a negative effect on learning [26]. The authors of the latter study asserted that multiple choice questions may be unsuitable for formative assessments, because the "lures" or distractors create "false knowledge" [26]. However, these adverse findings were based on the use of multiple choice questions without feedback. In that context, it is not surprising that incorrect answers could be "learned".

Importantly, our results substantially extend the observations of Krasne et al [14], who found that untimed "open-book" formative assessments were good predictors of performance in a summative examination, possibly related to factors such as the reduced pressure compared to a conventional examination format and an emphasis on assessment of higher order learning (e.g. application, evaluation, self-direction). Although our formative assessments employed multiple choice questions (as well as short-answer questions), they were in many respects similar in character to the "open-book" assessments described by Krasne and colleagues [14]. Unlike those assessments, however, ours were integrated across disciplines, broader in their scope, available for a longer period of time, and embedded throughout a program of study. These factors are likely to have increased their efficacy. Furthermore, recent data on the application of test-enhanced learning in higher education [27] validates our use of a combination of short-answer questions and multiple choice questions with immediate feedback in promoting retention of knowledge.

Student perceptions of the formative assessments were uniformly favourable, with consistent and increasingly positive evaluations in online feedback surveys. Correspondingly, there were high participation and repetition rates in the online assessments for each course. The fact that students on average completed the formative assessment within each course in less than one hour might have contributed to the popularity of the assessments, because they were perceived as an efficient means of study for time-poor students.

Our systematic approach to the design, development, implementation and continual improvement of online formative assessments is likely to have played a role in students' perceptions of the assessments, as well as their positive effect on student learning. For example, as part of a continuous improvement cycle, all items used in formative assessments were analysed with regard to difficulty, discrimination co-efficient and correlation with overall assessment outcome. Those that correlated poorly were edited or eliminated from the next cycle, while concepts that proved difficult for students to comprehend were flagged for course conveners.

A limitation of our study is that it is not possible to conclude whether there is a causal relationship between participation and/or performance in online formative assessments and EOC examination marks. It might be that "better" students, who were more highly motivated, were more likely to undertake the formative assessments [12, 16]. In our study, multiple attempts at each formative assessment were not associated with higher EOC examination marks. This might suggest that EOC examination performance was primarily influenced by the inherent properties of the students, rather than the salient effects of formative assessment with feedback. Nevertheless, although one reported study has demonstrated no relationship between the effect of online formative assessments and overall student performance in a program as measured by grade point average [16], the design of our study cannot exclude such a relationship. Proving a causal relationship would require a design in which students were randomly assigned to a "control group" within a cohort. This would be inequitable, because a group of students would be deprived of the opportunity to undertake formative assessments during the trial period [12].

The results of this study support the contention that well designed formative assessments can have significant positive effects on learning. There is considerable untapped potential for use of formative assessments to assist learning by medical students and postgraduate medical trainees.