The Collaborative Learning Development Exercise (CLeD-EX): an educational instrument promoting key collaborative learning behaviours in medical students

Background: Modern clinical practice increasingly relies on collaborative and team-based approaches to care. Regulatory bodies in medical education emphasise the need to develop collaborative and teamwork competencies and highlight the need to do so from an early stage of medical training. During undergraduate medical education the focus is usually to facilitate collaborative learning and increase feedback and reflection on this learning. This article describes a novel educational instrument, the Collaborative Learning Development Exercise (CLeD-EX), which aims to foster the development of key collaborative learning competencies in medical students. Methods: In this study, the “educational design research” framework was used to develop, implement and evaluate the CLeD-EX. This involved adopting a systematic approach towards designing a creative and innovative instrument which would help solve a real-world challenge in developing collaborative learning skills. The systematic approach involved a qualitative exploration of key collaborative learning behaviours which are influential in effective collaborative learning contexts. The identified competencies were employed in the design of the CLeD-EX. The design of the CLeD-EX included features to facilitate structured feedback by tutors to students, complemented by self-evaluation and reflection. The CLeD-EX was field-tested with volunteer junior medical students, using a controlled pre-test post-test design. Analysis of the completed CLeD-EX forms, self-perception surveys (i.e. pre-test and post-test surveys) and analyses of reflective reports were used to explore the educational impact of CLeD-EX, as well as its utility and practicality. Results: After using the CLeD-EX, students showed a significant improvement in

self-directed learning, critical thinking and group process as measured by a previously validated instrument. Both students and tutors recognised CLeD-EX as an effective instrument, especially as a structured basis for giving and receiving feedback and for completing the feedback loop. CLeD-EX was also found to be feasible, practical and focused, while promoting learning and effective interactions in small group learning.
Conclusion: The findings of this study support the introduction of an educational instrument such as the CLeD-EX to facilitate the development of students' skills in collaborative learning.

Background
Collaborative learning and teamwork are core competencies that impact on the quality of health care (1). Accreditation bodies for Medicine programs therefore, have developed frameworks to ensure that collaborative and teamwork skills are developed to standards required for effective clinical practice and patient care (2)(3)(4). Globally, medical schools are urged to place an emphasis on developing these skills from an early stage (5). At the start of a medicine program, medical students often work and learn together in small group learning contexts, therefore students require the capability to collaboratively learn together. Collaborative learning is broadly defined as "students working in groups of two or more, mutually searching for (knowledge) understanding, solutions, or meanings, or completing a task or creating a product. There is wide variability in collaborative learning activities, but most centre on the students' exploration or application of the course material, not simply the teacher's presentation or explication of it (content)" (6).
Despite the use of a variety of approaches such as small group learning activities (e.g. Problem based learning, team-based learning, etc) or group assessments (e.g. group projects), promoting effective collaborative learning in medical students continues to be a challenge. Simply putting students in small groups and expecting them to collaborate and learn together does not automatically lead to effective collaborative learning. Research has highlighted several challenges that students experience in collaborative learning such as unequal individual participation in group work (7), lack of effective communication (8) free-riding and social loafing on collaborative tasks (9) and dealing with difficult and dominant group members (10).
These challenges can be due to inexperience in working in groups, the lack of clarity around the purpose of the group work or due to the lack of collaborative learning skills among medical students.
It has been proposed that medical students would benefit from receiving structured guidance for learning and interacting within small group, collaborative contexts.
Importantly, medical students also identify the need for learning interactional skills relevant for collaboration and teamwork (11). Moreover, key recommendations from the literature include regular evaluation and the use of formative rather than summative assessments of collaborative group work, or using self-and peer evaluations to promote the students' skills for effective collaborative learning (12).
Previous research of using peer evaluation in formative assessment of student behaviours in small group learning activities has reported it as a useful process for providing feedback to students (13). A recent systematic review has shown that peer feedback in collaborative learning environments can aid in the development of students' professional behaviours (14). Moreover, self-evaluation can be valuable in group work assessment especially when used for the purpose of formative assessment (15). Although tutor evaluation is considered superior to self and peer evaluation in small group work (16); there are concerns related to feasibility when tutors are expected to provide individual students with meaningful and timely feedback on their performance (17). Therefore, providing a structured framework for evaluation and feedback by tutors could help overcome the challenges tutors face in providing useful and timely evaluation and feedback to students during collaborative learning activities.
Research has identified several instruments which have been used to assess and promote collaborative learning in small group environment (18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29)(30). There are several limitations identified in these instruments. Firstly, the content focus of these instruments was variable: some instruments focused on interactional skills, while others focused more on learning skills. Moreover, these instruments were limited by the absence of a clear format for tutors to provide feedback and the absence of a structure to guide student reflection on the feedback received. Few of these identified instruments (19,20,25,27) reported on the practicality and feasibility of use in busy educational contexts.
The literature reviewed above highlighted the need to develop an instrument that would be feasible and practical in busy educational contexts. The CLeD-EX was developed to address this need. The design of the CLeD-EX included a structured format for evaluation, feedback and reflection to promote key behaviours for effective collaborative learning in medical students. In this manuscript we report on the effectiveness, feasibility and impact of the CLeD-EX.

Research which led to the development of CLeD-EX
The "educational design research" framework (31,32) was used to develop, implement and evaluate the CLeD-EX. This involved adopting a systematic approach towards designing a creative and innovative instrument which would help solve realworld challenges in developing collaborative learning skills in junior medical students. The details of this work have been reported by the authors in prior publications (33,34). The systematic approach involved a qualitative exploration of key collaborative learning behaviours which are influential in promoting effective collaborative learning (33). In the next step, a modified Delphi study was conducted and the top six, key student behaviour which were influential in promoting effective collaborative learning were identified (34). These six behaviours were included in the CLeD-EX Instrument (Table: 1) Table 1: Key collaborative learning behaviours for inclusion in the design of the CLeD-EX instrument (34) Key collaborative learning behaviours for inclusion in the design of the CLeD-EX instrument (The student) Is well prepared for the learning session; (The student) Is involved in discussion and debate on different ideas; (The student) Appears willing to work and is engaged in the learning; (The student) Listens to others' points of view; (The student) Shares information with group members and voices own opinions; (The student) Reflects on feedback and responds appropriately to it

Design of the CLeD-EX Instrument
The CLeD-EX consisted of an evaluation and feedback component. A four-point frequency scale was selected for the evaluations, with response options: never; rarely; often; and always. A fifth option of unable to assess was also included along with a space to provide a comment. The CLeD-EX process was structured as follows: Stage: 1 Student self-evaluation: Each student rates themselves in relation to the behaviours on the CLeD-EX in the context of small group learning.
Stage: 2 Tutor evaluation and feedback: In this part, tutors evaluate the behaviours of each student during small group sessions. Space is allocated for tutors to provide feedback on two aspects: "Positive aspects of collaboration"; and "Areas that can be improved." The written feedback is further supplemented by a dialogue between the tutor and the student. The purpose of this oral feedback is to elaborate on the written feedback and discuss an action plan for improvement.
Stage: 3 Student reflections: In this section of the form, each student reflects on the entire activity and writes a reflective account on the following two aspects: "Issues which were raised in this exercise that need to be addressed (to improve my collaborative learning behaviours);" and "Action plan for improvement." The above design illustrates how the key tenets for effective feedback which includes self-evaluation; feedback; and feedforward (35) were incorporated in the CLeD-EX.
Two global items were included at the end of the self and tutor evaluations. The first global item focused on key collaborative behaviours and their impact on the quality of learning. The second item obtained the student's and tutor's perceptions of the overall collaborative skill of the learner.
Information was provided to both students and tutors regarding the purpose of the CLeD-EX, including provision of feedback to assist development and encouragement of reflection. Tutors were asked to indicate the amount of time it took to provide feedback (this was an important consideration related to the feasibility of the CLeD-EX). Following completion of the student reflection section of the form, students were asked to respond to two items relating to the usefulness of the CLeD-EX and the feedback received through the instrument.
The final design of CLeD-EX was reviewed for clarity, by the members of the project team and three tutors of small group learning at UNSW Medicine. The CLeD-EX was implemented via a fillable (smart) PDF form, which was emailed to students, completed online and saved in electronic form, then returned by email. In addition, students could print the form and complete it in pen and paper format if they preferred.

Field-test of CLeD-EX
The context for the CLeD-EX field-test was the scenario group learning sessions which are key components of Phase-1 (years 1 and 2) of the UNSW Medicine program. Scenario group learning is a variant of problem-based learning, designed and utilised at UNSW Medicine (36). Students, in groups of 14-15 facilitated by a tutor, spend two hours twice-weekly working on structured collaborative learning activities, presentations and tasks around relevant clinical scenarios.
The field test of the CLeD-EX involved an experimental format with a pre and posttest ( Figure 1). All students in two different courses in Phase 1 were invited to voluntarily participate in the CLeD-EX field-test. Students who agreed to participate were randomly allocated into control and experimental groups. At this stage, cluster randomisation was performed using existing structures of the Medicine program (whereby students are randomly assigned into four colleges, i.e. A, B, C and D, for administrative purposes). Small group, collaborative learning activities are scheduled at separate times for each college; the colleges were considered as separate clusters for this study. Volunteer students were randomised into control and intervention groups based on their colleges (i.e. clusters). The rationale for this cluster randomisation trial design was to avoid cross-contamination between the control and intervention groups within small group learning activities (37). However, for equity, the control group also received a copy of the CLeD-EX after completion of the study.

Evaluation of CLeD-EX Forms
All the CLeD-EX forms received after completion of the study were analysed. The rating on the forms were scored from 1 to 4 (1=Never, 2=Rarely, 3=Often, 4=Always). The response 'unable to assess' was handled as missing data. The analysis involved the following: Internal consistency reliability of CLeD-EX: analysed using Cronbach's Alpha for the self and tutor evaluation scales; Comparison of self-evaluation with tutor evaluation: analysed through comparison of means and relative effect sizes. The differences between statistical significance and practical significance were verified using the effect size, calculated via Cohen's D. Inter-rater agreement was calculated using Pearson correlation coefficients; CLeD-EX acceptability and feasibility: measured based on the time taken to complete the tutor evaluation and feedback. This was also analysed on the students' response to two items at the end of the form; and Qualitative analysis of open-ended feedback and student reflections. The written feedback was analysed using a modified version of a framework originally developed to evaluate written feedback on a workplace-based rating form (38). The taxonomy applied in the analysis of the written comments on the CLeD-EX involved the following categories: identifying the behaviour, identifying the performance gap and providing advice for developing a learning plan. Each component in the CLeD-EX form was scored dichotomously, absent (score=0) or present (score=1). The first author (MPI) analysed all the written comments based on this algorithm, while the research team (GV, AOS and CB) analysed a random sample of the written comments.
Any discrepancies were discussed and resolved. The depth and breadth of student discussion in the student reflections was analysed to see if the CLeD-EX triggered a change in their thinking and/or understanding and encouraged them to plan an action to improve or to act differently.

Pre-test and post-test evaluation
The pre and post-test was conducted using a previously validated questionnaire designed by researchers at McMaster University (39) to measure improvement in self-directed learning, critical thinking and group process. The 29 items of the instrument are classified into three subscales: self-directed learning (SDL); critical thinking (CT); and group processing (GP). The internal consistency reliability of these subscales was confirmed in this study with Cronbach alphas for each subscale as follows: SDL= 0.853; CT=0.7995; and GP=0.855. The response option for the items was on a six-point, ordinal Likert-type scale 0=never; 1=almost never; 2=sometimes; 3=often; 4=almost always; and 5=always. The score for each subscale was the total score of all items in that scale (a higher score represented a better outcome). Students completed the pre and post-tests online via the Key Surveys (UNSW: WorldAPP-Key Surveys -Online survey tool).
An independent sample t-test was conducted to explore the differences between the intervention and control groups in the pre-test and post-test. A paired samples ttest was conducted to evaluate change in the intervention group following the use of the CLeD-EX. A Preliminary analysis was conducted to ensure that there was no violation of the assumptions of normality. All quantitative data analysis was conducted using the statistical software package, IBM SPSS version 24.

Ethics
Ethics approval for this study was obtained from Human Research Ethics Advisory Panel (HREA) UNSW Sydney (Reference number: 2014-7-39).

Results
All Phase 1 students, across two courses (n= 840) were invited to participate in the CLeD-EX field-test. A total of 107 students volunteered to participate and completed the online pre and post-test (response rate: 12.7%). Within that cohort, 52 students were in the intervention group (Colleges A and D) while 55 were in the control group (Colleges B and C). All students were encouraged to submit the completed CLeD-EX forms; 93 completed CLeD-EX forms were received and analysed.

Internal consistency reliability of the CLeD-EX
Cronbach's alpha for the tutor-evaluation scale was 0.766 and for the selfevaluation scale the Cronbach alpha was 0.451.

Comparison of self-evaluation with tutor evaluation
Item by item paired t-tests (Table 2) were undertaken to compare students' selfevaluation with tutor evaluation. For all items, except the behaviour "…listens to other's point of view", students scored themselves significantly lower than their tutor. Tutors' evaluations of students' collaborative behaviours were significantly more positive than students' self-evaluations for five out of six key behaviours (  Inter-rater agreement: Pearson's correlation coefficient was calculated to analyse the correlations between the mean scores of student evaluation and mean scores of tutor evaluation. There was a significant correlation between the mean scores of individual items on self-evaluation and mean scores of individual items on tutor evaluation (r=0.887; p=0.019).
In the global, dichotomous item on the self-evaluation scale "overall my skills in collaborative learning are well-developed", 87 students answered "yes" while six students answered "no." On the tutor evaluation scale, the global item "Overall this student demonstrates skills which enhance the quality of collaborative learning" was answered positively by 88 out of 89 respondents.

Analysis of utility and feasibility
Two items focussed on utility and feasibility of CLeD-EX were completed by each student after completing the last section (Part 3 Student Reflection) in the CLeD-EX form. For the first item, "I found the CLeD-EX helped improve my skills in collaborative learning", 81.5% of students agreed. For the second item, "I was provided with useful feedback through the CLeD-EX process", 92.3% of students responded positively.
The feasibility of the CLeD-EX was ascertained based on the time taken for tutors to give feedback to students. The mean time recorded for completing the evaluation and providing feedback was 6.25 minutes. The minimum time recorded by the tutor on the CLeD-EX to provide feedback was two minutes and the maximum time was 15 minutes. The majority of tutors recorded that the time taken to give feedback to student was five minutes.

Analysis of written feedback: Tutors included written feedback in the majority of
the CLeD-EX forms (94.6%). Some forms included only comments about positive aspects related to collaboration (12.9%). When analysing the quality of feedback based on the modified framework (38) in the section on the form: 'areas that can be improved' the behaviour which needed to be improved was mentioned in 73 forms, while the gap was mentioned in 17 forms and advice on how to develop was included in 32 forms. This finding suggests that the CLeD-EX was successful in facilitating formative feedback by highlighting behaviours that need further development.
A brief review of the content of the written feedback suggested that the most common focus was on the students participating in discussions, for example, "Try to engage more with your peers during small group activities by exploring and discussing issues with them," and "I suggest you set yourself a goal of contributing one significant idea to the discussion per class." This was followed by the feedback on student preparation for the small group learning, for example: "Improving preparation can be one approach you could take. This involves reading ahead, which can be useful in providing a sound basis for learning in each SG session." These findings are consistent with the students' self-evaluation in which similar behaviours were scored the lowest, also on the tutor-evaluation scale. These findings suggest an important link between the students' self-evaluation and tutor evaluation. The tutors' written feedback also emphasises development of these behaviours in the student. Meanwhile, comments for improvement were mainly on aspects related to developing skills to learn more effectively in small group context.

Analysis of Student reflection:
In the 93 CLeD-EX forms received, reflections were recorded in 86 forms (92%). This indicates that the structure of the CLeD-EX appears to facilitate students to reflect on the collaborative learning experience.
Students in their reflective accounts discussed issues which were being raised through this exercise and their action plans for improvement. In the written reflections relating to the latter, students clearly expressed their desire to improve their behaviour for collaborative learning. This change was observed through comments which indicated an intention to change by developing a new understanding which was gained through this activity. In their reflective comments some students acknowledged the importance of collaborative learning and the usefulness of developing their collaborative learning behaviours. Through this process students have recognised its value for building their skills in collaboration and learning.
"In terms of other aspects of collaborative learning, I will continue to be engaged in all teamwork activities and listen to the opinions of others. In particular, I think our collaborative learning in SGs (scenario groups) will be enhanced if quieter members are drawn out to voice their opinions, so I'll keep this is mind. In short, teamwork is a great way of enhancing the quality of our learning, and not only does it deepen our knowledge, but it develops our broader social interpersonal skills." (C-255)

Students' Self-Directed Learning (SDL), Critical thinking (CT), and Group Processing (GP)
Comparison of pre-test self-evaluations by students (Table 3) revealed no significant difference between the intervention and control groups. .
In the post-test comparison (Table 4), the intervention group scores were significantly higher than the control group for Self-Directed Learning, Critical Thinking and Group Process.

Discussion
The findings of this study indicate that the CLeD-EX is an effective and feasible instrument to promote key collaborative learning behaviours in medical students.
Students indicated that the CLeD-EX was beneficial in terms of improving their collaborative learning and that the feedback provided through this process was useful. The qualitative findings illustrate how the CLeD-EX has a positive impact on learning.
Learners often report receiving insufficient feedback from tutors and whatever feedback is received is often not helpful, timely or constructive (40). Important reasons for these findings could be time constraints faced by teachers or lack of experience or structure in providing feedback to their students. The CLeD-EX aimed to address these issues by developing a process that would minimise the extra time load imposed on tutors, while providing a framework to give students feedback on these aspects. The feedback facilitated by the CLeD-EX included both written comments and oral dialogue with the tutor. In this study, only the written comments were analysed. Students may have received constructive comments on other aspects of their collaborative behaviours during the feedback dialogue.
There was a statistically significant difference between the mean summed score of students' self-evaluation and mean summed score of tutors' evaluations of collaborative behaviours, with a large effect size. Direct comparison of mean selfevaluation and mean tutor-evaluation revealed a consistent under-evaluation by students. Also, there was a significant correlation between the mean scores of individual items on self-ratings and mean scores of individual items on tutor evaluation. This indicates that although there is a difference between student selfevaluation (more stringent) and tutor evaluation (more lenient), the items may be ranked similarly. These findings are consistent with the literature which reports that student have a tendency to underestimate their performance during self-evaluation (41).
Comparisons between students' pre-test and post-test indicated a significant improvement in students' self-directed learning, critical thinking and group process interactions in the intervention group. In contrast, only the self-direction subscale improved in the control group, with a small effect size. This finding provides evidence for the efficacy of CLeD-EX from the students' perspective.
Self-directed learning is defined as "a form of education that involves the individual learner's initiative to identify and act on his or her learning needs (with or without assistance), taking increased responsibility for his or her own learning" (42). The metacognitive skills for self-directed learning includes planning, monitoring progress and evaluating whether goals are achieved (43). Enhancements in selfdirected learning skills are known to positively influence deeper approaches to learning (44). Furthermore, the focused design of the CLeD-EX with feedback and reflection, promoted students' self-awareness of the key aspects which need to be considered during collaborative learning.
Another important finding in the evaluation of the CLeD-EX was the significant improvement in students' self-reported critical thinking. The ability to think critically is vital to medical practice and its active training has to be part of the medical curriculum (45). The CLeD-EX components of self-evaluation, tutor evaluation and feedback, followed by reflection might have triggered a cycle of critical thinking in students to develop their collaborative learning skills. Those processes are also likely to have led to the significant improvement in group process reported in the intervention group.
Lastly, the importance of reflection in medical education and building this reflective capacity in learners is well-established. Focused and guided approaches to develop reflective capability is encouraged at all stages of medical education (46). Also, the qualitative findings illustrate how the CLeD-EX supported students to develop their skills in critical reflection and develop plans for further improvement.

Limitations
The voluntary nature of the field-test of the CLeD-EX, with no protected time allocated to engage in the CLeD-EX process, limited the number of student participants. The challenges of recruiting participants for research (47), especially educational research (48) are well-recognised. In addition, administering the CLeD-EX required tutor evaluation and feedback, requiring time and effort by tutors. All these aspects need to be considered when designing future strategies to implement CLeD-EX.

Conclusion and future directions
In conclusion, CLeD-EX is a feasible and effective educational instrument which focuses on developing key behaviours required for collaborative learning in medical students. Preliminary evaluation of the CLeD-EX confirms that it is feasible to use in busy educational settings and has a positive impact on learners.
Further research is needed into the acceptability of CLeD-EX, and students and tutors' experience of the CLeD-EX process. In addition, it will be worthwhile to explore its potential application to other healthcare disciplines such as dentistry, nursing, and allied health. The CLeD-EX could also be extended to postgraduate healthcare education, in which small group, collaborative learning is employed.

Abbreviations
CLeD-EX-Collaborative Learning Development Exercise

Consent for publication
Not applicable

Availability of data and materials
The data that support the findings of this study are available from University of New South wales but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of University of New South Wales.

Figure 1
Format of Field-test for CLeD-EX. E-Experimental group, C-Control group, T-Self-perception

Supplementary Files
This is a list of supplementary files associated with the primary manuscript. Click to download.