Teaching and assessing procedural skills: a qualitative study

Proficiency in procedural skills is an important training objective for Internal Medicine residents according to North American accrediting and certifying bodies [1–3]. In Canada these skills include central venous line and arterial line insertion, endotracheal intubation, thoracentesis, abdominal paracentesis, lumbar puncture and knee joint aspiration [3]. Acquiring these skills requires the development of several different abilities including psychomotor, clinical judgment, communication, decision making, and patient-focused interaction abilities [4]. Direct observation and expert feedback are crucial components to the development of these skills [5], yet appear to be challenging objectives for many programs to meet [6, 7].

In modern practice, internists are performing fewer procedures [8] and resident training time must be restricted to respect working hour rules [9]. Given such constraints, clinical exposure alone cannot be solely relied upon to offer procedural skill training that is scaffolded by direct observation and feedback [4].

Various approaches are available to ensure that residents have acquired these skills. For instance, logbooks have a long tradition as tools for assessment of procedural skills. Reports demonstrate that 52.3% of Canadian and 47.8% of American postgraduate programs use case or procedure logbooks as a form of assessment [10, 11]. At the time of those studies, logbooks were used at least two times more commonly than simulation. This popularity is likely due, in part, to the ease with which this form of assessment can be implemented. However, logbooks are a means of monitoring the breadth of experiences encountered and of documenting trainee progress. They do not reflect the performance, ensure direct observation or feedback nor ascertain the level of achievement of a trainee [12].

Other innovations aimed at preparing trainees to perform necessary procedural skills are based in the simulation lab. Studies using simulation devices to train physicians in bedside procedures such as central venous line insertion and thoracentesis, crisis resource management, as well as laparascopic and endoscopic skills are well established in the literature [13–20]. Assessment tools to measure proficiency in procedural skills have been created relying on methods such as the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS) and the Imperial College Surgical Assessment Device (ICSAD) [21, 22]. The MISTELS is designed to assess technical steps in laparoscopic surgery while the ICSAD has been specifically used for hand motion analysis for performance of technical skills. While useful for highly technical procedural skills, these tools address training and assessment of surgical skills, not of procedures typically done by internists.

Another potential tool for training and assessing residents in procedural skills is the objective structured clinical examination (OSCE). The OSCE has demonstrated validity and reliability in multiple settings [23]. Using multiple stations in an OSCE format was the framework used for the development of the Objective Structured Assessment of Procedural Skills (OSATS) [24, 25]. The OSATS is used to assess technical skills through both a procedure-specific checklist and a global rating scale of operative performance. Multiple studies have demonstrated high internal consistency and inter-rater reliability of the OSATS in laboratory multi-station settings and in the operating room [26]. A similar format was also shown to be useful for the assessment of minor surgical skills for clinical clerks and for family medicine residents [27, 28]. Structured clinical instructional modules (SCIM) were developed as teaching OSCEs to compensate for difficulty in accessing relevant clinical experience and to provide opportunities to learn about clinical situations that are infrequently encountered by trainees. [29, 30]. Additionally, the integrated procedural performance instrument (IPPI) was developed to assess a candidate’s ability to not only demonstrate the technical aspect of a procedural skill, but also the non-technical aspects such as communication, collaboration and professionalism [31]. These stations are logistically more complex and may require more time if included in a more traditional OSCE. Although OSCEs and IPPIs provide opportunities for training and assessment they are expensive and labour intensive.

As this brief review of the literature reveals, research has clearly established that laboratory-based simulation and OSCE frameworks hold great promise as settings for the teaching and assessment of procedural skills for internists. However, while considerable critical attention has focused on evaluating assessment tools, instructional modules, and different teaching and assessment settings, a similar depth of inquiry has yet to delve into the experiences of the trainees who actually face the challenge acquiring procedural skills. Indeed, when it comes to the teaching and assessment of procedural skills, research has carefully examined the delivery side of the question, but has paid less heed to the recipient side. Specifically, little is known about Internal Medicine residents’ individual experiences of 1) learning and acquiring procedural skills proficiency, 2) of practicing these techniques, or 3) of being assessed on their proficiency.

The study described here represents an effort to begin to address this gap. The purpose of this study was twofold. First, it sought to investigate residents’ experiences in the acquisition of procedural skills. Second, it explored resident perceptions of current assessment methods used to evaluate their ability to perform procedural skills. At our school, procedural skills stations were incorporated in a yearly formative Internal Medicine OSCE to evaluate the feasibility and some validity evidence of evaluating the residents using a modified OSATS format [32]. We saw the implementation of this OSCE format to assess procedural skills as an opportunity to ask residents to reflect on learning and assessing procedural skills in general. By using this OSCE as a stimulus for prompting reflection, we hoped to begin to describe some of the basic principles that are common across trainee experiences of acquiring procedural skill proficiency.

A fundamental qualitative descriptive approach [33] was the methodological orientation underlying this investigation. Given that the purposes of this study were descriptive in nature, this methodology was employed since it supports inductively identifying and clustering themes from participant statements in order to generate a description of phenomena. In acknowledgement that qualitative research is rarely produced via a single, “pure’ use of a methodology, it should be noted that this study has Grounded Theory overtones [33, 34]. This Grounded Theory overtone is realized in the use of constant comparison in this study’s data analysis work. Informed consent was obtained from all the study participants and approval for the study was obtained from The Ottawa Hospital Research Ethics Board.

Study participants represented all three years of post-graduate Internal Medicine training (8 resident year (R) 1s, 7 R2s, and 3 R3s). Analysis of the focus group data yielded 174 distinct comments from participants, which we grouped into five emergent themes: 1) learning procedural skills; 2) method of assessment; 3) realism of mannequins; 4) direct observation and feedback; and 5) importance of learning procedural skills. Each distinct comment was coded with its surrounding text (2 or 3 sentences) so that the analysis of each participant statement would include consideration of statement contexts. To be designated as a theme, each theme had to include comments from across the three focus groups, and across multiple participants within each focus group. The following reporting of results includes a description of each theme and sample comments from the participants.

This study provides insights into the perceptions and needs of Internal Medicine residents’ regarding the acquisition and assessment of their procedural skills. Study participants confirmed that it is important to acquire and be able to demonstrate proficiency in procedural skills. They described their present situation predominantly as a resident-directed learning and teaching structure in the clinical setting based on procedure opportunity rather than a structured curriculum. They also expressed a desire for more practice and feedback from direct observation by faculty supervisors. They perceived the OSCE to be an effective formative tool for testing procedural skills that provided a valuable learning opportunity with direct observation and immediate faculty expert feedback on their performance.

While this study provides interesting insights into resident perceptions of the OSCE as a mode of assessment for procedural skills, perhaps the most enlightening comments from participants were their reflections on the acquisition of procedural skills. Participants reported that, at the time of the study, they were responsible for directing the acquisition of and the attainment of proficiency in procedural skills. Participants clearly expressed a need for more directed learning opportunities, feedback, and structured assessments of their procedural skills. Current research confirms that trainees are increasingly participating in such independent learning contexts [45]. Studies are increasingly reporting that self-guided or self-regulated learning can provide effective learning environments for trainees [46–48]. However, as our findings confirm, researchers investigating self-guided or self-regulated learning repeatedly warn that some level of supervision should be maintained [45, 49–53], and that complete learner autonomy should not necessarily be the ultimate goal of medical education [54]. Thus, while the resident-directed learning described by our participants may appear to adhere to principles of adult-learning and self-directed learning [55, 56], the ad-hoc approach to the acquisition and assessment of procedural skills which is buttressed by limited feedback opportunities is problematic. As our participants report, input from skilled and qualified evaluators (be it in the form of direct observation, feedback or some other form) is an essential component to the success of their self-directed learning processes.

Residents reported that log books were insufficient to ensure proficiency of their procedural skills. The use of log books, however, can keep track of a resident’s procedure skills training, provide an overview of experiences to date, be valuable to monitor progress, and identify deficiencies in training opportunities. In short, log books should not be discounted altogether.

The proliferation of simulation centers in many medical schools provides a potentially safe environment for acquisition and practice of procedural skills. Despite the availability of simulation centers across Canada, a survey of Canadian Internal Medicine residents and program directors conducted in 2008 confirmed that less than 50% of programs were using simulators to teach procedural skills and that 90% of residents were still taught using the traditional “see one, do one, teach one” approach [57]. Two national surveys from the residency accreditation bodies in Canada and the USA, support this notion with reports of simulation use for assessment of less than 30% [10, 11].

We hypothesize that DSGL might be able to offer the direction our participants called for, while circumventing the significant challenge of limited supervisor availability. Our qualitative findings help to identify gaps that residents face in acquiring procedural skills and are starting points for addressing these deficiencies.

Based on the principles of transformational change and learning discussed by Kneebone [59] the acquisition of procedural skills should probably be learned in three distinct phases. Firstly, there should be learner-centered practice of the procedural aspect of the skill with a well-defined goal, opportunities for deliberate practice, direct observation and feedback to reach mastery [60]. Although the OSCE may satisfy the need for direct observation and feedback, deliberate and DSGL practice might best be completed in simulation laboratory settings on mannequins. Studies are showing that DSGL provides long-term benefits when compared to instructor-regulator learning in a simulated setting [61]. In addition, simulation-based education (SBE) of procedural skills is valued by residents, demonstrates high clinical competence and retention of skills after mastery training and is translated into better patient outcome [62, 63].

The second phase necessary to develop proficiency in procedural skills should again be learner-centered with a patient-focused simulation as described by Kneebone [59]. This setting allows contextual learning of the skill with a simulated patient, providing an opportunity to practice the whole scope of the skill (informed consent, collaboration with other health care professionals, and communication with the patient) in various, potentially challenging situations in a safe environment. This would give the resident a chance to integrate the other skills necessary to properly apply the procedural aspects described above. It would provide a safe environment to allow for potential mistakes which may be beneficial for deeper learning [64]. Assessment of this phase has been demonstrated using the integrated procedural performance instrument [65, 66]. The third phase is the integration of the procedure into direct patient care which will require supervision, feedback and assessment of proficiency as well. A tool such as the Direct Observation of Procedural Skills (DOPS) could be used to facilitate this final phase [67].

This study is not without limitations. One limitation is that this study was conducted at a single site and so the perceptions of the residents may be only applicable to one program. Based on the 2010 survey of Canadian Internal Medicine residency programs, our program was not unique since many other programs are not yet taking advantage of simulation and procedural skills curricula [57]. Although this has likely improved since the time of the survey, residents continue to be involved in procedural skills in the clinical setting and may have similar perceptions when faced with the translation of these skills on real patients.

Secondly, the timing of the focus group session eight weeks following the OSCE may have affected the residents’ recall of the use of the OSCE as a teaching and assessment tool. Conducting the sessions immediately after the debriefing session served to remind participants of their experience and provided an opportunity to recall those experiences without the emotional reaction that may have been present immediately after the OSCE itself.

Also, the focus group sample may have been biased by the self-selection of participants; however, more than one third of those who participated in the OSCE agreed to participate in the focus group. Only three senior residents participated in the study which may limit the opinions of this group but this number was proportional to the number of senior residents who participated in the OSCE. In addition, first year residents outperformed the third year residents in some of the procedural skills stations suggesting prior experience with acquiring the procedural skills by the first year residents [32]. Having more senior residents present for the focus groups may have led to further elaboration on the role of senior residents and faculty as supervisors and feedback providers.

Surveys do show that learning procedural skills is essential for the practice of medicine, but that there are few opportunities to practice in the clinical setting and that residents lack confidence in performing these skills [8, 20, 68]. These findings are confirmed by our residents, and the focus group comments elaborate the reasons and experiences behind these findings. In addition, many themes identified were general and not specific to any particular form of assessment. Thus, we propose that our findings may be transferable to other learning and assessment settings and to other programs.

As procedural skills move beyond the SBE environment and are integrated back into direct patient care, future research documenting the perceptions of residents will be necessary to ensure the transformation change is complete and to assess if similar themes arise in various settings.

Procedural skills will remain within the scope of practice of internists for the foreseeable future [69]. A serendipitous self-guided approach to acquisition and assessment of procedural skills may not be meeting the needs of postgraduate trainees. Residents participating in this study expressed the importance of direct expert observation with feedback for the assessment of procedural skills. As training programs move towards competency-based training, steps should be taken to provide opportunities for directed self-guided learning for step-wise acquisition of procedural skills including direct observation and feedback. Greater expansion and availability of simulation centers may address some of the gaps identified.