Arthroscopic proficiency: methods in evaluating competency

Cadaveric training models and surgical simulation

Advancing technical complexity taught within reduced work-hour training programshas driven the need for alternative strategies in arthroscopic skilldevelopment. Traditionally, the cadaver specimen in the bioskills laboratory hasremained the highest fidelity model [23]. Few would contest the likeness of the human cadaveric specimen toreproduce arthroscopy conditions or the value of positional and tactile feedbackusing instrumentation in this environment. The use of fresh cadaveric specimensas the primary teaching platform in instructional courses for board certifiedsurgeons supports this claim. The educational benefits of managing the nuancesof arthroscopic equipment and troubleshooting problems with fluid management,the light source, and shavers should also not be underestimated [17]. In addition, when arthroscopic trained surgeons were polled ontraining methods contributing to self-perceived proficiency in all-arthroscopicrotator cuff repair, practice on cadaveric models was third, second only tofellowship training and hands-on courses [24].

Financial considerations as well as specimen availability limit formalarthroscopic training on cadavers within most orthopaedic program curricula [4]. The cost of acquiring specimens combined with the inherent costs ofmaintaining the equipment and personnel of a bioskills laboratory are difficultto quantify and usually depend on industry support and/or sponsorship. There arealso concerns regarding uniformity between specimens with variability in bothanatomy and internal pathology [4].

To avoid these obstacles, the concept of computer-based simulation forarthroscopic training and skill acquisition has emerged. Embraced by theaviation industry, simulators remain a core competency in pilot training andcredentialing [11]. The development of less-expensive high performance computerscombined with advances in graphical and force-feedback technology (haptics) hasaccelerated this movement. Proposed simulators would allow for the quantitativeassessment of technical ability performed within the confines of a safe andcontrolled environment. Advantages include the absence of time constraints orsupervising faculty, uniform training scenarios of adjustable complexity andpathology as well as substantial saving from costly disposable equipment andtraining time within the operating room [15, 23].

The use of laparoscopic and endoscopic simulators has been incorporated into manytraining programs, as the validity of such models has been previouslyestablished [25–30]. A systematic review of randomized controlled trials of laparoscopicsimulators reported improved task performance by trainees and a greaterreduction in operating time, error and unnecessary movements as compared tostandard laparoscopic training [31].

Knee simulators have been shown to reliably distinguish between novice and expertarthroscopists [12, 32] and demonstrate the learning potential of identifying anatomicallandmarks and triangulation skills [35]. There is only a single study demonstrating the transfer validity ofarthroscopic skills to the operating room for diagnostic knee arthroscopy [13]. However, there was no true control group having only comparedsimulator training versus no training.

Outcome measures that were able to discriminate skill level and expertise inshoulder simulators include; time to completion of tasks, distance and pathtraveled by probe and the number of probe collisions [5, 11, 15, 34]. A follow-up study conducted at the 3-year period showedsignificantly improved simulator performance after an increase in arthroscopicexperience [10]. A positive correlation of arthroscopic task performance betweensimulator and cadaveric models has also been observed in shoulder arthroscopy [15]. A subsequent investigation demonstrated a significant relationshipbetween the performance of basic arthroscopic tasks in a simulator model andresident arthroscopic experience, supporting the use of simulators as beneficialeducational tools to improve arthroscopic skills [36].

Technological advances have made the potential widespread use of simulators moreaffordable, but additional hurdles exist. The availability of content experts,mainly surgeons that can provide domain-specific surgical knowledge to allowdevelopers to generate realistic simulations is a limiting factor [9]. Further understanding of the psychomotor and cognitive components ofthe surgical process is still necessary for its translation into the virtualworld.

Psychomotor analysis and arthroscopic technical ability

Analysis of force sensors has been reported as a valuable method to assessinterference between surgical tools and tissue in endoscopic sinus surgery andlaparoscopic surgical training [41, 42]. In arthroscopy, excessive force applied through instruments mayresult in iatrogenic damage to the patient, often as damage to articularcartilage [11, 23, 43]. Therefore, measurements of force may provide an objective means ofevaluating tactile surgical performance. Assessment of force torque signatureshave been shown to correlate with level of arthroscopic experience in the knee,where expert surgeons had fewer collision incidences, greater variety of forcemagnitudes and superior efficiency [38]. The use of excessive and unnecessary force patterns by trainees wasalso demonstrated in a knee simulator when compared to that of experiencedsurgeons [32]. However, distinguishing harmful from harmless contact in tissuemanipulation and dissection can be challenging and these studies were small andoften lacking a complete assessment of each area of the knee. The concept ofabsolute maximum probing force (AMPF) during menisci manipulation has beenintroduced and significant differences between the expert and novicearthroscopists have been identified [39].

Electromagnetic motion tracking systems have been employed to plot instrument tiptrajectory as an objective evaluation tool. The validity of motion analysis toassess surgical skills in terms of precision and economy of movement has beenshown within the laparoscopy literature [44, 45]. In knee arthroscopic simulators, level of expertise has beenassociated with reduced probe path traveled and number of movements and improvedeconomy of movements [37]. Similarly, the path length of the probe and scissors wassubstantially shorter and probe velocity faster in more experienced surgeonswhen performing partial meniscectomy in knee models [32]. These finding have also been demonstrated in virtual realitysimulators of the shoulder where probe path length was shorter for specialistsand probe velocity was nearly double that of novices [5]. Reduced traveled probe distance has been suggested to correlate withsmoothness of scope manipulation during shoulder joint inspection and probingtasks. Yet, motion analysis investigations have only been performed within asimulated environment and only involving basic arthroscopic tasks. It is unclearif improved efficiency of movements in these models translates into improvedperformance within the operating theatre.

In addition to force and motion analysis, simple visual parameters have beendescribed as an objective method for evaluating technical skill [40]. The prevalence of instrument loss, lookdowns and triangulation timeis able to discriminate novice, resident and expert skill levels in a kneesimulator.

Validated assessment tools

The current paradigm of arthroscopic training relies on the apprenticeship modelwhere residents are evaluated by a precepting surgeon as their level ofinvolvement is subsequently increased. The subjectivity of this method has beencriticized and shown to not necessarily reflect the actual level of skill [23, 43]. This assessment is not based on a pre-determined level ofperformance, but rather on global assessment by the precepting surgeon partlydetermined by that surgeon’s experience and spectrum of patients withintheir practice [9]. Ideally, an assessment tool should be feasible and practical whileremaining as objective as possible [37, 46].

The implementation of various procedure-specific checklists [47–49] and global rating scales [50–52] has been well described in other surgical disciplines and theObjective Structured Assessment of Technical Skill (OSATS) is the most widelyaccepted “gold standard” for objective skills assessment [53]. Yet, evidence suggests that these methods are valid for feedback andmeasuring progress of training rather than examination or credentialing [54].

The Orthopaedic Competence Assessment Project, developed by the BritishOrthopaedic Specialist Advisory Committee, is part of the competency-basedtraining structure implemented by the surgical royal colleges in the UnitedKingdom [55, 58]. It consists of an intra-operative technique section comprised of 14criteria, but has not been subjected to independent testing. However, amodification of this procedural-based assessment (Additional file 2: Appendix 2-A) combined with an OSATS global ratingscale (Additional file 2: Appendix 2-B) was developedto evaluate the transfer validity of a simulator in diagnostic knee arthroscopy [13]. Although improved performance in the simulator-trained group wasdemonstrated, the only comparison was an untrained group of individuals.

Recently, more comprehensive knee scoring systems have been introduced. TheArthroscopic Skills Assessment Form is a 100-point tool used to objectivelyevaluate diagnostic knee arthroscopy assigning points for correctly identifyingstructures and time to completion as well as point deductions for iatrogeniccartilage injury [56]. It was able to distinguish between the novice, experienced andexpert arthroscopists in the cadaver knee model. The Objective Assessment ofArthroscopic Skills (OAAS) instrument consists of multiple skill domains eachrated on an expertise-based scale with 5 skill-level options [57]. When combined with an anatomical checklist, the OAAS discriminatedbetween skills levels of various levels of training with excellent internalconsistency and test-retest reliability.

Quantifying arthroscopic proficiency

Despite being amongst the most commonly performed procedures by orthopaedicsurgeons, consensus on what constitutes arthroscopic competence and the numberof procedures to attain it remains uncertain [16, 18]. This is compounded by increasing technical sophistication ofprocedures and the demand for accountability and satisfactory outcomes bypatients [59]. Competency in arthroscopy typically develops during completion of aresidency curriculum as defined by the Residency Review Committee for the ACGME,but there is no suggestion for a recommended case volume of procedures [19]. Certification examinations test for proficiency in contentcomprehension and decision-making capabilities, yet there is no objectivetesting to evaluate arthroscopic technical competencies at the end of residency [20–22].

Objective data regarding competence in arthroscopy is sparse and guidelinesspecifying achievement and maintenance of competence are vague. The ArthroscopyAssociation of North America (AANA) does not quantify competence, but onlyrequires that 50 arthroscopic cases be performed annually to maintain activemembership [60]. However, the AANA does acknowledge that completion of an orthopaedicresidency does not guarantee competence in arthroscopy and that privilegesshould be granted by the regulating bodies of individual hospitals and shouldconsist of an observational period for direct skill assessment [61]. The American Board of Orthopedic Surgery (ABOS) requires a one-yearaccredited ACGME sports medicine fellowship and at least 75 arthroscopy cases tobe eligible for subspecialty certification in sports medicine [62].

Considerable variation exists when attempting to assign a numerical value forarthroscopic competency in the literature. A survey of U.S. orthopaedicdepartment chairs and sports medicine fellowship directors identifiedsubstantial variability in the number of repetitions to become proficient inarthroscopy [18]. For instance, the average number for diagnostic arthroscopy of theknee was 45 with suggested repetitions ranging from 8 to 250. There was also atendency for physicians who perform little or no arthroscopy to underestimatethe experience needed for proficiency. This finding was confirmed in a similarsurvey performed in Europe amongst orthopaedic residents and attending staff aswell as the trend for residents to overestimate the average number of casesrequired for competency [63]. Here, a mean of 120 procedures was estimated by residents forarthroscopic ACL reconstruction compared to 90 by staff physicians.