This was a single institution observational cohort study conducted at an Australian university in August 2021. It was designed to use a previously validated tool, the Script Concordance Test (SCT), to measure clinical reasoning in third year undergraduate paramedic students. The study was approved by the Monash University Human Research Ethics Committee (MUHREC - 2021-29,344).
Development of the script concordance test
Test construction occurred in accordance to SCT developmental guidelines published in an AMEE Guide by Lubarsky et al. [11]. The structure of the SCT aims to reflect the varying shades of clinical ambiguity inherit in real-world practice by presenting clinicians with a series of clinical vignettes with incomplete or ill-defined medical data. Development of the SCT was reliant upon three fundamental steps: creating clinical vignettes and test items formulated from actual cases, selection of the reference panel, and construction of the scoring matrix [11].
Test drafting required authors to create quality test questions representative of the paramedic discipline ensuring both content validity and clinical ambiguity. Clinical vignettes were constructed in alignment with the medical curriculum utilised by clinical faculty throughout the three-year undergraduate degree at Monash University. General medical domains including cardiac, respiratory, trauma, obstetric, and neurological pathologies were utilised, in addition to areas surrounding pharmacology and clinical practice guidelines. These domains are representative of the universal knowledge base students should be well versed in upon completion of the undergraduate degree. Clinical vignettes and questions were created based upon attracting both a wide range of responses from the available options and limiting the necessity for students to rely upon factual recall.
Three of the project team with vast clinical experience (BF, ES & BS) were utilised to verify the validity of the test aiming to ensure that scenarios were relevant to the paramedic discipline, required a degree of decision-making, and were correctly formatted. A fourth team member (LR) was responsible for proofreading test items prior to finalisation and dissemination of the test. Draft amendments occurred utilising collaborative software allowing for easy adjustments to occur amongst authors. A total of 11 clinical vignettes comprising of 28 questions were utilised, with the most frequent clinical questions exploring management and treatment modalities (n = 23), with the remaining questions related to patient assessment or disease pathology (n = 5). See Fig. 1 for example question and format.
Reference panel selection
Developmental guidelines outline that 15 panel members are required for high stakes examinations, in addition to optimal panel composition requiring experts to be representative of the paramedic profession with sound clinical experience. Fifteen panel members were initially recruited, with one participant failing to complete the test. The final reference panel comprised of 14 experienced clinicians who are employed as operational paramedics with current registration and a minimum of 5 years’ experience. The reference panel consisted of Advanced Life Support (ALS) paramedics (n = 4), Intensive Care (IC) paramedics (n = 5) and Intensive Care Helicopter Emergency Medical Services (HEMS) paramedics (n = 5).
Identical tests were provided to the reference panel clinicians with the same test conditions and parameters. As the tailored SCT is characteristic of commonly encounter medical presentations, the reference panel clinicians completed the test without prior preparation.
Scoring matrix
Test responses were graded in accordance with the literature by utilising the aggregate scoring method [11, 17, 19]. Contrary to traditional examinations which require test-takers to select the allocated single best answer, the SCT awards both partial and full credits to participant responses.
Scoring matrices are based upon the distribution amongst the reference panel for responses for each clinical question. Full credits are awarded to the modal answer (commonly chosen answer amongst the panel members); partial credits awarded to alternative answers dependent on the fraction of the reference panel selecting that response; remaining responses not chosen by the reference panel are awarded 0. Results are matched to explore the level of concordance between student and reference panel responses.
Data collection
The SCT was introduced to students as part of a face-to-face tutorial sessions during week three of their final academic semester. They were provided with an introductory session aiming to familiarise participants with the structure and components of the SCT. Due to COVID restrictions at the time, a pre-recorded introduction by the chief researcher was provided to participants. Students were introduced to the objectives of the research, the importance of clinical reasoning within paramedic practice, and provided with an exemplar clinical vignette and questions.
Data analysis
Data were reported using descriptive statistics including, means (standard deviations), medians (inter quartile ranges), ranges or frequencies where appropriate. Comparisons between the reference panel and student participants was evaluated using t-tests between mean scores. Standardisation of participant scores was completed using a standardisation method previously described by Piovezan et al. [17] and Charlin et al. [20]. One student response was excluded from analysis as a result of missing data, greater than 50% of responses. Internal consistency was evaluated and reported using Cronbach alpha. A post-test evaluation of the student experience using the SCT was reported as percentages who agreed or strongly agreed with statements about the test and process. Data were analysed using Stata version 15 (StataCorp, College Station, Texas) and statistical significance was assigned when P < 0.05.