Table 7 Themes identified and example responses regarding OSCEs and SimSurg as viewed by GP assessors

OSCE benefits

OSCEs assess specific knowledge, skills and attitudes through specific task instructions. This is not the case in SimSurg where candidates perform an overall consultation. [OSCEs] “more specific to given clinical problem, might help in assessing GP skills that initial assessment might indicate are uncertain, or poor or indeed good.”

OSCEs can assess specific situations such as Basic Life Support, or a discussion with a colleague, which are not assessed in SimSurg.

“Benefit ought to be evidence of specific individual skills - Basic life support is a very useful test if someone says they attend lots of post-graduate training but cannot do BLS, it provides some evidence that they are not benefitting from educational activities. Talking to a practice nurse, a worried colleague, or the son of a nursing home patient are all examples of skills that are better tested in the OSCE model.”

SimSurg benefits

Specific skills, such as prescribing, are integrated into the consultation rather than as a separate assessment. “GPs do not practice in bits and whole consultations are generally much more informative regarding information gathering and information sharing than OSCEs.”

SimSurg is more “authentic” to real consultations, as the scenario is based on the simulated patients’ presenting complaint. “SimSurg is the nearest thing to real consulting, which is what GPs have to do. They are ‘real life’ situations. They cannot be ‘fixed’ as observed practice [during peer review visits] can be, with doctors suspected of inviting regular repeat patients who know what to say… The SimSurg is observed remotely which is more realistic than two or three assessors crowding into a room with the doctor and the role player.”

SimSurg is more “global”, providing a broader, more “holistic” overall picture of candidate performance. “Simulated surgery offers a global picture of the doctors consulting which is often very valuable as some GPs have not been seen practicing for real e.g. if suspended. A GP can do an examination (OSCE) but when asked to integrate that into a wider whole - may perform poorly”

OSCE limitations

OSCEs are more intimidating as the assessor is in the room. “I think it is more intimidating for doctor bring assessed to have assessor a in the room as opposed to watching on CCTV”

Low face validity

“…in reality we do not address specific aspects, but do so within the context of a consultation”

Penalises OSCE-naïve candidates

“Many older GPs don’t understand the concept of an OSCE so attempt a whole consultation even when they have been briefed.”

SimSurg limitations

Each scenario usually takes longer than a similar OSCE scenario. “Whole consultations are needed which takes up time and may be testing similar skills repeatedly.”

Less candidate instruction means there is more potential for the consultation to go off tangent. “In the SimSurg, the doctor may decide to defer an examination or a more detailed questioning, and the assessors may not be able to fault this decision - in the OSCE it is clearer whether or not the doctor has the relevant skill/knowledge.”

SimSurg can “mask” poor performance, as good interpersonal skills can hide candidate knowledge or competency deficiencies

“Good consulting skills/generic interpersonal skills can mask some weaknesses in knowledge or practical skills. Simulations do generally not test specific examination or practical skills or the ability to make good documentation etc. This is particularly a problem with training doctors who will need to have more procedural skills than experienced GPs because of hospital attachments”