Table 4 Review of methodological quality of studies using outcome measures to assess the impact of PVCs

Using interactive video to add physical assessment data to computer based patient simulations [22]

Used to justify criterion validity of the intervention used.

Study aims not clear. If presumed to be to elucidate whether the simulations are an effective learning experience the outcome used partially confirms the simulation represent normal practice not that the simulation improved performance or was an useful education tool

Author notes that in actual practice chart noting is done under time pressure whereas with this simulation there was more time available to make case-notes more complete.

No comment made on this. The absence of details on whether a specific proforma for extracting information from the case-notes was used makes it difficult to assess how comparisons were made.

No

No

The development of shared cognition in paediatric residents analysing a patient video versus a paper patient case [23]

Process of capturing concept link formation described with the reason for using verbal protocol analysis supported by published evidence.

Yes

Authors note a small number of participants and only one case so reliability may be questioned.

The positioning of the simulated recall exercise straight after the group work limits loss of content due to degradation of memories.

Yes

Yes

They also note the ability to accurately recall and record all concept links is not established.

Methodology of using simulated recall in individuals following the group discussions not well supported.

Authors comment although not blinded the interviewers were not recording cognitive processes just the thoughts that lead to them.

They were aware the interviewers were not blinded to the intervention group of the participants.

Enhancing diagnostic accuracy among non-experts through use of video cases [24]

Previous work by the author has shown improved cognitive processes when PVCs utilised. Improved diagnostic accuracy is the natural conclusion of relevant or improved cognitive processing.

Yes

Study acknowledges the diagnostic accuracy as an outcome is only a short term variable of learning.

Authors argue increase in diagnostic reasoning in non-experts in may promote further literature study and learning.

Yes

Yes

Authors touch on, but don’t specifically note, the outcome measure is not directly related to the intervention rather the group discussion following the intervention.

Introduction of patient video clips into computer-based testing: Effects on item statistics and reliability estimates [25]

No justification for answer analysis is given although standard methodology applied.

Yes

Problems with questions with low discrimination values identified and subject to supplemental analysis.

Items with RPB values of zero or less than 0.2 removed although no explanation of why these values were chosen.

Not Applicable

Not Applicable

Video-based test questions: A novel means of evaluation [26]

No justification for questionnaire methodology given although this format is an accepted primary approach to gathering information on satisfaction with a process.

Yes although reliability of results must be treated with caution.

No comments made

No comments made

Not Applicable

Not Applicable

Video-based test questions: A novel means of evaluation [26]

No justification given

Aims of the study not clear

No comments made

No comments made

Not Applicable

Not Applicable

A comparison of critical thinking in groups of third-year medical students in text, video, and virtual PBL case modalities [27]

There is a theoretical association between Problem Based Learning and critical thinking. A sound research framework exists to analyse discourse and code for content. Therefore a process, discourse analysis , exists to examine the outcomes of PBL in respect to critical thinking.

Yes

The outcome measure depends on the validity and reliability of the coder.

Only one author did all the coding but coding agreement was checked using a sample of transcripts with two others (one not involved in the study).

Yes

Yes

Although the coder is blind to group type it is possible for this to be suggested by the dialogue.

Time pressures on face-to-face groups may limit opportunities to refine critical thinking compared to the virtual groups.

Comparison of text and video cases in a postgraduate problem-based learning format [28]

A coding system for cognitive and metacognitive thinking has been established. It is theoretically plausible a video case would improve thinking processes.

Yes. Although the actual reason for improved educational outcome in PBL has yet to be defined and the coding schema chosen is only one way of evaluating cognitive and metacognitive processes.

The outcome measure itself is not examined although the article notes the use of one author for both groups and the low numbers of residents in both groups mean caution is required in interpretation.

The use of clause frequency enables variability in group sizes to be addressed.

Yes

Yes

Use of animation-enhanced video clips for teaching abnormal breathing patterns [13]

No justification given for questionnaire methodology although this format is an accepted primary approach to gathering information on satisfaction with a process.

Yes although the reliability of results must be treated with caution.

No comment made. The authors note that some respondents gave inconsistent written comments when compared with their agreement with statements and comment it was likely they had mis-understood the question.

No comments made

Yes

No

Using web-based video to enhance physical examination skills in medical students [14]

Summative clinical skills assessment has been utilised (and presumably validated although this is not stated) in the learning institution the study is taking place in.

Yes, although dependant on the reliability of the Clinical Skills Assessment.

The authors comment they did not track the utilisation of the video clip website by students and note a prospective, randomised controlled study would have been more accurate.

No comments made

Potentially Not

Yes

Teaching the plantar reflex [15]

No justification given. Process for assessing performance described although the standardised rating scale was not demonstrated.

Yes but only if the assessment system is valid.

The authors acknowledge the effect of the entrance test in providing education in itself.

No comment made

Potentially Not

Yes

A videotape-based training method for improving the detection of depression in residents of long-term care facilities [29]

Outcome measures well described but no comment on the reason for using them.

As patient outcome not measured methodology can only assess how the training programme improves performance in the outcomes tested.

Authors not a large sample size is needed to counter affects of attrition.

Authors tested after a control period and withheld feedback to participants about their test performances.

Potentially Not

Yes

They also not the vignette video test may improve practice in its own right.

Staff were allowed to choose timing and type of session according to their needs with no control for group or individual sessions.

Advantages of video trigger in problem-base learning [30]

No justification for questionnaire methodology given although this format is an accepted primary approach to gathering information on satisfaction with a process.

Yes as long as sample of participants valid.

No comments made

No comments made

Yes

Yes

A triangulated approach to the assessment of teaching in childhood epilepsy [31]

Triangulation used as a more complete and robust measure to validate findings.

Yes although must be employed in a methodological fashion. However the use of video clips was not the sole purpose of the study and questions not posed to determine this.

Very little attention paid to confounding influences and the fact that the cross –validation was not particular well demonstrated.

No comments made

No

Not applicable

How video cases should be used as authentic stimuli in problem-based medical education [32]

Focus groups a well refined qualitative tool which all deep analysis of concepts presented.

Yes

Clear acknowledgement of the problems with individuals dominating or evading group discussion.

Very experienced facilitator used

Yes

Yes

Visual expertise in paediatric neurology [33]

Important differences in perception between experts and novices studying dynamic stimuli has been documented . Authors note this field is underexplored in the medical domain but use a high quality eye tracking machine and linked to spoken cognitive processes

Yes

Note that outcome method was novel and made efforts to triangulate findings to gold standard outcomes (such as correct diagnosis)

Used variety of experience in subject population

Yes

Yes

An evaluation of the effectiveness of a videotape programme on inter-observer reliability in outcome assessment for osteoarthritis [34]

Outcome measure used in previous studies to assess performance in musculo-skeletal examination.

Yes

A larger matrix and more observers and patients may have been used to improve reliability.

The authors own previous work has indicated the 6x6 is pragmatic and representative.

No

Yes

Osteoarthritis antirheumatic drug trials: Effects of a standardized instructional videotape on the reliability of observer-dependent dependent outcome measures [35]

Outcome measure used in previous studies to assess performance in musculo-skeletal examination.

Yes

A larger matrix and more observers and patients may have been used to improve reliability.

The authors own previous work has indicated the 6x6 is pragmatic and representative.

No

Yes