Skip to main content

Table 1 Sources of validity evidence (adapted from Messick 1995, Andreatta and Gruppen 2009 and Cook and Beckman 2006)

From: Validity of a new assessment rubric for a short-answer test of clinical reasoning

Source of validity evidence

Description of validity evidence

Method of generating validity evidence for the CHAT

1.Instrument content

Extent to which instrument content is relevant to the construct of interest

Developing a test blueprint for clinical reasoning in OMPT

2.Response process

Extent to which the cognitive and physical processes required by the instrument can represent the construct of interest

Examining feasibility and acceptability of scoring procedure

3.Instrumentā€™s internal structure

Extent to which the transformation of assessment results into a score reflects the underlying construct

Calculating internal consistency

Establishing inter-rater reliability

Conducting factor analysis

4.Relationships between assessment scores and other variables

Extent to which assessment results relate with other variables that possess a predicted association with the construct of interest

Examining correlation with other clinical reasoning measures

5.Consequences of decisions made based on assessment results

Evidence pertaining to intended and unintended consequences of interpreting and using assessment results

Establishing and examining method of determining pass/fail on case history examination