Table 1 Types of test validity evidence

… test content

Checking the match between the assessment’s content and what it claims to measure

Getting the assessment reviewed by multiple content experts to ensure even and thorough coverage of a concept

… response processes

Making sure that the test takers’ responses align with the underlying concepts being measured

Think-aloud interviews with students where they explain the reasoning behind each answer they chose on the test

… internal structure

Analyzing how the assessment questions relate to each other and map onto the underlying concepts being measured

Using a statistical modeling approach like factor analysis to analyze how responses to different questions correlate

… relations with other variables

Relating test scores to external variables. The external variable could be the results of a similar test (convergent evidence), a test expected to be different (discriminant evidence), or non-test outcomes like students’ future course enrollment decisions

Comparing the assessment to a previously established assessment of similar concepts, expecting to find a positive correlation in student scores

… consequences of testing

Evaluating the soundness of using the assessment in practice, including potential impacts of the testing itself and appropriateness of interventions made based on student results

For an assessment being used to group students into leveled sections, gathering evidence on the learning outcomes of this intervention