Table 3 Summary of validity and reliability evidence for the GeDI
Validity/reliability evidence type and descriptiona | CTT framework (Price et al. 2014) | Rasch framework (present study) |
|---|---|---|
Construct validity  Instrument appropriately represents the specified knowledge domain | Textbook analysis, expert survey, student interviews, review of student work and literature review for misconceptions | Rasch model fit, Rasch dimensionality analysis, item fit, person reliability |
Substantive validity  Participants use the thought processes that were anticipated for each item | Student interviews | (None) |
Internal structure validity  Items capture a single construct | Cronbach’s alpha | Rasch dimensionality test, person and item reliability |
External structure validity: Â Scores are appropriately associated (positively or negatively) with an independent measure | (None) | (None) |
Generalization validity  Score inferences hold true in different administrative contexts | Five campuses over two geographic areas (Southeast/Midwest) | New population (Northeast) |
Consequential validity  Considers positive or negative consequences of score use | Not applicable | Not applicable |
Reliability  Reproducibility of scores | Test–retest | Item and person reliabilities |
Item properties  Individual item performance characteristics | Difficulty, discrimination | Item measures, item fit statistics, Wright map |
Item order effects  Possible item interactions and associated sequence biases | (None) | ANOVA of Rasch-scaled scores from forms rotating item-suite order |