Test order in teacher-rated behavior assessments: Is counterbalancing necessary
Article
Kooken, J, Welsh, ME, McCoach, DB et al. (2017). Test order in teacher-rated behavior assessments: Is counterbalancing necessary
. PSYCHOLOGICAL ASSESSMENT, 29(1), 98-109. 10.1037/pas0000314
Kooken, J, Welsh, ME, McCoach, DB et al. (2017). Test order in teacher-rated behavior assessments: Is counterbalancing necessary
. PSYCHOLOGICAL ASSESSMENT, 29(1), 98-109. 10.1037/pas0000314
Counterbalancing treatment order in experimental research design is well established as an option to reduce threats to internal validity, but in educational and psychological research, the effect of varying the order of multiple tests to a single rater has not been examined and is rarely adhered to in practice. The current study examines the effect of test order on measures of student behavior by teachers as raters utilizing data from a behavior measure validation study. Using multilevel modeling to control for students nested within teachers, the effect of rating an earlier measure on the intercept or slope of a later behavior assessment was statistically significant in 22% of predictor main effects for the spring test period. Test order effects had potential for high stakes consequences with differences large enough to change risk classification. Results suggest that researchers and practitioners in classroom settings using multiple measures evaluate the potential impact of test order. Where possible, they should counterbalance when the risk of an order effect exists and report justification for the decision to not counterbalance.
