A MIXED-STRATEGIES RASCH TESTLET MODEL FOR LOW-STAKES TESTLET-BASED ASSESSMENTS

Abstract

In low-stakes assessments, a lack of test-taking motivation inevitably occurs because test scores impose inconsequential effects on test takers' academic records. A common occurrence is that some test takers are unmotivated and simply apply random guessing strategy rather than solution strategy in taking a test. Testlet effects also arise because educational assessment items are frequently written in testlet units. A challenge to psychometric measurement is that conventional item response theory models do not sufficiently account for test-taking motivation heterogeneity and testlet effects. These construct-irrelevant variances affect test validity, accuracy of parameter estimates, and targeted inferences. This study proposes a low-stakes assessment measurement model that can simultaneously explain test-taking motivation heterogeneity and testlet effects. The performance and effectiveness of the proposed model are evaluated through a simulation study. Its utility is demonstrated through an application to a real standardized low-stakes assessment dataset. Simulation results show that overlooking test-taking motivation heterogeneity and testlet effects adversely affected model-data fit and model parameter estimates. The proposed model improved model-data fit and classification accuracy and well recovered model parameters under test-taking motivation heterogeneity and testlet effects. For the real data application, the item response dataset, which was originally calibrated with the Rasch model, was fitted better by the proposed model. Both test-taking motivation heterogeneity and testlet effects were identified in the real dataset. Finally, a set of variables selected from the real dataset is used to explore potential factors that characterize the latent classes of test-taking motivation. In the science assessment, science proficiency was associated with test-taking motivation heterogeneity.