INVESTIGATING DIFFERENTIAL ITEM FUNCTION AMPLIFICATION AND CANCELLATION IN APPLICATION OF ITEM RESPONSE TESTLET MODELS

Abstract

Many educational tests use testlets as a way of providing context, instead of presenting only discrete multiple-choice items, where items are grouped into testlets (Wainer & Kiely, 1987) or item bundles (Rosenbaum, 1988) marked by shared common stimulus materials. One might doubt the usual assumption of standard item response theory of local independence among items in these cases. Plausible causes of local dependence might be test takers' different levels of background knowledge necessary to understand the common passage, as a considerable amount of mental processing may be required to read and understand the stimulus, and different persons' learning experiences. Here, the local dependence can be viewed as additional dimensions other than the latent traits. Furthermore, from the multidimensional differential item functioning (DIF) point of view, different distributions of testlet dimensions among different examinee subpopulations (race, gender, etc) could be the cognitive cause of individual differences in test performance. When testlet effect and item idiosyncratic features of individual items are both considered to be the reasons of DIF, it is interesting to investigate the phenomena of DIF amplification and cancellation resulting from the interactive effects of these two factors.

This dissertation presented a study based on a multiple-group testlet item response theory model developed by Li et al. (2006) to examine in detail different situations of DIF amplification and cancellation at the item and testlet level using testlet characteristic curve procedures with signed/ unsigned area indices and logistic regression procedure. The testlet DIF model was estimated using a hierarchical Bayesian framework with the Markov Chain Monte Carlo (MCMC) method implemented in the computer software WINBUGS. The simulation study investigated all of the possible conditions of DIF amplification and cancellation attributed to person-testlet interaction effect and individual item characteristics. Real data analysis indicated the existence of testlet effect and its magnitudes of difference on the means and/or variance of testlet distribution between manifest groups imputed to the different contexts or natures of the passages as well as its interaction with the manifest groups of examinees such as gender or ethnicity.