A COMPARISON OF DIFFERENT METHODS THAT DEAL WITH CONSTRUCT SHIFT IN VALUE ADDED MODELING: IS VERTICAL SCALING NECESSARY?

Abstract

Construct shift is a term used to describe the change of tests in the construct they intend to measure. In tests across multiple grades where curriculum change occurs, construct shift is expected to exist. This presents a problem to many VAM models that assume scores across multiple grades are on a common developmental scale, since these scores cannot be placed on the same scale through vertical scaling. There are three methods currently available to deal with construct shift: the CU method ignores construct shift, carry out the vertical scaling process with a unidimensional IRT model, and directly use the vertically scaled scores in specific VAM models that require vertical scaling; the CB method models construct shift, carry out the vertical scaling process with a bifactor model and use the scores on the general factor in specific VAM models that require vertical scaling; the SU method does not use vertical scaling but directly applies the scores at each grade in the generalized persistence (GP) model. A simulation study was conducted to compare the impacts of construct shift upon teacher rank ordering estimation with those three methods.

Results suggest that the performances of all three methods are subject to the influence of magnitude of construct shift and choice of teacher effect persistence pattern. The CB and the CU methods perform similarly, while the SU method is superior to them in most simulation conditions. Only with large magnitude of construct shift is the CB method slightly better than the SU method in terms of the last year's teacher effect estimation. The CB method performs better than the CU method with large magnitude of construct shift, while they perform similarly with small or medium magnitude of construct shift. It is concluded that the SU method performs the best among those three methods and is recommend for use in practice.