College of Education
Permanent URI for this communityhttp://hdl.handle.net/1903/1647
The collections in this community comprise faculty research works, as well as graduate theses and dissertations..
Browse
10 results
Search Results
Item Informant Discrepancies: Understanding Differences in Parent and Teacher Ratings of Children's Executive Functions and Social Skills(2015) Albrecht, Jessica; Teglasi, Hedy; Counseling and Personnel Services; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)Researchers and practitioners in the field of psychology frequently use parent and teacher rating scales in the assessment, diagnosis, and treatment of young children. However, research has shown that agreement between parents and teachers on rating scales is low to moderate. The present study examined this phenomenon, termed "informant discrepancy", for the Behavior Rating Inventory of Executive Functions (BRIEF) and the Social Skills Improvement System (SSIS). Parents and teachers completed these scales for the same sample of 73 Kindergarten children. Results indicated that parent-teacher agreement was low at the scale and item levels, within-informant correlations were higher than between-informant correlations, mean differences in parent and teacher ratings may be explained by differences in the home and school contexts, and informants identified different children as having significant problems with executive functions and social skills. Implications of the findings for research and practice are discussed.Item DIFFERENT APPROACHES TO COVARIATE INCLUSION IN THE MIXTURE RASCH MODEL(2014) Li, Tongyun; Jiao, Hong; Measurement, Statistics and Evaluation; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)The present dissertation project investigates different approaches to adding covariates and the impact in fitting mixture item response theory (IRT) models. Mixture IRT models serve as an important methodology for tackling several important psychometric issues in test development, including detecting latent differential item functioning (DIF). A Monte Carlo simulation study is conducted in which data generated according to a two-class mixture Rasch model (MRM) with both dichotomous and continuous covariates are fitted to several MRMs with misspecified covariates to examine the effects of covariate inclusion on model parameter estimation. In addition, both complete response data and incomplete response data with different types of missingness are considered in the present study in order to simulate practical assessment settings. Parameter estimation is carried out within a Bayesian framework vis-à-vis Markov chain Monte Carlo (MCMC) algorithms. Two empirical examples using the Programme for International Student Assessment (PISA) 2009 U.S. reading assessment data are presented to demonstrate the impact of different specifications of covariate effects for an MRM in real applications.Item CROSS-CLASSIFIED MODELING OF DUAL LOCAL ITEM DEPENDENCE(2014) Xie, Chao; Jiao, Hong; Measurement, Statistics and Evaluation; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)Previous studies have mainly focused on investigating one source of local item dependence (LID). However, in some cases, such as scenario-based science assessments, LID might be caused by two possible sources simultaneously. In this study, such kind of LID that is caused by two factors simultaneously is named as dual local item dependence (DLID). This study proposed a cross-classified model to account for DLID. Two simulation studies were conducted with the primary purpose of evaluating the performance of the proposed cross-classified model. Data sets with DLID were simulated with both testlet effects and content clustering effects. The second purpose of this study was to investigate the potential factors affecting the need to use the more complex cross-classified modeling of DLID over the simplified multilevel modeling of LID by ignoring cross-classification structure. For both simulation studies, five factors were manipulated, including sample size, number of testlets, testlet length, magnitude of the testlet effects represented by standard deviations (SDs), and magnitude of the content clustering effects represented by SDs. The difference between the two simulation studies was that, simulation study 1 constrained the SDs of the testlet effects and content clustering effects as the same across testlets and content areas, respectively; simulation study 2 released this constraint by having mixed SDs of the testlet effects and mixed SDs of the content clustering effects. Results of both simulation studies indicated that the proposed cross-classified model yielded more accurate parameter recovery, including item difficulty, persons' ability, and random effects' SD parameters with smaller estimation errors than the two multilevel models and the Rasch model which ignored one or both item clustering effects. The two manipulated variables, the magnitude of the testlet effects and the magnitude of the content clustering effects, determined the necessity of using the more complex cross-classified model over the simplified multilevel models and the Rasch model: the larger the magnitude of the testlet effects and the content clustering effects, the more necessary to use the proposed cross-classified model. Limitations are discussed and suggestions for future research are presented at the end.Item A MIXED-STRATEGIES RASCH TESTLET MODEL FOR LOW-STAKES TESTLET-BASED ASSESSMENTS(2013) Chen, Ying-Fang; Jiao, Hong; Measurement, Statistics and Evaluation; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)In low-stakes assessments, a lack of test-taking motivation inevitably occurs because test scores impose inconsequential effects on test takers' academic records. A common occurrence is that some test takers are unmotivated and simply apply random guessing strategy rather than solution strategy in taking a test. Testlet effects also arise because educational assessment items are frequently written in testlet units. A challenge to psychometric measurement is that conventional item response theory models do not sufficiently account for test-taking motivation heterogeneity and testlet effects. These construct-irrelevant variances affect test validity, accuracy of parameter estimates, and targeted inferences. This study proposes a low-stakes assessment measurement model that can simultaneously explain test-taking motivation heterogeneity and testlet effects. The performance and effectiveness of the proposed model are evaluated through a simulation study. Its utility is demonstrated through an application to a real standardized low-stakes assessment dataset. Simulation results show that overlooking test-taking motivation heterogeneity and testlet effects adversely affected model-data fit and model parameter estimates. The proposed model improved model-data fit and classification accuracy and well recovered model parameters under test-taking motivation heterogeneity and testlet effects. For the real data application, the item response dataset, which was originally calibrated with the Rasch model, was fitted better by the proposed model. Both test-taking motivation heterogeneity and testlet effects were identified in the real dataset. Finally, a set of variables selected from the real dataset is used to explore potential factors that characterize the latent classes of test-taking motivation. In the science assessment, science proficiency was associated with test-taking motivation heterogeneity.Item THE MIXTURE DISTRIBUTION POLYTOMOUS RASCH MODEL USED TO ACCOUNT FOR RESPONSE STYLES ON RATING SCALES: A SIMULATION STUDY OF PARAMETER RECOVERY AND CLASSIFICATION ACCURACY(2013) Cho, Youngmi; Harring, Jeffrey R; Measurement, Statistics and Evaluation; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)Response styles presented in rating scale use have been recognized as an important source of systematic measurement bias in self-report assessment. People with the same amount of a latent trait may be a victim of a biased test score due to the construct's irrelevant effect of response styles. The mixture polytomous Rasch model has been proposed as a tool to deal with the response style problems. This model can be used to classify respondents with different response styles into different latent classes and provides person trait estimates that have been corrected for the effect of a response style. This study investigated how well the mixture partial credit model (MPCM) recovered model parameters under various testing conditions. Item responses that characterized extreme response style (ERS), middle-category response style (MRS), and acquiescent response style (ARS) on a 5-category Likert scale as well as ordinary response style (ORS), which does not involve distorted rating scale use, were generated. The study results suggested that ARS respondents could be almost perfectly classified from other response-style respondents while the distinction between MRS and ORS respondents was most difficult followed by the distinction between ERS and ORS respondents. The classifications were more difficult when the distorted response styles were present in small proportions within the sample. Ten-items and a sample size of 3000 appeared to warrant reasonable threshold and person parameter estimation under the simulated conditions in this study. As the structure of mixture of response styles became more complex, increased sample size, test length, and balanced mixing proportion were needed in order to achieve the same level of recovery accuracy. Misclassification impacted the overall accuracy of person trait estimation. BIC was found to be the most effective data-model fit statistic in identifying the correct number of latent classes under this modeling approach. The model-based correction of score bias was explored with up to four different response-style latent classes. Problems with the estimation of the model including non-convergence, boundary threshold estimates, and label switching were discussed.Item Dynamic Bayeian Inference Networks and Hidden Markov Models for Modeling Learning Progressions over Multiple Time Points(2012) Choi, Younyoung; Mislevy, Robert J; Measurement, Statistics and Evaluation; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)The current study examines the performance of a Bayesian Inference Network (BIN) for modeling Learning Progressions (LP) as a longitudinal design approach. Recently, Learning Progressions, defined by measurable pathways that a student may follow in building their knowledge and gaining expertise over time (National Research Council, 2007; Shin, Stevens, Short & Krajcik, 2009), have captured attention in mathematics and science education (Learning Progressions in Science Conference, 2009). While substantive, psychological, instructional, and task developmental aspects has been proposed in the LP framework, few assessment design frameworks have been designed to link the theory embodied in a progression, tasks that provide evidence about a student's level on that progression, and psychometric models that can link them. Specially, few psychometric models have been proposed to characterize the relationship between student performance and levels on learning progressions in a longitudinal design approach. This dissertation introduces an approach to modeling LPs over multiple time points using Bayesian Inference Networks, referred to as dynamic Bayesian Inference Networks (DBINs). The DBINs are a framework for modeling LPs over time by integrating the theory embodying LPs, assessment design, and interpretation of student performances. The technical aspects of this dissertation cover the fundamental concepts of the graphical model for constructing a DBIN. It is shown that this modeling strategy for change over multiple time points is equivalent to a hidden Markov model. An expectation-maximization (EM) algorithm is presented for estimating the parameters in the model. Two simulation studies are conducted that focus on the construction of a simple DBIN model and an expanded DBIN model with a covariate. The extension that incorporates a covariate for students is useful for studying the effect of instructional treatments, students' background, and motivation on a student's LP. An application illustrates the ideas with real data from the domain of beginning computer network engineering drawn from work in the Cisco Networking Academy.Item RANDOMIZATION-BASED INFERENCE ABOUT LATENT VARIABLES FROM COMPLEX SAMPLES: THE CASE OF TWO-STAGE SAMPLING(2012) Li, Tiandong; Mislevy, Robert J.; Measurement, Statistics and Evaluation; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)In large-scale assessments, such as the National Assessment of Educational Progress (NAEP), plausible values based on Multiple Imputations (MI) have been used to estimate population characteristics for latent constructs under complex sample designs. Mislevy (1991) derived a closed-form analytic solution for a fixed-effect model in creating plausible values assuming a classical test theory model and a stratified student sample and proposed an analogous solution for a random-effects model to be applied with a two-stage student sample design. The research reported here extends the discussion of this random-effects model under the classical test theory framework. Under the simplified assumption of known population parameters, analytical solutions are provided for multiple imputations in the case of the classical test theory measurement model and two-stage sampling and their properties are verified in reconstructing population properties for the unobservable latent variables. With the more practical assumptions of unknown population and cluster means, this study empirically examines the reconstruction of population attributes. Next, properties of sample statistics are examined. Specifically, this research explores the impact of the variance components and sample sizes on the sampling variance of the MI-based estimate for the population mean. Findings include significant predictors and influential factors. Last, the relationships between the sampling variance of the estimate of the population mean based on the imputations and those based on observations of the true score and the observed score are discussed. The sampling variance based on the imputed score is expected to be the higher boundary of that based on the observed score, which is expected to be the higher boundary of that based on the true score.Item ESTIMATING UNKNOWN KNOTS IN PIECEWISE LINEAR-LINEAR LATENT GROWTH MIXTURE MODELS(2011) Kohli, Nidhi; Hancock, Gregory R; Harring, Jeffrey R; Measurement, Statistics and Evaluation; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)A piecewise linear-linear latent growth mixture model (LGMM) combines features of a piecewise linear-linear latent growth curve (LGC) model with the ideas of latent class methods all within a structural equation modeling (SEM) context. A piecewise linear-linear LGMM is an appropriate framework for analyzing longitudinal data that come from a mixture of two or more subpopulations (i.e., latent classes) where each latent class incorporates a separate growth trajectory corresponding to multiple growth phases from which repeated measurements arise. The benefit of the model is that it allows the specification of each growth phase to conform to a particular form of overall change process within each latent class thereby making these models flexible and useful for substantive researchers. There are two main objectives of this current study. The first objective is to demonstrate how the parameters of a piecewise linear-linear LGMM, including the unknown knot, can be estimated using standard SEM software. A series of Monte Carlo simulations empirically investigated the ability of piecewise linear-linear LGMMs to recover true (known) growth parameters of distinct populations. Specifically, the current research compared the performance of the piecewise linear-linear LGMM under different manipulated conditions of 1) sample size, 2) class mixing proportions, 3) class separation of location of knot, 4) the mean of the slope growth factor of the second phase, 5) the variance of the slope growth factor of the second phase, and 6) residual variance of the observed variables. The second objective is to address the issue of model mis-specification. It is important to analyze this issue because applied researchers have to make model selection decisions. Therefore, the current research examined the possibility of extracting spurious latent classes. To achieve this objective 1-, 2-, and 3-class piecewise linear-linear LGMMs were fit to data sets generated under different manipulated conditions using a 2-class piecewise linear-linear LGMM as a population model. The number of times the correct model (i.e., 2-class piecewise linear-linear LGMM) was preferred over incorrect models (i.e., 1- and 3-class piecewise linear-linear LGMMs) using the Bayesian Information Criterion (BIC) was examined. Results suggested that the recovery of model parameters, specifically, the variances of growth factors were generally poor. In addition, none of the manipulated conditions were systematically related to the outcome measures, parameter bias and variability index of parameter bias. Furthermore, among all the manipulated conditions, the residual variance of observed variable had the strongest statistically significant effect on both the model convergence rate and the model selection rate. Other manipulated conditions that had an impact on the model convergence rate and/or the model selection rate were the growth factor mean of slope of the second phase, the growth factor variance of slope of the second phase, and the class mixing proportion. The manipulated conditions whose levels had no influence on either the model convergence rate or the model selection rate were sample size and the class separation of location of knot.Item EFFECTS OF UNMODELED LATENT CLASSES ON MULTILEVEL GROWTH MIXTURE ESTIMATION IN VALUE-ADDED MODELING(2011) Yumoto, Futoshi; Hancock, Gregory R; Mislevy, Robert J; Measurement, Statistics and Evaluation; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)Fairness is necessary to successful evaluation, whether the context is simple and concrete or complex and abstract. Fair evaluation must begin with careful data collection, with clear operationalization of variables whose relationship(s) will represent the outcome(s) of interest. In particular, articulating what it is in the data that needs to be modeled, as well as the relationships of interest, must be specified before conducting any research; these two features will inform both study design and data collection. Heterogeneity is a key characteristic of data that can complicate the data collection design, and especially analysis and interpretation, interfering with or influencing the perception of the relationship(s) that the data will be used to investigate or evaluate. However, planning for, and planning to account for, heterogeneities in data are also critical to the research process, to support valid interpretation of results from any statistical analysis. The multilevel growth mixture model is a new analytic method specifically developed to accommodate heterogeneity so as to minimize the effect of variability on precision in estimation and to reduce bias that may arise in hierarchical data. This is particularly important in the Value Added Model context - where decisions and evaluations about teaching effectiveness are made, because estimates could be contaminated, biased, or simply less precise when data are modeled inappropriately. This research will investigate the effects of un-accounted for heterogeneity at level 1 on the precision of level-2 estimates in multilevel data utilizing the multilevel growth mixture model and multilevel linear growth model.Item USING LATENT PROFILE MODELS AND UNSTRUCTURED GROWTH MIXTURE MODELS TO ASSESS THE NUMBER OF LATENT CLASSES IN GROWTH MIXTURE MODELING(2011) Liu, Min; Hancock, Gregory R.; Measurement, Statistics and Evaluation; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)Growth mixture modeling has gained much attention in applied and methodological social science research recently, but the selection of the number of latent classes for such models remains a challenging issue. This problem becomes more serious when one of the key assumptions of this model, proper model-specification is violated. The current simulation study compared the performance of a linear growth mixture model in determining the correct number of latent classes against two less parametrically restricted options, a latent profile model and an unstructured growth mixture model. A variety of conditions were examined, both for properly and improperly specified models. Results indicate that prior to the application of linear growth mixture model, the unstructured growth mixture model is a promising way to identify the correct number of unobserved groups underlying the data by using most model fit indices across all the conditions investigated in this study.