THE ROLE OF READING COMPREHENSION IN LARGE-SCALE SUBJECT-MATTER ASSESSMENTS

Abstract

This study was designed with the overall goal of understanding how difficulties in reading comprehension are associated with early adolescents' performance in large-scale assessments in subject domains including science and civic-related social studies. The current study extended previous research by taking a cognition-centered approach based on the Evidence-Centered Design (ECD) framework and by using U.S. data from four large-scale subject-matter assessments: the IEA TIMSS Science Study of 1999, IEA CIVED Civic Education Study of 1999, and the 1970s IEA Six Subject surveys in Science, and in Civic Education. Using multiple-choice items from the TIMSS science and CIVED tests, the study identified a list of linguistic features that contribute to item difficulty of subject-matter assessments through the Coh-Metrix software, human rating, and multiple regression analysis. These linguistic features include word length, word frequency, word abstractness, intentional verbs, negative expressions, and logical connectives. They pertain to different levels of Kintsch's reading comprehension model: surface level, textbase level, and situation model. Integrating this item-level information into multiple regression analysis and Multidimensional IRT modeling, the study provided feasible methods (1) to estimate reading demand of test items in each subject-matter assessment, and (2) to partial out variance related to high level of reading demand of some test items and independent of the domain proficiencies that the subject-matter assessment was intended to measure. Overall, results suggested that reading demands of all test items in TIMSS Science and CIVED tests were within the reading capabilities of almost all of the students, and these two tests were not saturated with high reading demand. In addition, multiple regression results from the earlier Six Subject Surveys showed that an independent measure of students' general vocabulary was highly correlated with their achievement in the domains of science and civic-related social studies. On average, boys outperformed girls in both subject domains, and students from home with ample literacy resources outperformed students from homes of few literacy resources. In the science assessment, interactions were found between gender and word knowledge, home literacy resources and word knowledge, meaning the correlation between vocabulary and science performances differed by gender and home background.