Effects of Artificial Intelligence Generated Study Tools on Learning and Metacomprehension

Abstract

Recent advances in artificial intelligence technologies such as ChatGPT have provided new tools for students to implement when studying text materials. However, little empirical research has been conducted to evaluate whether use of such technologies is actually beneficial for learning and metacognitive outcomes. The present study connects the fields of artificial intelligence and metacognition to evaluate the effects of artificial intelligence-generated study materials on memory, comprehension, and metacognitive monitoring. Participants read a series of text passages about the natural and social sciences, made metacomprehension judgments, studied the text passages either by re-reading, reviewing a non-interactive ChatGPT-generated study guide, or reviewing an interactive ChatGPT-generated study guide with fill-in-the-blank components, and finally took a multiple-choice test assessing their memory and comprehension of the texts. While reviewing ChatGPT study guides did not improve overall, memory, or comprehension test performance above and beyond re-reading, participants reported greater enjoyment and engagement with reviewing interactive and non-interactive ChatGPT study guides as compared to re-reading text passages. For participants who reviewed interactive ChatGPT study guides, there was a boost in test performance on questions which matched fill-in-the-blank components and a reduction in overconfidence for metacognitive judgments, resulting in better absolute and relative metacomprehension accuracy compared to participants who re-read text passages or reviewed non-interactive ChatGPT study guides. Overall, results suggest that artificial intelligence generated study materials do not improve test performance relative to re-reading, but do provide some benefits for metacomprehension accuracy.