DATA-DRIVEN DECISION-MAKING FOR SOCIETAL SYSTEMS: AUDITING MOBILITY, CONTEXTUAL DATA AND MODELS FOR HIGH-STAKES APPLICATIONS

Abstract

High-stakes decisions in domains such as public health and economic policy are increasingly mediated by behavioral data and predictive models, yet the phenomena they target often evolve faster than conventional measurement systems can track. This has driven growing reliance on human mobility and contextual data as real-time proxies and inputs to predictive models, a dependence made especially visible during the COVID-19 pandemic. Yet these data, and the models built on top of them, are rarely subjected to the kinds of systematic evaluation needed to establish whether they can be trusted, for whom their benefits hold, and under what conditions their measurements and predictions fail to hold. This dissertation argues that mobility, contextual data, and the associated downstream models must be audited along three dimensions before being trusted in high-stakes settings: utility, fairness, and validity. It further argues that the same scrutiny should be applied to emerging alternatives for direct behavioral traces, such as large language models used to infer or simulate human mobility when observational data are sparse or unavailable.

The first study examines the utility of mobility data for COVID-19 case prediction, asking not whether it improves performance on average, but under what conditions across counties, datasets, and modeling choices its value can be reliably demonstrated. The second study conducts a fairness audit of the U.S. COVID-19 Forecast Hub, examining whether the case prediction models used to inform CDC communications performed equitably across racial, ethnic, and urbanization dimensions. The third study evaluates the validity of foot traffic as a proxy for economic activity, assessing whether visit-spend relationships are stable across business categories, time periods, and neighborhood demographic contexts. The fourth study evaluates whether large language models can serve as an alternative for behavioral traces by asking whether they encode empirically meaningful knowledge about how neighborhood context relates to movement, or merely produce plausible but unverified outputs.

Together, these four studies demonstrate that the trustworthiness of mobility and contextual data is not a fixed property but a conditional one, dependent on the task, the population, and the context in which these data are used. The dissertation contributes a unifying auditing framework and a set of empirical evaluations that together make the case for treating utility, fairness, and validity as fundamental criteria in any data-driven system where predictions shape real-world decisions.