ADAPTING NEXT-TOKEN PREDICTION FOR PRACTICAL DOMAINS AND TRAINING REGIMES

Abstract

Modern transformer-based machine learning has produced powerful systems, but most problems of practical interest are not natural language. This dissertation studies what it takes to bring the autoregressive next-token prediction recipe out of its native habitat and into practical domains and training regimes that look different from conventional benchmark NLP. The first half of the dissertation concerns adapting the recipe to new practical domains, primarily by changing how data is represented to the model rather than the architecture itself. STEP shows that a standard decoder-only transformer with a causal language modeling loss outperforms specialized architectures on tabular event prediction across financial fraud, product rating, loan default, and customer churn datasets, given column-aware tokenization and simple training-time augmentations. Abacus Embeddings introduces positional encodings designed for digit significance rather than absolute sequence position, enabling transformers to generalize arithmetic to inputs $6\times$ longer than those seen during training. The second half concerns adapting the recipe at the level of the training regime itself. GATES is a self-distillation framework that derives supervision online from consensus among privileged-context rollouts, improving language models on document-grounded question answering and out-of-distribution math benchmarks without ground-truth labels, verifiable rewards, or external graders. We also study practical concerns that arise once models are trained and deployed: a systematic taxonomy of adversarial coercion attacks on aligned language models, which exposes fundamental vulnerabilities in autoregressive generation, and earlier work on privacy in neural auction mechanisms, which shows how learned systems can leak private information through their outputs in strategic settings. Across these contributions, the recurring observation is that the autoregressive next-token prediction recipe is more general than its origins suggest, but only when it is adapted carefully---through the right representations, the right training signals, and a clear understanding of where it can fail---to each new setting.