In this dissertation we develop mathematical models of tumor heterogeneity and drug resistance in cancer chemotherapy. Resistance to chemotherapy is one of the major causes of the failure of cancer treatment. Furthermore, recent experimental evidence suggests that drug resistance is a complex biological phenomena, with many influences that interact nonlinearly. Here we study the influence of such heterogeneity on treatment outcomes, both in general frameworks and under specific mechanisms.

We begin by developing a mathematical framework for describing multi-drug resistance to cancer. Heterogeneity is reflected by a continuous parameter, which can either describe a single resistance mechanism (such as the expression of P-gp in the cellular membrane) or can account for the cumulative effect of several mechanisms and factors. The model is written as a system of integro-differential equations, structured by the continuous ``trait," and includes density effects as well as mutations. We study the limiting behavior of the model, both analytically and numerically, and apply it to study treatment protocols.

We next study a specific mechanism of tumor heterogeneity and its influence on cell growth: the cell-cycle. We derive two novel mathematical models, a stochastic agent-based model and an integro-differential equation model, each of which describes the growth of cancer cells as a dynamic transition between proliferative and quiescent states. By examining the role all parameters play in the evolution of intrinsic tumor heterogeneity, and the sensitivity of the population growth to parameter values, we show that the cell-cycle length has the most significant effect on the growth dynamics. In addition, we demonstrate that the agent-based model can be approximated well by the more computationally efficient integro-differential equations, when the number of cells is large. The model is closely tied to experimental data of cell growth, and includes a novel implementation of transition rates as a function of global density.

Finally, we extend the model of cell-cycle heterogeneity to include spatial variables. Cells are modeled as soft spheres and exhibit attraction/repulsion/random forces. A fundamental hypothesis is that cell-cycle length increases with local density, thus producing a distribution of observed division lengths. Apoptosis occurs primarily through an extended period of unsuccessful proliferation, and the explicit mechanism of the drug (Paclitaxel) is modeled as an increase in cell-cycle duration. We show that the distribution of cell-cycle lengths is highly time-dependent, with close time-averaged agreement with the distribution used in the previous work. Furthermore, survival curves are calculated and shown to qualitatively agree with experimental data in different densities and geometries, thus relating the cellular microenvironment to drug resistance.