Compartment fires with different ventilation conditions exhibit different dynamical behaviors, ranging from steady fuel-limited fires to unsteady air-limited fires. Numerical simulations are here performed to study compartment fires in a configuration corresponding to a scaled-down model developed at University of Maryland, in which experimental data are available. The simulations use Fire Dynamics Simulator (FDS) developed by National Institute of Science and Technology (NIST). Four different cases are studied that are representative of different fire conditions: steady over-ventilated fires; steady under-ventilated fires; and unsteady fires with partial flame quenching; unsteady fires leading to total flame quenching. To account for air vitiation and flame extinction effects, a new flame extinction model is developed and integrated into FDS. It is found that the new model improves the numerical predictions and offers the potential of a better representation of the flame dynamics and upper-layer gas composition.