Optomechanical system, a hybrid system where mechanical and optical degrees of freedom are mutually coupled, is a new platform for studying quantum optics. In a typical optomechanical setup, the cavity is driven by a large amplitude coherent sate of light to enhance the effective optomechanical coupling. This system can be linearized around its classical steady state, and many interesting effects arise from the linearized optomechanical interaction, such as the dynamical modification of the properties of the mechanical resonator and the modulation of the amplitude and phase of the light coming out the of cavity. When the single-photon optomechanical coupling is comparable to the optical and mechanical loss, we must also keep the nonlinear interactions in the hamiltonian, which make it possible to study optomechanically induced nonlinear phenomena such as photon-blockade, Kerr nonlinearity, etc.

In this thesis, we study quantum optics with optomechanical systems both in the linear and nonlinear regime, with emphasis on its applications in force sensing and environmental engineering. We first propose a mirror-in-the-middle system and show that when driving near optomechanical instability, the optomechanical interaction will generate squeezed states of the output light. This system can be used to detect weak forces far below the standard quantum limit. Subsequently, we find that this particular driving scheme can also lead to enhanced optomechanical nonlinearity in a certain regime and by measuring the output field appropriately. We study the photon-blockade effect and discuss the conditions for maximum photon antibunching. We then focus on thermal noise reduction for mechanical resonators, by designing a system of two coupled resonators whose damping is primarily clamping loss. We show that optomechanical coupling to the clamping region enables dynamical control over the coupled mechanical resonator. This leads to the counterintuitive outcome: increasing optical power simultaneously reduces the temperature and linewidth of the mechanical mode, in contrast to direct optomechanical cooling. We also consider the Brillouin scattering induced optomechanical interaction in ring wave-guide resonators where phonon scattering via impurities is present. We find that it is possible to realize chiral transport behavior of phonons by modifying the phonon environment with optomechanics. We study a simple few-mode theory and it can explain experimental data well. Finally, we study a continuum multi-mode theory and calculate the phonon Green's function using a diagrammatic perturbative expansion, showing that a decrease in the phonon diffusion constant is possible with increasing optical pump power.