For more than two decades, the standard quantum limit (SQL) has served as a benchmark for researchers involved in ultra-sensitive force and displacement detection. In this thesis, I discuss a novel displacement detection technique which we have implemented that has allowed us to come within a factor of 4.3 from the limit, closer than any previous effort. Additionally, I show that we were able to use this nearly quantum-limited scheme to observe the thermal motion of a 19.7 MHz in-plane mode of a nanomechanical resonator down to a temperature of 56 mK. At this temperature, the corresponding thermal occupation number of the mode was <nth> ~ 60. This is the lowest thermal occupation number ever demonstrated for a nanomechanical (or larger) device. We believe that the combination of these two results has important and promising implications for the future study of nanoelectromechanical systems (NEMS) at the quantum limit.

The detection scheme that we used was based upon the single-electron transistor (SET). The SET has been demonstrated to be the world's most sensitive electrometer and is considered to be a near-ideal linear amplifier. We used standard lithographic techniques for the on-chip integration of the SET with both a microwave-matching network and nanomechanical resonator. The SET served as a transducer of the resonator's motion: fluctuations in the resonator's position modulated the SET impedance. The microwave-matching circuit allowed us to read-out the modulation of the SET's impedance with ~ 75 MHz bandwidth. The combination of microwave-matching circuit and SET is known as the radio-frequency single-electron transistor (RFSET). Including the nanomechanical resonator, the configuration is called the radio-frequency single-electron transistor displacement detector.

In this thesis, I discuss the basics of quantum-limited measurement and some of the subtleties of observing mechanical quantum phenomena. I then discuss the basics of the RFSET displacement detector, its ultimate limits, its engineering and operation, the first generation results, and finally what improvements could be made to future generation devices.