Hysteresis in smart actuators presents a challenge in control of these actuators. A fundamental idea to cope with hysteresis is inverse compensation. But due to the open loop nature of inverse compensation, its performance is susceptible to model uncertainties and to errors introduced by inverse schemes. In this paper we develop a robust control framework for smart actuators by combining inverse control with the $l_1$ robust control theory. We show that, for both the rate-independent hysteresis model and the rate-dependent one, the inversion error can be bounded in magnitude and the bound is quantifiable in terms of parameter uncertainties and the inversion scheme. Hence we can model the inversion error as an exogenous disturbance and attenuate its impact by robust control techniques. Through the example of controlling a magnetostrictive actuator, we present a systematic controller design method which guarantees robust stability and robust trajectory tracking while taking actuator saturation into account. Simulation and experimental results are provided.