The Visible Infrared Imaging Radiometer Suite (VIIRS) onboard Suomi National Polar Orbiting Partnership (SNPP) uses a solar diffuser (SD) as its radiometric calibrator for the reflective solar band calibration. The SD is made of Spectralon™ (one type of fluoropolymer) and was chosen because of its controlled reflectance in the Visible/Near-Infrared/Shortwave-Infrared region and its near-Lambertian reflectance property. On-orbit changes in VIIRS SD reflectance as monitored by the Solar Diffuser Stability Monitor showed faster degradation of SD reflectance for 0.4 to 0.6 µm channels than the longer wavelength channels. Analysis of VIIRS SD reflectance data show that the spectral dependent degradation of SD reflectance in short wavelength can be explained with a SD Surface Roughness (length scale << wavelength) based Rayleigh Scattering (SRRS) model due to exposure to solar UV radiation and energetic particles. The characteristic length parameter of the SD surface roughness is derived from the long term reflectance data of the VIIRS SD and it changes at approximately the tens of nanometers level over the operational period of VIIRS. This estimated roughness length scale is consistent with the experimental result from radiation exposure of a fluoropolymer sample and validates the applicability of the Rayleigh scattering-based model. The model is also applicable to explaining the spectral dependent degradation of the SDs on other satellites. This novel approach allows us to better understand the physical processes of the SD degradation, and is complementary to previous mathematics based models.