Magnetoencephalography is a noninvasive tool that measures the magnetic activity of the brain. Its high temporal resolution makes it useful for studying auditory and speech models. However, it suffers from poor signal to noise ratio caused by corruption from non-stationary external noise, biological artifacts, and non-auditory neural noise in the brain. We remove external noise from neural channels using a frequency domain block least mean square adaptive filter with the help of three reference sensors that measure environmental noise alone. Significance tests that build on F-statistics present ample evidence of the benefit of such de-noising by increasing the number of significant channels and reducing the variability of false positives. Finally, the least significant and noisiest channel is filtered and used to de-noise neural signals while minimizing interference with the auditory signal. We propose a method for finding such reference channels and assess performance through receiver operating characteristics and statistical significance.