Evaluating the Reliability of Explainable Machine Learning for Spacecraft Actuator Fault Detection

Abstract

LIME and SHAP are widely used algorithms for fault detection classifier explanations, but their specific reliability, validated against known ground truth, in safety-critical aerospace applications has not been rigorously assessed. This study builds and employs a Basilisk anomaly simulation infrastructure to evaluate both methods on spacecraft actuator faults, varying fault timing, magnitude, and subsystem. Random Forest classifiers are used for detection of thruster and reaction wheel degradation from coupled physics-based input features. Analysis shows both LIME and SHAP achieve accurate feature attribution with severe faults, and performance degrades with fault subtlety. In addition, both methods are highly sensitive to classifier input and performance, yielding high performance in well- trained scenarios but worse performance under poor classification ability. SHAP is also seen to more regularly prioritize component-specific features than LIME, which may become desirable as correlation between features becomes stronger. Overall, these findings establish a baseline for applying interpretations of fault detection models to aerospace systems that can enhance downstream operator decision making.