In practice, various ad hoc approaches for designing reliability test programs have been observed. Many of these approaches rely on previously established rules of thumb for which the underlying rationale is indefensible. As a consequence, those who use such approaches are unlikely to maintain a firm resource commitment for the conduct of reliability test program activities. Furthermore, it is difficult to ascertain the impact that budgetary cuts will have on the adequacy of the reliability test program with any degree of accuracy.

The contributions of this research are as follows. This dissertation presents a novel 7-step planning process to aid practitioners in designing adequate reliability test programs. This planning process serves as a tool to systematically identify, quantify, and mitigate evaluation risks subject to resource constraints. By performing the 7 steps associated with this planning process, practitioners will be able to logically justify reliability test program requirements and more effectively articulate the significance of evaluation risks associated with a particular reliability test program design. Additionally, it is a straightforward process to assess the impact of a reduction in reliability test program resources.

This planning process includes a step for assessing the level of risk associated with key aspects of the reliability test program. One such consideration that is of paramount importance is the adequacy of the test configuration of the system. Hence, we present a simulation-based approach for assessing the adequacy of the test configuration of a complex system-of-systems. For the purpose of demonstration, an application of this approach to air defense systems is included; however, the approach is valid for any type of system.

As well, this dissertation presents an evaluation risk assessment process for reliability test programs--adapted from the traditional failure mode and effects analysis (FMEA) process. This process can be applied to any reliability test program, irrespective of the manner in which the plan was formulated. Just as a FMEA facilitates the identification of potential weaknesses in a system architecture, this evaluation risk assessment process is designed to surface reliability test program weaknesses and gauge the potential impact of each weakness to the system reliability evaluation.