U.S. Army Evaluation Center

This paper illustrates the development of Bayesian assurance test plans for system reliability assuming that binomial data will be collected on the system and that previous information is available from component testing. The posterior consumer's and producer's risks are used as the criteria for developing the test plan. Using the previous component information reduces the number of tests needed to achieve the same levels of risk. The proposed methodology is illustrated with two examples.

This paper offers several contributions to the area of discrete reliability growth projection. We present a new, logically derived model for estimating the reliability growth of complex, one-shot systems (i.e., the reliability following implementation of corrective actions to known failure modes). Multiple statistical estimation procedures are utilized to approximate this exact expression. A new estimation method is derived to approximate the vector of failure probabilities associated with a complex, one-shot system. A mathematically-convenient functional form for the s -expected initial reliability of a one-shot system is derived. Monte-Carlo simulation results are presented to highlight model accuracy with respect to resulting estimates of reliability growth. This model is useful to program managers, and reliability practitioners who wish to assess one-shot system reliability growth.

In the recent past, the Defense Science Board (DSB) Task Force report on Developmental Test and Evaluation revealed a significant increase in the number of DoD weapon system programs evaluated as not being operationally suitable. The primary reason is the lack of material readiness due to poor system Reliability, Availability, and Maintainability (R AM). The report shows that nearly half of U.S. Army systems from 1997-2006 failed to demonstrate their established reliability requirements during Operational Testing. As a result of the DSB findings and associated DoD Reliability Improvement Working Group (RIWG) report, a series of department policies have been established that place increased emphasis not only on reliability growth planning and tracking, but also on reliability best practices, and reliability language for defense acquisition contracts. As such, it is now department policy “for programs to be formulated to execute a viable RAM strategy that includes a reliability growth program as an integral part of design and development.” The most recent policy further stipulates, “For new or restructured programs DOT&E will not approve TESs and TEMPs lacking a reliability growth curve or software failure profile.” This paper presents a detailed Reliability Growth (RG) planning approach that may be utilized for developing RG programs for discrete-use systems, thereby facilitating the implementation of the aforementioned DoD reliability policies. More specifically, this approach, hereafter referred to as PM2-Discrete, may be utilized for developing RG programs and associated planning curves that: (1) portray planned reliability achievement as a function of program resources; (2) serve as a baseline against which demonstrated reliability values may be compared throughout a test program (for tracking purposes); and (3) illustrate and quantify the feasibility of a test program in achieving interim and final reliability goals. In particular, PM2-Discrete possesses a series of management metrics that may be used to assess the effectiveness of proposed RG programs. These metrics serve as concomitant measures of programmatic risk and system maturity that may also be assessed during testing for progress reporting purposes. A methodology overview and application of PM2-Discrete is given, as well as an abbreviated overview of relevant literature within the area of RG planning. Note that derivations of the model equations (not presented herein), are available and may be referenced in.

Until recently, military standard 785 [1] was the primary reliability program standard used by the US Department of Defense (DoD). MIL-STD-785 consisted of a menu of tasks that program managers could select from in order to construct a reliability program for a specific acquisition. Many systems subjected to operational field testing by the DoD failed to demonstrate their reliability requirements. MIL-STD-785 was cancelled in 1998 after two industry standards were developed as alternatives: IEEE 1332 [2] and SAE JA1000 [3]. The requirements set forth in both IEEE 1332 and SAE JA1000 are very concise and consist primarily of statements of three objectives. Many systems still fail to demonstrate their reliability requirements during operational field testing as depicted in Figure 1 [4].

Bayesian estimation procedures are derived herein that may be utilized to evaluate reliability growth of discrete systems, such as guns, rockets, missile systems, torpedoes, etc. One of the advantages of these Bayesian procedures is that they directly quantify the epistemic uncertainties in model parameters (i.e., the shape parameters of the beta distribution), as well as six reliability growth metrics of basic interest to program management. These metrics include: (1) the initial system reliability; (2) the projected reliability following failure mode mitigation; (3) reliability growth potential (i.e., the theoretical upper-limit on reliability achieved by finding and fixing all failure modes via a specified level of fix effectiveness); (4) the expected number of failure modes observed during testing; (5) the probability of observing a new failure mode; and (6) the fraction of the initial system probability of failure associated with failure modes for which program management is aware. These metrics give reliability practitioners the means to estimate the reliability of discrete systems undergoing development, address model goodness-of-fit concerns, quantify programmatic risk, and assess system maturity. Analytical results are presented to obtain Bayes' estimates of the beta shape parameters under a delayed corrective action strategy (i.e., when corrective actions are installed on system prototypes at the end of the current test phase). A Monte Carlo simulation approach is given for constructing uncertainty distributions on each of the aforementioned reliability growth management metrics. Bayesian probability limits for inference on interval estimation are obtained in the usual manner (i.e., via desired percentiles of the uncertainty distributions). These uncertainty distributions are found to be approximated very well by beta and/or Gaussian random variables. These methods are illustrated by simple numerical examples. In particular Bayes' estimates the beta shape parameters are obtained from a small dataset, and compared against the true parameter values. Bayesian epistemic uncertainty distributions are also constructed for the reliability growth management metrics via the proposed Monte Carlo approach. This methodology is useful to program managers and reliability practitioners that wish to quantitatively assess the reliability growth program of one-shot systems developed under a delayed corrective action strategy.

This paper explores the computation of Circular Error Probable from a bivariate regression model as well as upper confidence limits computed at covariate values of interest. Upper confidence limits were computed based on a bootstrap methodology and a bootstrap calibration of the delta method. The bootstrap calibration provided an improvement in the nominal coverage for the delta method at smaller sample sizes. An example is provided using munition data from an Army test event to show how these approaches can be applied to data sets from test events with small sample sizes.

Since all weapons and ammunitions operate as one system, the reliability of one affects the reliability of the entire system. Deficiencies in new ammunition design and development can severely reduce a weapon's reliability by sho rtening weapon part life or decreasing the Mean Rounds Between Stoppages (MRBS). The reliability of a system is directly related to the life cycle cost of a system. A highly reliable system requires less maintenance and less spare parts. Sustainment cost would be reduced with less need for maintenance action and less spare parts. Since historically, sustainment cost constitutes more than 50% of the life cycle cost of any system, changes in the sustainment cost affect the life cycle cost. The amount of increase in the overall life cycle cost can help determine whether the cost is beneficial to pursue further failure analysis and corrective action implementation. Aside from cost, other factors affect the decision to perform further fixes on a system to increase reliability. A highly reliable system also increases soldier moral and soldier safety, increases the mission success rate and possibly shortens the mission completion time. A new ammunition that causes shortened barrel and bolt life, decreased reliability and increased sustainment cost of its host weapon system may be acceptable depending on other benefits this new ammunition may provide. A thorough reliability and life cycle cost analysis can help determine the advantages and disadvantages of a new ammunition from a statistical perspective.

This paper describes the processes for implementing new Department of the Army (DA) reliability policy directives issued by the Assistant Secretary of the Army for Acquisition, Logistics, and Technology, or ASA(ALT). It highlights key points of the implementation guide prepared by the U.S. Army Evaluation Center (AEC) and the U.S. Army Test and Evaluation Command (ATEC). The implementation plan herein utilizes GEIA-STD-0009 for developing contract language promoting reliability best practices. The Army Materiel Systems Analysis Activity's (AMSAA's) new Reliability Program Scorecard is used for performing early-on engineering evaluations of contractor reliability engineering activities. Note that both of these tools are outcomes of the Department of Defense (DoD) Reliability Improvement Working Group (RIWG). The implementation plan also addresses the Defense Science Board (DSB) recommendation and the new DoD acquisition regulations to establish Reliability Growth (RG) programs for developmental systems. Growth programs and associated planning curves are constructed via AMSAA's Planning Model based on Projection Methodology, hereafter referred to as PM2. Finally, AMSAA's COnsumption, HOlding, Repair and Transportation (COHORT) cost model is utilized to identify life cycle cost impacts for systems that breach their established reliability thresholds.

A system-of-systems is defined as “a set or arrangement of systems that results from independent systems integrated into a larger system that delivers unique capabilities.” Given practical resource constraints, it is rare that the full-field configuration of the system-of-systems can be exercised during an operational reliability demonstration test. However, as we consider various potential operational test configurations for a given system-of-systems during the reliability test program planning process, it is critical to understand how testing a configuration that is smaller than the full-field configuration decreases the adequacy of the test by reducing the accuracy of the system-of-systems’ reliability estimate that is based on the test results. Thus, it is useful to assess the adequacy of potential system-of-systems’ operational test configurations before adopting one. We present a novel simulation-based method that can be employed to assess the adequacy of a given test configuration for any type of system-of-systems. To illustrate how this simulation-based method can be used to aid in the identification of the best alternative from among a group of potential operational test configuration alternatives, we include an example application using a notional air defense system-of-systems. Trade-offs with respect to cost, schedule, and accuracy are addressed within the context of this application.

The purpose of this paper is to discuss assessing system reliability based on results from both developmental and operational testing used together. This paper will discuss the planning, analysis and reporting of reliability for a tactical military vehicle utilizing integrated Developmental Test/Operational Test (DT/OT). A series of DT and OT tests were performed in sequence with the intent of using data from all phases in assessing vehicle reliability. By introducing operationally realistic conditions during DT with Soldier involvement, the reliability degradation usually observed when progressing from DT to OT did not occur. Military reliability testing is often limited by the number of available test assets at the time of testing and the program schedule. This results in competing test priorities such as requiring assets for survivability testing versus reliability testing versus performance testing. This program leveraged the required transport to get the test assets across the country to increase reliability mileage across more test assets to assess vehicle variability in early life. Further limiting the test asset availability was the availability of the armor kits for the vehicles; some vehicles were tested with armor kits and some had no armor kits. The results from DT and OT were compared on a failure mode basis to determine whether the data could be combined; ultimately providing senior decision makers with an estimate of vehicle reliability.

An engineered product is said to experience reliability growth when (1) a failure mode is observed and investigated until the root cause is identified and (2) an understanding of the root cause failure mechanism is then developed and used to formulate and implement a corrective action that reduces or eliminates the probability or frequency of the failure mode's recurrence within the product population. This article addresses how one identifies failure modes during developmental product testing, develops an understanding of them, and formulates corrective actions to mitigate them. A brief introduction to statistical reliability‐growth modeling is provided, along with lessons learned from its application to complex repairable assemblies and products. Statistical reliability‐growth modeling was developed to pool data across evolving product configurations as failure modes are observed and mitigated during a sequence of developmental tests. In most applications, one does not have sufficient data to generate final configuration estimates of satisfactory uncertainty; the pooling of data from current and prior configurations, using a method that accounts for the impact of corrective actions, can provide a clearer understanding of reliability growth and support improved product development decision‐making (e.g., when to grant production approval). The most crucial lesson learned from several decades of applications is that successful reliability‐growth programs require that all but the most elusive failure modes be identified and mitigated prior to product‐level reliability‐growth testing; this key engineering activity enables the product to meet the high level of initial reliability required for success.

The purpose of this paper is to discuss the reliability projection for a complex military system. For the purposes of this paper we will consider this a mobile system with standalone subsystems integrated onto the platform. The reliability of the overall system is determined by these subsystems as well as the reliability of the mobile platform itself. At the time the reliability projection was made, the system had been previously tested in developmental and operational environments. Following the first developmental test (DT), the system entered into operational testing (OT). After completion of the initial DT and OT testing, the system underwent corrective actions to address issues uncovered in testing. After correction, the system underwent additional developmental testing. After the last phase of developmental testing, analysis was done to project the system's reliability prior to entering the second OT. While taking into consideration the subsystems and their impact on the overall affect on the platforms reliability, the analysis concluded the reliability of a specific component greatly affected the projected reliability of the platform in an operational environment.

Bayesian estimation procedures are derived herein that may be utilized to evaluate reliability growth of discrete-use systems, such as guns, rockets, missile systems, torpedoes, etc. One advantage of these Bayesian procedures is that they directly quantify the epistemic uncertainties in model parameters, i.e., the shape parameters of the beta distribution, as well as several reliability growth metrics of basic interest to program management. These metrics include: (1) the initial system reliability; (2) the projected reliability following failure mode mitigation; (3) reliability growth potential, i.e., the theoretical upper-limit on reliability achieved by finding and fixing all failure modes via a specified level of effectiveness; (4) the expected number of failure modes observed during testing; (5) the probability of observing a new failure mode and; (6) the fractional contribution of correctable failure modes to the initial probability of failure. These metrics and associated model equations give reliability practitioners the means to: (1) assess reliability achievement of discrete-use systems undergoing development; (2) address model goodness-of-fit concerns and; (3) quantify programmatic risk, and system maturity.