CHAPTER 2. RELIABILITY PROGRAM EVALUATION AND REVIEW
2.1 Program Creation and Revisions.
2.1.1 Main Parts of an Operator’s Reliability Program. The elements of an operator’s reliability program are typically:
• A data collection system,
• A performance standards system,
• Analysis and recommendation,
• Internal approval and implementation, and
• A reporting and display format.
Note: Figure 2-1, Reliability Program Management and Administration, is a closed-loop, process-oriented flow diagram of the reliability process described in this AC.
2.1.2 Documentation Within the Operator’s Manual. You should ensure your standards for determining time limitations contain the following:
1. A general description of the reliability program, including definitions of (or reference to) significant terms used in the reliability program.
Note: You may not use your reliability program as a means to change the meaning or intent of regulatory-based definitions, restrictions, limitations, or reporting requirements. When defining terms, you should be as specific as possible and when applicable, provide qualifying criteria to avoid ambiguity and confusion when executing program requirements.
2. The application of the reliability program by aircraft fleet type and model.
3. Procedures and standards for data collection and ensuring applicability/data quality.
4. The organizational responsibilities, training, and experience requirements.
5. Duties and responsibilities for personnel who perform program requirements.
This includes personnel such as those who monitor performance standards or alert/event levels, analyze operational data, and board members or personnel who perform decision-making functions.
6. The procedures for monitoring and revising the performance standards.
7. Data analysis methods.
8. The corrective action recommendation process, including approval and implementation.
9. Reports used or generated, and the frequency.
2-1
10. Standards and procedures for adjusting time limitations.
11. Description or references to forms unique to the reliability program.
12. Management and administration procedures, including reliability program revision process.
13. Requirements or references to self-audits and other monitoring procedures for the performance and effectiveness of the reliability program.
14. Description or organizational chart of reliability program organizational authority and delegation. The administration of a reliability program generally requires a board or other organizational body within an operator’s maintenance organization. This board should be comprised of experienced and competent personnel with decision-making authority to approve changes to the operator’s maintenance schedule.
2.1.3 Identifying Participants. You should identify in your manual those participants who have authority and responsibility for your reliability program, including:
1. The Director of Maintenance (DOM) or equivalent who has overall authority and responsibility for the entire program.
2. The person who has direct authority and responsibility for the reliability program process.
3. The individual positions in the operator’s organization and their role within the reliability program, including responsibilities and authority.
2-2
Figure 2-1. Reliability Program Management and Administration Reliability Review and Regulatory Continuous Chapter Program Creation Acceptance, Evaluation 2 and Revision OpSpec Data Collection System Performance Standards System Analysis and Identify Alert Level Goals Recommendations Data Trend Monitoring Sources Variation Identify Event Level Goals from Variation from Identify Performance Composite Performance Performance Performance Data Standards Index Standard Standard Chapter Types Scheduled Maintenance Analysis Findings Index Chapter Other Performance Data Standards System Quality Scheduled Non-Performance Standard Chapter Task Driven Request Effectiveness Analysis Correction/Action - Internal Approval and Implementation Internal Recommendation Implementation Approved Change Approval Process Chapter Chapter Continue to Monitor Chapter Reporting and Display Format 2-3
2.1.4 Reliability Personnel Training/Technical Competency.
2.1.4.1 You should determine your own specialized technical standards and training documentation requirements for persons whose role is to collect, analyze, and compile reliability data or reports. The desired competency and capability standards should be defined and be based on the level of complexity of the operational and scheduled maintenance data you collect (see Table 2-1, Competency Training).
2.1.4.2 You may adjust the content, frequency, and method of training provided to individuals depending on an evaluation of the individual’s roles and responsibilities, previous training, hands-on maintenance experience, and/or demonstration of aptitude, and the results being achieved.
2.1.4.3 The FAA expects reliability program participants with approval authority to fully understand their roles and responsibilities relative to the reliability program.
Table 2-1. Competency Training Subject Training Reliability Personnel Knowledge Standards 14 CFR part 121/135 operation specifications (OpSpecs); part 91K management specifications (MSpecs) Familiarization and awareness Systems analysis training General training in quality standards and statistical analysis methods Reliability data training Experience or training in conducting and reporting analysis results for operational data and other scheduled maintenance findings Maintenance Steering Group (MSG):
current or applicable Understanding of applicable MSG methodology used in developing the maintenance schedule Risk assessment training The ability to assess risk as it relates to a situation, failure, or hazard Root cause analysis training, including human factors Method of problem solving that identifies the root causes of faults, failures, deficiencies, or hazards.
Failure Mode and Effects Analysis (FMEA) training Method of analysis that identifies the failure modes and failure effects related to specific faults or failures Technical training Aircraft-specific (make, model, and series (M/M/S)) 2-4
2.2 Regulatory Requirements. There is no regulatory requirement that an operator must have or maintain a reliability program. However, if an operator elects to use a reliability program, certain regulations, OpSpecs, and requirements must be taken into consideration.
2.2.1 OpSpecs/MSpecs.
• D072, Aircraft Maintenance—Continuous Airworthiness Maintenance Program (CAMP) Authorization.
• D074, Reliability Program Authorization: Entire Aircraft.
• D075, Reliability Program Authorization: Airframe, Powerplant, Systems or Selected Items.
• D079, Reliability Program Contractual Arrangement Authorization.
• D088, Maintenance Time Limitations Authorization.
2.2.2 Reliability Program Submission. Your initial reliability program submission and subsequent revisions do not require FAA approval. However, prior to use, you must submit your initial reliability program and each subsequent revision to the FAA office with oversight responsibility. You may use your program after the principal maintenance inspector (PMI) has notified you that the program is acceptable and has issued the appropriate OpSpec/MSpec authorizing its use.
2.3 Continuous Evaluation. As part of an effective Continuing Analysis and Surveillance System (CASS), you continually monitor the performance and effectiveness of your reliability program, and make revisions as necessary. You should also conduct periodic, documented reviews to determine that your performance standards are realistic and are effective in identifying deficiencies.
2-5
CHAPTER 3. DATA COLLECTION SYSTEM
3.1 Identification of Data Sources. An operator should identify data sources from all of these four general categories of an operator’s fleet:
• Systems,
• Components/line replaceable units (LRU),
• Structures, and
• Engines/APUs.
3.2 Identification of Data Types. The data collection system should include the identification of data types. Table 3-1, Data Types lists typical types of operational data;
however, all of these data types do not need to be included in the reliability program, nor does this list prohibit the use of other types of information.
Table 3-1. Data Types Data Type In-Service Data Routine Task Findings Aircraft maintenance log (mechanical irregularities) X X Engine in-flight shut down (IFSD) X Operational interruptions (delays, cancellations, diversions, etc.) X Unscheduled component removals X Component maintenance findings X X Sampling inspections X X Special inspections X X Maintenance deferred with minimum equipment list (MEL) X X Chronic systems X X Unscheduled maintenance X Integrated Vehicle Health Monitoring
a) Aircraft health monitoring
b) Engine health monitoring X X Materials usage X X 3-1
Data Type In-Service Data Routine Task Findings Extended Operations (ETOPS) Event Reports, Reduced Vertical Separation Minimum (RVSM), Category II/III (CAT II/III) X Scheduled maintenance findings X X Accidents or incidents X X Unconfirmed components or part removal X Unscheduled engine removal X X Service Difficulty Reports (SDR) X X Mechanical Interruption Summary Report (MISR) X Flight logs X
3.3 Data Quality.
3.3.1 General. Data sources and associated data types form the foundation of any reliability program. Data can be considered of sufficient quality when it is accurate, free from substantive recording errors, and comprehensive enough in both scope and detail to facilitate its intended function in operations, analysis, and decision making.
3.3.2 Ensuring Validity and Accuracy of Data. You should develop a process to validate the accuracy of the data used in support of your reliability program. A typical data accuracy/data quality process includes:
• Adopting comprehensive data and maintenance recording standards and instructions;
• Forms and instructions designed to achieve compliance with the data standards;
• Performing and documenting data audits to detect deficiencies and correct data irregularities;
• Establishing a documented method for providing feedback to organizational elements demonstrating deviations from the data standards; and
• A common coding convention or system used to correlate mechanical irregularities in aircraft systems with defects found during routine maintenance tasks, shop teardown reports and other relevant data.
Note: While an electronic means of collecting, storing, retrieving, and analyzing data is not required by rule, limitations in these areas may restrict or preclude you from exercising the requisite responsibilities necessary to implement and maintain an effective reliability program.
3-2
CHAPTER 4. PERFORMANCE STANDARDS SYSTEM
4.1 Identification of Performance Standards. A performance standard is expressed numerically, as a number, rate, ratio, or percentage. It may be calculated by the number of events occurring in a specified operating period expressed in flight cycles, flight-hours, operating hours, or calendar-time. Performance standards are used to identify an acceptable measure of reliability in terms of system or component failures, pilot reported mechanical irregularities, operational delays and cancellations, scheduled maintenance findings, or some other event which serves as the basis for the standard. When used to develop your acceptable performance, control limits or alert values should be based on generally accepted statistical methods, such as standard deviation or the Poisson distribution. Other programs may apply an average or baseline method. Additionally, the program should include procedures for periodic review and adjustment of the standards, as well as detailed procedures for monitoring new fleet types until sufficient operating experience is available for computing a performance standard. Acceptable performance standards should be adjustable based on your operational experience, as well as the effects of fleet age, operational, seasonal, and environmental considerations. Performance standards should be associated with the specific aircraft system as related to the data types being measured.
4.2 Techniques for Determining Deviations From Standards. An operator may elect to use one or a combination of the techniques listed in Table 4-1, Techniques for Determining Deviation From Performance Standards, or any another acceptable method that identifies when the operator-defined acceptable performance level is not achieved. Regardless of the method an air carrier chooses, it should be proactive and should be capable of identifying emerging and current operational problems that the carrier may need to correct. The method chosen should not be so liberal that abnormal deviations from an acceptable standard would not be identified as a problem or as an underlying deficiency. Nor should the method be so conservative that it would result in excessive data “noise” from which too many deviations from the standard are identified. An effective method will result in relevant and meaningful data from which potential, emerging, and real-time deficiencies can be identified, analyzed, and addressed.
4-1
Table 4-1. Techniques for Determining Deviation From Performance Standards Technique Method Action Driver Reliability Program Requirement Alert-Based Identifies deviations from a defined standard based on previous performance. The alert level is set so an alert is triggered by an increase in failure rate or scheduled findings to a degree beyond normal variation.
Statistical characterization of historical failure rates or scheduled maintenance findings is required to determine the appropriate alert threshold for the system under consideration.
Investigation required when the performance falls outside normal variation.
Identification of data type(s);
definition of the method used to calculate the alert level.
Trend Monitoring Graph or table that tracks current performance to identify out-of-limit conditions or trends of deterioration.
Understanding of the measurement units that have a relationship to system failures
(e.g., hours vs.
cycles). Aircraft system data is typically supplemented by data from component removals and confirmed failures.
Investigation required when a metric falls outside performance limits or is predicted to do so in the near term based on the current trend.
Identification of data types;
timeframes selected for monitoring performance (for example, monthly, quarterly, or yearly averages).
Definition of units of measurement and demonstration so they are appropriate to the type and frequency of events being recorded.
4-2 Inputs Required f or Analysis
Technique Method Inputs Required for Analysis Action Driver Reliability Program Requirement Event-Based An event-based program monitors and develops recommendations in response to specific operational events.
The number and range of inputs must be sufficient to allow data analysis that results in meaningful conclusions.
Much of the information that is compiled to assist in the day-to-day operation of the operator’s maintenance program may be effectively used as a basis for this type of continuous mechanical performance analysis.
Investigation of:
the lowest performing items, AND any item with a significant change in performance rank, AND events of significant operational or safety consequences.
Identification of data types;
definition of performance level and rate of change that would drive investigation.
Index-Based Multiple data types combined to produce an index ranking of performance
(e.g., pilot reports, delays/cancellations, or routine task findings).
Knowledge of the parameters and their interactions that can be used to indicate operational reliability drivers.
Investigation of the lowest performing items, AND of any item with a significant change in performance index trend.
Identification of data types;
definition of the index calculation method; definition of performance level and rate of change that would drive investigation;
individual data types may be weighted to reflect operator’s performance goals and philosophy.
4-3
4.2.1 Alert-Based Programs. The purpose of an alert level is to identify significant deviations from a previously acceptable standard of performance. The level should not be set so high that a major increase in the failure rate does not produce an alert, nor so low that the normal distribution of failures results in excessive alerts. The actual alert level will therefore normally depend on the distribution or “scatter” observed in the failure rates of the system under review. You should recognize that alert levels are not minimum acceptable airworthiness levels, but rather are a means of readily identifying increases in failure rates that fall outside the parameters of what is considered normal performance variation. In every case, further investigation is warranted but may or may not result in identification of a problem or deficiency. There are several recognized methods of calculating alert levels, any one of which may be used provided that the method chosen is fully defined in the operator’s program document.
4.2.2 Trend Monitoring Programs. When data is prepared as a running graphical or tabular display of current performance, these data depict trends as well as out-of-limits conditions. Aircraft systems performance data is usually reinforced by reports of component removals or confirmed failures. The choice of units of measurement is not critical, provided that they are constant throughout the operation of the program and are appropriate to the type and frequency of the events or conditions being recorded.
To assess deteriorating performance, the operator should determine timeframes associated with monitoring performance. For example, a program could monitor the performance standards to measure the most current month, 3-month, and 12-month performance.
4.2.3 Event-Based Programs. An event-based program monitors and develops recommendations in response to specific operational events. Event-based programs may be used by any size of organization and applied to any size of fleet. This technique should have sufficient amount of data input in order to have the capability of analyzing the data to arrive at meaningful and actionable conclusions. Much of the information compiled to assist in the day-to-day operation of your maintenance program may be effectively used as a basis for this type of continuous mechanical performance analysis. As a measure of maintenance schedule effectiveness, some data types are better suited for event-based programs than others. Events having safety implications or significant operational impact, regardless of the number of occurrences, should be investigated.
4.2.4 Index-Based Program. This method involves a composite index created and presented using multiple data types. The data should be correlated to a specific aircraft system/subsystem to produce a performance ranking relative to all systems/subsystems being monitored.
4.2.5 Identifying Operational Performance Variations. The following are examples of ways to identify operational performance variations from your performance standards:
4-4
1. Alert-Based:
• Chronic aircraft system alerts.
• Component removal alerts.
• Delay and cancellation alerts.
2. Trend Monitoring:
• Exhaust gas temperature (EGT) exceedances/trends.
• Aircraft Health Monitoring data.
• Fuel, oil, hydraulic fluid consumption.
• Maximum continuous thrust (MCT) margins (Extended Operations (ETOPS) requirement).
3. Event-Based:
• In-flight shutdowns.
• Air turnbacks/diversions.
• Cancellations.
4. Index-Based:
• Ranking of worst-performing aircraft systems/subsystems.
• Routine task findings.
• Minimum equipment list (MEL) management program effectiveness.
4.3 Non-Performance-Standard-Driven Requests. The operator may have other reasons to consider an adjustment to the maintenance schedule which are not related to negative operational performance. For example:
• A review of all or part of the current maintenance schedule to ensure maintenance is not occurring too frequently;
• Aircraft appearance concerns;
• Modification/product improvement response: review of tasks for effectiveness in light of modification/product improvement; or
• Maintenance concerns that originate from other elements of the Continuous Airworthiness Maintenance Program (CAMP), rather than as a deficiency in the maintenance schedule.
4.3.1 Design Approval Holder (DAH) Source Documents. Source document revisions, including Maintenance Review Board Report (MRBR) or Maintenance Planning Document (MPD) revisions are generated to benefit the aircraft operator community and are a function of aggregating in-service operating experience of the aircraft make/model. These revisions address global in-service experience and reflect new design configurations and new rules. While not required, you should perform a 4-5