CHAPTER 4. PERFORMANCE STANDARDS SYSTEM
4.1 Identification of Performance Standards. A performance standard is expressed numerically, as a number, rate, ratio, or percentage. It may be calculated by the number of events occurring in a specified operating period expressed in flight cycles, flight-hours, operating hours, or calendar-time. Performance standards are used to identify an acceptable measure of reliability in terms of system or component failures, pilot reported mechanical irregularities, operational delays and cancellations, scheduled maintenance findings, or some other event which serves as the basis for the standard. When used to develop your acceptable performance, control limits or alert values should be based on generally accepted statistical methods, such as standard deviation or the Poisson distribution. Other programs may apply an average or baseline method. Additionally, the program should include procedures for periodic review and adjustment of the standards, as well as detailed procedures for monitoring new fleet types until sufficient operating experience is available for computing a performance standard. Acceptable performance standards should be adjustable based on your operational experience, as well as the effects of fleet age, operational, seasonal, and environmental considerations. Performance standards should be associated with the specific aircraft system as related to the data types being measured.
4.2 Techniques for Determining Deviations From Standards. An operator may elect to use one or a combination of the techniques listed in Table 4-1, Techniques for Determining Deviation From Performance Standards, or any another acceptable method that identifies when the operator-defined acceptable performance level is not achieved. Regardless of the method an air carrier chooses, it should be proactive and should be capable of identifying emerging and current operational problems that the carrier may need to correct. The method chosen should not be so liberal that abnormal deviations from an acceptable standard would not be identified as a problem or as an underlying deficiency. Nor should the method be so conservative that it would result in excessive data “noise” from which too many deviations from the standard are identified. An effective method will result in relevant and meaningful data from which potential, emerging, and real-time deficiencies can be identified, analyzed, and addressed.
4-1
Table 4-1. Techniques for Determining Deviation From Performance Standards Technique Method Action Driver Reliability Program Requirement Alert-Based Identifies deviations from a defined standard based on previous performance. The alert level is set so an alert is triggered by an increase in failure rate or scheduled findings to a degree beyond normal variation.
Statistical characterization of historical failure rates or scheduled maintenance findings is required to determine the appropriate alert threshold for the system under consideration.
Investigation required when the performance falls outside normal variation.
Identification of data type(s);
definition of the method used to calculate the alert level.
Trend Monitoring Graph or table that tracks current performance to identify out-of-limit conditions or trends of deterioration.
Understanding of the measurement units that have a relationship to system failures
(e.g., hours vs.
cycles). Aircraft system data is typically supplemented by data from component removals and confirmed failures.
Investigation required when a metric falls outside performance limits or is predicted to do so in the near term based on the current trend.
Identification of data types;
timeframes selected for monitoring performance (for example, monthly, quarterly, or yearly averages).
Definition of units of measurement and demonstration so they are appropriate to the type and frequency of events being recorded.
4-2 Inputs Required f or Analysis
Technique Method Inputs Required for Analysis Action Driver Reliability Program Requirement Event-Based An event-based program monitors and develops recommendations in response to specific operational events.
The number and range of inputs must be sufficient to allow data analysis that results in meaningful conclusions.
Much of the information that is compiled to assist in the day-to-day operation of the operator’s maintenance program may be effectively used as a basis for this type of continuous mechanical performance analysis.
Investigation of:
the lowest performing items, AND any item with a significant change in performance rank, AND events of significant operational or safety consequences.
Identification of data types;
definition of performance level and rate of change that would drive investigation.
Index-Based Multiple data types combined to produce an index ranking of performance
(e.g., pilot reports, delays/cancellations, or routine task findings).
Knowledge of the parameters and their interactions that can be used to indicate operational reliability drivers.
Investigation of the lowest performing items, AND of any item with a significant change in performance index trend.
Identification of data types;
definition of the index calculation method; definition of performance level and rate of change that would drive investigation;
individual data types may be weighted to reflect operator’s performance goals and philosophy.
4-3
4.2.1 Alert-Based Programs. The purpose of an alert level is to identify significant deviations from a previously acceptable standard of performance. The level should not be set so high that a major increase in the failure rate does not produce an alert, nor so low that the normal distribution of failures results in excessive alerts. The actual alert level will therefore normally depend on the distribution or “scatter” observed in the failure rates of the system under review. You should recognize that alert levels are not minimum acceptable airworthiness levels, but rather are a means of readily identifying increases in failure rates that fall outside the parameters of what is considered normal performance variation. In every case, further investigation is warranted but may or may not result in identification of a problem or deficiency. There are several recognized methods of calculating alert levels, any one of which may be used provided that the method chosen is fully defined in the operator’s program document.
4.2.2 Trend Monitoring Programs. When data is prepared as a running graphical or tabular display of current performance, these data depict trends as well as out-of-limits conditions. Aircraft systems performance data is usually reinforced by reports of component removals or confirmed failures. The choice of units of measurement is not critical, provided that they are constant throughout the operation of the program and are appropriate to the type and frequency of the events or conditions being recorded.
To assess deteriorating performance, the operator should determine timeframes associated with monitoring performance. For example, a program could monitor the performance standards to measure the most current month, 3-month, and 12-month performance.
4.2.3 Event-Based Programs. An event-based program monitors and develops recommendations in response to specific operational events. Event-based programs may be used by any size of organization and applied to any size of fleet. This technique should have sufficient amount of data input in order to have the capability of analyzing the data to arrive at meaningful and actionable conclusions. Much of the information compiled to assist in the day-to-day operation of your maintenance program may be effectively used as a basis for this type of continuous mechanical performance analysis. As a measure of maintenance schedule effectiveness, some data types are better suited for event-based programs than others. Events having safety implications or significant operational impact, regardless of the number of occurrences, should be investigated.
4.2.4 Index-Based Program. This method involves a composite index created and presented using multiple data types. The data should be correlated to a specific aircraft system/subsystem to produce a performance ranking relative to all systems/subsystems being monitored.
4.2.5 Identifying Operational Performance Variations. The following are examples of ways to identify operational performance variations from your performance standards:
4-4
1. Alert-Based:
• Chronic aircraft system alerts.
• Component removal alerts.
• Delay and cancellation alerts.
2. Trend Monitoring:
• Exhaust gas temperature (EGT) exceedances/trends.
• Aircraft Health Monitoring data.
• Fuel, oil, hydraulic fluid consumption.
• Maximum continuous thrust (MCT) margins (Extended Operations (ETOPS) requirement).
3. Event-Based:
• In-flight shutdowns.
• Air turnbacks/diversions.
• Cancellations.
4. Index-Based:
• Ranking of worst-performing aircraft systems/subsystems.
• Routine task findings.
• Minimum equipment list (MEL) management program effectiveness.
4.3 Non-Performance-Standard-Driven Requests. The operator may have other reasons to consider an adjustment to the maintenance schedule which are not related to negative operational performance. For example:
• A review of all or part of the current maintenance schedule to ensure maintenance is not occurring too frequently;
• Aircraft appearance concerns;
• Modification/product improvement response: review of tasks for effectiveness in light of modification/product improvement; or
• Maintenance concerns that originate from other elements of the Continuous Airworthiness Maintenance Program (CAMP), rather than as a deficiency in the maintenance schedule.
4.3.1 Design Approval Holder (DAH) Source Documents. Source document revisions, including Maintenance Review Board Report (MRBR) or Maintenance Planning Document (MPD) revisions are generated to benefit the aircraft operator community and are a function of aggregating in-service operating experience of the aircraft make/model. These revisions address global in-service experience and reflect new design configurations and new rules. While not required, you should perform a 4-5
documented review of revisions to DAH source documents to determine if a change in your maintenance schedule is warranted. A defined review period or set of timeline criteria should be included in your procedures, and should be based on your:
• Aircraft’s performance,
• Operating environment,
• Experience, and
• Program’s goals and philosophy.
Note: Use of DAH source documents when determining your time limitations is not mandatory. However, if you choose to use source documents as substantiating data in your program, the data must be used as described in this AC.
4.3.1.1 Since the DAH will have substantiated the task and interval for the global fleet, you may perform an abbreviated analysis appropriate to the level of risk presented. For example, analysis of a revised economic task may consist of merely reviewing current reliability metrics and performance standards. Depending on risk, revisions to operational and safety tasks may warrant a more comprehensive analysis than a simple review of your current performance metrics. Analysis of operational data directly related to the revised task should be used to determine if acceptable levels of reliability are being realized. This process should be well-defined in your manual, and include documentation requirements for the data analysis performed and substantiated conclusions. It should be risk-based, and may be a process separate from your optimization process. Your analysis and criteria for adopting or rejecting the revision should be consistent, regardless of the type of change being evaluated (added task, deleted task, escalation, de-escalation).
4.3.1.2 You may choose to use DAH source documents as supplemental data when substantiating revisions to your time limitations. However, just as you cannot select which individual data points or pieces of your data sets to use and which to discard, you may not select only the source document data that support task escalations and deletions while ignoring task de-escalations and task additions. If you choose to use DAH source document data, your defined processes must include an evaluation of all tasks and intervals that have been revised, as well as defined criteria, which results in a consistent action regardless of the type of change being evaluated. Therefore, the following analysis guidelines should be followed:
1. Evaluate the revised task and determine risk (e.g., determine the task type, Failure Effect Category (FEC) code, task procedures, and targeted failure mode).
4-6
2. For DAH source document task deletions or task interval increases:
• If the analysis of the operator’s reliability metrics and performance standards related to the revised tasks show acceptable levels of reliability, you may or may not elect to adopt the change.
• If the analysis of your reliability metrics and performance standards related to the revised tasks show unacceptable levels of reliability, you should not adopt the change until you develop recommendations to address this variation. The DAH revision may be reconsidered after acceptable performance levels are achieved.
3. For source document task additions or interval decreases:
• If the analysis of the operator’s reliability metrics and performance standards related to the revised tasks show acceptable levels of reliability, the operator may or may not elect to adopt the change.
• If the analysis of the operator’s reliability metrics and performance standards related to the revised tasks show unacceptable levels of reliability, the operator should adopt the change unless it has developed its own recommendations as outlined in this AC for addressing variations from performance standards.
4. The operator may elect to adopt all, some, or none of the DAH source document revisions based on the risk-based analysis of relevant reliability metrics and performance standards.
5. This risk-based analysis process must be defined in the carrier’s program, which should include documentation requirements specifying source documents used, the risk assigned, and the data used in the analysis to support the conclusion.
6. DAH source document task revisions not adopted upon initial review should be archived and available for review based on future operational reliability concerns.
4-7
CHAPTER 5. ANALYSIS AND RECOMMENDATION
5.1 Root Cause Analysis of Variations From Performance Standard. You should perform and document an analysis in response to the triggers defined in your performance standards system. In addition to the data types you identify, your root cause analysis should also consider other aspects of your operation such as:
1. Associated flight defects and reductions in operational reliability;
2. Timing: defects occurring or discovered at line and main bases; in-flight vs.
ground operations;
3. Deterioration observed during routine maintenance;
4. Post-heavy-maintenance findings;
5. Service bulletins and modification evaluations;
6. Adequacy of maintenance equipment;
7. Technical publications and instructions;
8. Staff training (see Table 2-1, Competency Training);
9. Effects of variation in utilization (high/low);
10. Effects of seasonality; and
11. Fleet commonality.
5.1.1 Techniques and Tools. Examples of analytical techniques and tools that may be used include:
1. Comparisons of operational data types from internal and external sources.
2. Interpretation of data type trends.
3. Evaluation of repetitive defects, including:
• No Fault Found (NFF). NFF occurs when a system is tested after a fault is reported but the fault is not replicated during the test.
• Rogue Units. A rogue unit is a single serialized line replaceable unit (LRU) which has demonstrated a history of identical system faults which may or may not result in an exceedance of an operator’s defined number of repetitive unscheduled removals within an associated short service life.
• Chronic Units. A chronic unit is a single serialized LRU which has demonstrated a history of different system faults resulting in an exceedance of an operator’s defined number of repetitive unscheduled removals within an associated short service life.
• Chronic Systems/Aircraft. A chronic system or aircraft is identified by a specific aircraft serial number which has demonstrated a history of repetitive unscheduled maintenance defects within the same system/subsystem during an operator-defined period of time.
5-1
4. Confidence testing of expected and achieved results.
5. Studies of life-bands and survival characteristics.
6. Investigative testing/sampling programs.
7. Structural review/analysis. To monitor and control structural integrity, competent personnel should review structural service bulletins and industry reports for applicability and urgency. Structural discrepancies should be reviewed with an emphasis on major structural discrepancies.
8. Weibull analysis. Weibull analysis determines the failure distribution profile and the predictability of failure, and is used to determine whether a component or system shows signs of deteriorating performance with age.
This statistical method helps in determining if a component may benefit from a restoration task or regular preventative maintenance.
Note: When appropriately applied, Weibull analysis methods are not subject to yield limitations as described below.
9. Pareto analysis. Pareto analysis is a simple rank ordering of the number or rate of failures or defects for a given data source. This graphical representation may be effectively used for aircraft out-of-service events, delay and cancellation events, minimum equipment list (MEL) applications/extensions, and other metrics as deemed appropriate. Pareto charts are used to identify the top subjects, which are often targeted for performance improvements.
10. Failure Mode and Effects Analysis (FMEA).
11. Maintenance Steering Group (MSG) analysis methods.
12. Other root cause analysis tools.
5.2 Analysis of Task Effectiveness. You should routinely perform an analysis to determine the applicability and effectiveness of the tasks contained within the maintenance schedule. This is in addition to the analysis of the task and related operational data performed during optimization. The evolution of your maintenance schedule is not necessarily to simply increase individual task intervals, but to substantiate the accomplishment of an effective task at the appropriate interval, without compromising safety or negatively impacting operational performance beyond that which is acceptable.
5.2.1 Data Selection Criteria. An operator should include in its reliability program a defensible method for determining the relevant type and amount of data that adequately represents its fleet with respect to the maintenance task under consideration. The following describes a defensible method for data selection:
• Consistent,
• Measurable,
• Unbiased,
• Factual, 5-2
• Accurate, and
• Repeatable.
5.2.2 Data Sampling. In compliance with your Continuing Analysis and Surveillance System (CASS) obligation under part 121, § 121.373 to determine effectiveness of your maintenance schedule, 100 percent of your performance standard data is collected and analyzed. Because of this CASS requirement, the concept of sampling may be applied as an effective and efficient tool when analyzing other data types/sets in support of your program’s requirements. Data sampling allows you to analyze a defined portion of data from a relevant data set, rather than analyzing the entire amount of data that may be available. As with any effective data analysis process, the FAA expects you to collect and analyze additional data types and/or different data sets as the particular circumstances dictate.
5.2.2.1 When executing these CASS requirements in the context of data sampling, you will encounter variables where differences between the following aspects of your program become significant. Examples include:
1. The type of task being evaluated.
2. The finding or type of finding considered unacceptable for a given task (see paragraph 1.6).
3. Related vs. unrelated data.
4. Significant vs. nonsignificant findings.
5. Data variables:
• Discrete data (i.e., can only take on certain values, can be counted);
• Continuous data (i.e., can take on infinite number of values, must be measured); and
• Data quality/error rate.
6. Qualitative vs. quantitative analysis.
7. Other differences in analysis methods and tools:
• Those designed to predict failures or future performance, and
• Those designed to monitor real-time performance.
5.2.2.2 Due to these variations and the wide range of analysis tools you may choose to employ, it is not possible in this AC to provide prescriptive details or criteria for you to apply when determining what data types/sets to collect and analyze, your data sample set, or which analysis methods you should use.
5.2.2.3 Regarding data sampling specifically, it is likewise not possible to provide a one-size-fits-all sampling method or formula that is effective for every data type and analysis method available. You should not attempt to define or 5-3
rely on any one method or formula that establishes a sample size to be used in every case. You must respond to the needs of the analysis being performed, which includes performing any additional analyses as discussed above.
Therefore, you should design and execute your processes and methods, including your sampling methods, to be agile enough to respond to the specific data and analysis needs of each situation. The keys in executing an effective analysis process are:
1. Recognizing the significance and the effect of variations in your operations and data;
2. Utilizing the appropriate data types/sets, amount of data, and analysis methods and tools for the given situation;
3. Following through with the actions necessary to ensure program goals and objectives are met; and
4. Using competent personnel to perform these functions to ensure a consistent outcome.
Note: Personnel competencies are discussed in paragraph 2.1.4 and in AC 120-79, Developing and Implementing an Air Carrier Continuing Analysis and Surveillance System, Chapter 6, Personnel Who Perform CASS Functions; and in AC 120-16, Air Carrier Maintenance Programs, Chapter 10, Personnel Training.
5.2.2.4 Regardless of the method used to determine your sample data requirements, your sampling process must result in confidence that the sample data is applicable and representative of the population. Your data sampling process should consider data provided by outliers within your population, while preventing the possibility that a decision is inappropriately influenced by data from aircraft that are at the extremes of the data set. The following is provided for reference only, and is not intended to suggest any single method described herein as the recommended method or formula to determine sample size in every situation.
1. The process for selecting a data set that reflects the fleet composition should consider the following criteria:
• Operational history,
• Configuration differences,
• Time elapsed since major conversions or refurbishments,
• Utilization differences,
• High time/cycle and low time/cycle aircraft,
• Seasonality and environmental conditions, 5-4
• Sustained storage periods, and
• Task yield.
Note: Aircraft that don’t meet minimum task yield standards should not be included in the random sample. Findings from scheduled maintenance tasks actually performed at a lower yield may be an indicator that the published interval is too high. Further analysis will be necessary, including analysis of related in-service data.
2. Include one or more of the following means when defining a data selection method. Data should be selected from the available data pool generated using the criteria above, such as:
• A defined number of aircraft (typically used with smaller fleets).
• A percentage of fleet size (e.g., a minimum percentage of randomly selected aircraft), plus outlier aircraft from the criteria applicable under paragraph 5.2.2.4, item 1 (e.g., high utilization and low utilization, high flight-hours and low flight-hours, high cycles and low cycles, etc.).
• A statistical formula (primarily used for quantitative data analysis).
• Data collected over a defined period of time. Here, the sample is all data from the data set that is collected during the specified period, rather than all data that may be available or that has been collected (particularly useful when identifying performance trends).
3. Depending on the task being analyzed, your sampling method should allow flexibility to account for benefits in analyzing the task performed repetitively on specified aircraft vs. analyzing tasks performed once on a greater number of aircraft. As stated above, collecting additional data and/or different data sets may be necessary.
4. You should ensure fleet representation is maintained in the data throughout your analysis process (see Figure 5-1, Data Selection Example).
5-5