4-6
• Experience, and
• Program’s goals and philosophy.
Note: Use of DAH source documents when determining your time limitations is not mandatory. However, if you choose to use source documents as substantiating data in your program, the data must be used as described in this AC.
4.3.1.1 Since the DAH will have substantiated the task and interval for the global fleet, you may perform an abbreviated analysis appropriate to the level of risk presented. For example, analysis of a revised economic task may consist of merely reviewing current reliability metrics and performance standards.
Depending on risk, revisions to operational and safety tasks may warrant a more comprehensive analysis than a simple review of your current performance metrics. Analysis of operational data directly related to the revised task should be used to determine if acceptable levels of reliability are being realized. This process should be well-defined in your manual, and include documentation requirements for the data analysis performed and substantiated conclusions. It should be risk-based, and may be a process separate from your optimization process. Your analysis and criteria for adopting or rejecting the revision should be consistent, regardless of the type of change being evaluated (added task, deleted task, escalation, de-escalation).
4.3.1.2 You may choose to use DAH source documents as supplemental data when substantiating revisions to your time limitations. However, just as you cannot select which individual data points or pieces of your data sets to use and which to discard, you may not select only the source document data that support task escalations and deletions while ignoring task de-escalations and task additions. If you choose to use DAH source document data, your defined processes must include:
1. An evaluation of all tasks and intervals that have been revised, as well as defined criteria, which results in a consistent action regardless of the type of change being evaluated.
2. A risk-based analysis process in your program, including documentation requirements specifying source documents used, the risk assigned, and the data used in the analysis to support the conclusion.
4.3.1.3 Therefore, the following analysis guidelines should be followed:
1. Evaluate the revised task and determine risk (e.g., determine the task type, FEC code, task procedures, and targeted failure mode).
2. For DAH source document task deletions or task interval increases:
• If the analysis of the operator’s reliability metrics and performance standards related to the revised tasks show acceptable levels of reliability, you may or may not elect to adopt the change.
4-7
• If the analysis of your reliability metrics and performance standards related to the revised tasks show unacceptable levels of reliability, you should not adopt the change until you develop recommendations to address this variation. The DAH revision may be reconsidered after acceptable performance levels are achieved.
3. For source document task additions or interval decreases:
• If the analysis of the operator’s reliability metrics and performance standards related to the revised tasks show acceptable levels of reliability, the operator may or may not elect to adopt the change.
• If the analysis of the operator’s reliability metrics and performance standards related to the revised tasks show unacceptable levels of reliability, the operator should adopt the change unless it has developed its own recommendations as outlined in this AC for addressing variations from performance standards.
4. The operator may elect to adopt all, some, or none of the DAH source document revisions based on the risk-based analysis of relevant reliability metrics and performance standards.
5. DAH source document task revisions not adopted upon initial review should be archived and available for review based on future operational reliability concerns.
5-1 CHAPTER 5. ANALYSIS AND RECOMMENDATION
5.1 Root Cause Analysis of Variations From Performance Standard. You should perform and document an analysis in response to the triggers defined in your performance standards system. In addition to the data types you identify, your root cause analysis should also consider other aspects of your operation such as:
1. Associated flight defects and reductions in operational reliability;
2. Timing: defects occurring or discovered at line and main bases; in-flight vs. ground operations;
3. Deterioration observed during routine maintenance;
4. Post-heavy-maintenance findings;
5. Service bulletins and modification evaluations;
6. Adequacy of maintenance equipment;
7. Technical publications and instructions;
8. Staff training (see Table 2-1, Recommended Competency Training);
9. Effects of variation in utilization (high/low);
10. Effects of seasonality; and
11. Fleet commonality.
5.1.1 Techniques and Tools. Examples of analytical techniques and tools that may be used include:
1. Comparisons of operational data types from internal and external sources.
2. Interpretation of data type trends.
3. Evaluation of repetitive defects, including:
• No Fault Found (NFF). NFF occurs when a system is tested after a fault is reported but the fault is not replicated during the test.
• Rogue Units. A rogue unit is a single serialized LRU that has demonstrated a history of identical system faults that may or may not result in an exceedance of an operator’s defined number of repetitive unscheduled removals within an associated short service life.
• Chronic Units. A chronic unit is a single serialized LRU that has demonstrated a history of different system faults resulting in an exceedance of an operator’s defined number of repetitive unscheduled removals within an associated short service life.
• Chronic Systems/Aircraft. A chronic system or aircraft is identified by a specific aircraft serial number that has demonstrated a history of repetitive unscheduled maintenance defects within the same system/subsystem during an operator-defined period of time.
5-2
4. Confidence testing of expected and achieved results.
5. Studies of life-bands and survival characteristics.
6. Investigative testing/sampling programs.
7. Structural review/analysis. To monitor and control structural integrity, competent personnel should review structural service bulletins and industry reports for applicability and urgency. Structural discrepancies should be reviewed with an emphasis on major structural discrepancies.
8. Weibull analysis. Weibull analysis determines the failure distribution profile and the predictability of failure, and is used to determine whether a component or system shows signs of deteriorating performance with age. This statistical method helps in determining if a component may benefit from a restoration task or regular preventative maintenance.
Note: When appropriately applied, Weibull analysis methods are not subject to yield limitations as described below.
9. Pareto analysis. Pareto analysis is a simple rank ordering of the number or rate of failures or defects for a given data source. This graphical representation may be effectively used for aircraft out-of-service events, delay and cancellation events, MEL applications/extensions, and other metrics as deemed appropriate. Pareto charts are used to identify the top subjects, which are often targeted for performance improvements.
10. Failure Mode and Effects Analysis (FMEA).
11. MSG analysis methods.
12. Other root cause analysis tools.
5.2 Analysis of Task Effectiveness. You should routinely perform an analysis to determine the applicability and effectiveness of the tasks contained within the maintenance schedule. This is in addition to the analysis of the task and related operational data performed during optimization. The evolution of your maintenance schedule is not necessarily to simply increase individual task intervals, but to substantiate the accomplishment of an effective task at the appropriate interval, without compromising safety or negatively impacting operational performance beyond that which is acceptable.
5.2.1 Data Selection Criteria. An operator should include in its reliability program a defensible method for determining the relevant type and amount of data that adequately represents its fleet with respect to the maintenance task under consideration. The following describes a defensible method for data selection:
• Consistent,
• Measurable,
• Unbiased,
• Factual,
5-3
• Accurate, and
• Repeatable.
5.2.2 Data Sampling. In compliance with your CASS obligation under 14 CFR §§ 91.1431,
121.373, and 135.431 to determine effectiveness of your maintenance schedule, 100 percent of your performance standard data is collected and analyzed. Because of this CASS requirement, the concept of sampling may be applied as an effective and efficient tool when analyzing other data types or sets in support of your program’s requirements.
Data sampling allows you to analyze a defined portion of data from a relevant data set, rather than analyzing the entire amount of data that may be available. As with any effective data analysis process, the FAA expects you to collect and analyze additional data types or different data sets as the particular circumstances dictate.
5.2.2.1 When executing these CASS requirements in the context of data sampling, you will encounter variables where differences between the following aspects of your program become significant. Examples include:
1. The type of task being evaluated.
2. The finding or type of finding considered unacceptable for a given task (see paragraph 1.6).
3. Related vs. unrelated data.
4. Significant vs. nonsignificant findings.
5. Data variables:
• Discrete data (i.e., can only take on certain values, can be counted);
• Continuous data (i.e., can take on infinite number of values, must be measured); and
• Data quality/error rate.
6. Qualitative vs. quantitative analysis.
7. Other differences in analysis methods and tools:
• Those designed to predict failures or future performance, and
• Those designed to monitor real-time performance.
5.2.2.2 Due to these variations and the wide range of analysis tools you may choose to employ, it is not possible in this AC to provide prescriptive details or criteria for you to apply when determining what data types/sets to collect and analyze, your data sample set, or which analysis methods you should use.
5.2.2.3 Regarding data sampling specifically, it is likewise not possible to provide a one-size-fits-all sampling method or formula that is effective for every data type and analysis method available. You should not attempt to define or rely on any one method or formula that establishes a sample size to be used in
5-4 every case. You must respond to the needs of the analysis being performed, which includes performing any additional analyses as discussed above.
Therefore, you should design and execute your processes and methods, including your sampling methods, to be agile enough to respond to the specific data and analysis needs of each situation. The keys in executing an effective analysis process are:
1. Recognizing the significance and the effect of variations in your operations and data;
2. Utilizing the appropriate data types/sets, amount of data, and analysis methods and tools for the given situation;
3. Following through with the actions necessary to ensure program goals and objectives are met; and
4. Using competent personnel to perform these functions to ensure a consistent outcome.
Note: Personnel competencies are discussed in paragraph 2.1.4; in AC 120-79, Developing and Implementing an Air Carrier Continuing Analysis and Surveillance System, Chapter 6, Personnel Who Perform CASS Functions; and in AC 120-16, Air Carrier Maintenance Programs, Chapter 10, Personnel Training.
5.2.2.4 Regardless of the method used to determine your sample data requirements, your sampling process must result in confidence that the sample data is applicable and representative of the population. Your data sampling process should consider data provided by outliers within your population, while preventing the possibility that a decision is inappropriately influenced by data from aircraft that are at the extremes of the data set. The following is provided for reference only, and is not intended to suggest any single method described herein as the recommended method or formula to determine sample size in every situation.
1. The process for selecting a data set that reflects the fleet composition should consider the following criteria:
• Operational history,
• Configuration differences,
• Time elapsed since major conversions or refurbishments,
• Utilization differences,
• High time/cycle and low time/cycle aircraft,
• Seasonality and environmental conditions,
• Sustained storage periods, and
• Task yield.
5-5 Note: Aircraft that don’t meet minimum task yield standards should not be included in the random sample. Findings from scheduled maintenance tasks actually performed at a lower yield may be an indicator that the published interval is too high. Further analysis will be necessary, including analysis of related in-service data.
2. Include one or more of the following means when defining a data selection method. Data should be selected from the available data pool generated using the criteria above, such as:
• A defined number of aircraft (typically used with smaller fleets).
• A percentage of fleet size (e.g., a minimum percentage of randomly selected aircraft), plus outlier aircraft from the criteria applicable under paragraph 5.2.2.4, item 1 (e.g., high utilization and low utilization, high flight-hours and low flight-hours, high cycles and low cycles, etc.).
• A statistical formula (primarily used for quantitative data analysis).
• Data collected over a defined period of time. Here, the sample is all data from the data set that is collected during the specified period, rather than all data that may be available or that has been collected (particularly useful when identifying performance trends).
3. Depending on the task being analyzed, your sampling method should allow flexibility to account for benefits in analyzing the task performed repetitively on specified aircraft vs. analyzing tasks performed once on a greater number of aircraft. As stated above, collecting additional data and/or different data sets may be necessary.
4. You should ensure fleet representation is maintained in the data throughout your analysis process (see Figure 5-1, Data Selection Example).
5-6 Figure 5-1. Data Selection Example
5.2.3 Task Yield.
5.2.3.1 Any scheduled maintenance task analysis that includes either defects generated or a lack of defects generated, should consider the respective interval yield. This is because the current task interval is established with the intent of detecting degradation or potential failure at that interval. If the task is performed significantly earlier than the specified interval, the data may not be valid to support an evaluation of the specified interval.
Note: Operators should include in their analysis process defects discovered at relatively low yield intervals since this may be an indication that the current task interval is not effective.
5.2.3.2 To arrive at a yield solution, the defined interval and the interval at which the task was actually accomplished must be calculated using the same parameter (flight-hours, flight cycles, or calendar). In some cases, you may need to perform a conversion of values to arrive at the same usage parameter.
5-7 The yield calculation divides the interval at which the task was actually accomplished by the defined interval.
• A minimum acceptable value for average yield of all task accomplishments contributing data to the task analysis is 90 percent.
• No single task accomplishment within the data set used for task analysis should have a yield less than 80 percent.
Note: Analysis of consecutive task accomplishments may be useful when evaluating task and interval effectiveness. Failures discovered at a relatively low yield should be considered as such in the overall analysis. However, tasks performed at a relatively low yield with no findings should not be used to determine or analyze failure rates at the published interval, or as zero findings data when substantiating escalations from that interval. The second or subsequent task, by itself resulting in a higher overall yield, may be useful for these purposes provided in-service failures are identified and incorporated into the data.
5.2.3.3 Example:
• Actual task interval: 7,100 flight-hours.
• Task defined interval: 7,500 flight-hours.
• Yield calculation: 7,100 flight-hours/7,500 flight-hours = 94.7 percent task yield.
5.2.4 Data Preparation and Related Considerations. Typically, operational data are collected and consistently coded so correlation to a maintenance schedule task is evident to analysts or engineers performing the analysis (see paragraph 3.3). To ensure comprehensiveness and relevance of data when performing analysis, the operator should ensure all data findings are directly related to the task being analyzed.
5.2.4.1 Related Data. Related data is a maintenance defect discovered during scheduled or unscheduled maintenance to which the corrective action is directly associated to the task being analyzed or its consequence of functional failure.
5.2.4.2 Unrelated Data. Unrelated data is a maintenance defect discovered during scheduled or unscheduled maintenance for which the corrective action is not directly associated to the task being analyzed or its consequence of functional failure. Depending on risk, it may be necessary to investigate unrelated data findings separately from the evaluation of the given task and interval.
Note: Unrelated findings that would otherwise be considered significant as defined below should be thoroughly investigated to determine overall program effectiveness.
5-8
5.2.4.3 Significant Finding. A significant finding is a maintenance record considered directly related to the task being analyzed that indicates a functional failure or significant degradation/wear has occurred. These are primary analysis concerns. Examples of significant findings include:
1. Systems that:
• Affect airworthiness or safety on the ground or in flight.
• Are undetectable or unlikely to be detected during operations.
• Have significant operational impact.
• Have significant economic impact.
2. Structures that:
• Affect airworthiness or safety on the ground or in flight.
• Have reportable defects under the SDR system.
• Have Level 2 corrosion.
• Have Level 1 corrosion found other than at the scheduled maintenance interval.
5.2.4.4 Non-Significant Finding. A non-significant finding is a maintenance defect considered directly related to the task being analyzed that does not indicate a functional failure or significant degradation or wear and which does not fall into an example category for significant finding (e.g., Level 1 corrosion found at the scheduled maintenance interval). Depending on risk, these may or may not be primary analysis concerns.
5.2.5 Summary of Data. Each interval adjustment should include a summary of the data that was used for the analysis. Relevant, accurate, and a sufficient amount of data constitute the foundation for an operator’s standards determining their scheduled maintenance time limitations. This data summary should demonstrate that the processes and procedures of the operator’s reliability program regarding data selection were followed.
5.2.6 Evaluation of Data Sources. Table 5-1, Data Sources by Task Type, contains recommended data sources that you may use for analysis to determine time limitations for maintenance schedule tasks. The data is divided into primary and secondary sources.
1. Primary data elements should be used as the main data source for analysis by the operator to substantiate an effective task interval.
2. Secondary data sources should be used to further support or validate analysis conclusions, or when further investigation is warranted. This data should not be used as the sole source for determining time limitations of a task. Operators may identify additional data sources for analysis.
5-9 Table 5-1. Data Sources by Task Type Task Type Primary Data Secondary Data Systems tasks1 (excluding Failure Effect Categories (FEC) 8 & 9) In-service operational data Routine task findings Propulsion tasks1 (excluding FEC 8 & 9) In-service operational data Routine task findings Structures tasks2 Routine task findings3 In-service operational data Zonal tasks Routine task findings3 In-service operational data FEC 8 & 9 tasks Routine task findings/sampling at proposed interval3 FEC 8 ONLY: MSG-derived functional failure data/expand data set/previous accomplishment of task Enhanced zonal analysis procedure (EZAP)/electrical wiring interconnection system (EWIS) tasks Routine task findings3 In-service operational data Lube tasks Volume of consumables, rates/sampling at proposed interval, and/or degree of wear noted on component teardown reports Operational reliability data Filter tasks Analysis of remaining life at current interval/sampling at proposed interval Operational reliability data4 1 The primary data source for systems and powerplant tasks (excluding FEC 8 & 9 tasks) is the operator’s operational reliability data. The effectiveness of these tasks is measured by the ability of the maintenance schedule task to detect and correct defects in the system before the defect impacts the operational environment.
2 Include any damage (fatigue cracks or corrosion), whether or not it was discovered as a result of a scheduled maintenance task.
3 It is important to note that because failures associated with these tasks are normally found during the accomplishment of the routine task, this data may not be captured as part of an operator’s normal reliability data elements. Therefore, some means of capturing those failures for analysis is necessary.
4 Only for those tasks where failures would be evident to the flightcrew or maintenance personnel and would be recorded in the operational data (such as logbook writeups).
5-10
5.2.7 Analysis Process. Once the operator has identified and obtained the data, the operator must analyze the data in order to substantiate interval adjustment or task revision.
To perform the correct level of analysis, the operator should develop standardized decision logic based on the type of task and data being analyzed, and should be performed by appropriately trained and competent personnel. For examples of this decision logic, see the appendices within this AC.
5.2.8 Result of Analysis. If the analysis shows the impact on operations for a given system is low, the task may be considered effective regardless of the number of scheduled maintenance findings. Conversely, a high degree of operational impact could indicate an ineffective task, even if the number of scheduled maintenance task findings is low.
Other combinations are also possible. Table 5-2, Review of Systems/Powerplant Tasks, represents examples of situations that may be identified during the analysis. This table is relative; conclusions should be based on your unique operations. It is not definitive or exhaustive, but is intended to only provide guidance to operators when evaluating tasks and findings. The existence of scheduled maintenance findings by itself is not an indication of an ineffective program.
Table 5-2. Review of Systems/Powerplant Tasks1 Low Number of Unscheduled Maintenance Defects High Number of Unscheduled Maintenance Defects Low Impact on Operations High Impact on Operations Low Impact on Operations High Impact on Operations Low Number of Scheduled Maintenance Defects Effective but not optimized Ineffective Note 2 Ineffective High Number of Scheduled Maintenance Defects Effective and optimized Ineffective Effective but not optimized Ineffective 1 FEC 8 and 9 tasks are designed to detect failures that are not evident during normal operations. Therefore, these failures do not typically result in an impact to in-service operations. (Refer to Maintenance Steering Group - 3rd Task Force (MSG-3).)
2 Additional analysis is required to determine task effectiveness.
5.2.9 Data Standards. You should develop a standard for how to collect, validate, display and archive task data, along with the results of any analysis performed, including conclusions and recommendations to be considered during your internal approval process. You should consider adding a special emphasis task section to your Summary of Data that contains information related to certain tasks that would warrant special analysis or consideration, such as: