AC 39-8A
1.7 Background.
1.7.1 In 1991, the AIA tasked a working group called “the CAAM committee” (here after referred to “the CAAM”) with developing effective methods for identifying, prioritizing, and resolving safety-related problems occurring on commercial airplane engines. The CAAM analyzed 10 years of engine, propeller, and APU events and developed an event characterization system. The CAAM then grouped the events by the observed outcome (non-containment, fire, etc.) and assigned an HL.
1.7.2 For each observed outcome, the CAAM calculated the historical conditional probabilities of the most serious events, HLs 3 and 4, using a methodology to estimate the risks of safety-related problems occurring on airplane engines, propellers, and APUs. In 1994, the CAAM first published these results in the AIA PC-342 and AIA PC-342-1.
1.7.3 Since then, the CAAM has updated the database of the safety-significant propulsion system and APU historical malfunctions for 1992 through 2000 and 2001 through 2012 and published all three sets of results in reports in 1999, 2005, and 2017 (“CAAM 1,” “CAAM 2,” and “CAAM 3” reports).2 1F The FAA has used the information in these reports to help identify and prioritize responses to potential engine, propeller, and APU unsafe conditions.
1.8 CAAM Publications.
CAAM will be continually engaged in the collection, evaluation, and publication of data and guidance material. The future CAAM publications will be posted on the website Engines and Propellers - Data and Reports | Federal Aviation Administration. Due to the evolution and revision of definitions and other factors, various datasets should only be used with the guidance established for that dataset. The following further elaborates on the relationship of current CAAM work to, and applicability of, prior published data.
Consistent with historic CAAM practices, the current CAAM effort is not intended to supersede or invalidate prior work, but rather to build upon, supplement, and advance what has come before. No single CAAM report or activity can be taken in isolation. The published CAAM reports provide a valuable history and will remain an integral part of current and contemplated committee work for the foreseeable future. As new datasets are developed, published, and updated the companion background and guidance material will, as applicable, indicate what prior work it may replace or make obsolete (if that is the case). While some data/information may be supplanted, familiarity with topic evolution through prior work will still have relevance and value. The FAA encourages users of any CAAM data to become familiar with the entire body of work as the current effort is inextricably tied and related to previous work. As the application of the datasets in CAAM reports and this AC and how to best interpret the data for the purpose of numerical assessments should be based on the definitions, nomenclatures and guidance
2 These reports are available at https://www.faa.gov/aircraft/air_cert/design_approvals/engine_prop/engine_sp_topics 1-7
AC 39-8A specified in respective datasets. The material in these datasets is general in nature and should only be utilized in conjunction with a thorough assessment of the individual situation to which it is applied.
1-8
AC 39-8A CHAPTER 2. IDENTIFYING UNSAFE CONDITIONS
2.1 Identifying Unsafe Conditions.
When a potential unsafe condition is identified by the stakeholder, the stakeholder should conduct a quantitative or qualitative assessment for the actual risk of the event of interest. See chapter 3 for further guidance on performing the assessment.
2.2 Root Cause Analysis and/or Contributing Factors.
The preliminary information received by the stakeholder regarding the potential unsafe condition may be insufficient to determine the appropriate corrective action. Therefore, it is often necessary to seek additional information to determine the root causes and contributing factors to the potential unsafe condition. Unsafe conditions may be caused either in combination or separately by improper design, manufacture, maintenance, or operation. The contribution of these individual elements should be evaluated by the stakeholder in order to ascertain the probability of future occurrences as well as the effectiveness of candidate corrective actions. Unsafe conditions that are not mitigated by conditional factors may require expeditious action unless the root causes, failure distribution, and risk factor can be established and determined to be non-urgent by the stakeholder.
Note: Root cause problem assessments may identify concerns in other products of the same or similar type design or usage. In these cases, consideration of corrective action beyond the initially identified population may be necessary.
2.3 Hazard Ratio of Prior Events.
The stakeholder(s) should review the prior examples of similar occurrences of the potential unsafe condition and their associated airplane-level effects to help with the determinations described in all aspects of the CAAM processes. The CAAM reports are one source that provides information on previous occurrences of a variety of propulsion system and APU events. This information is especially helpful in estimating the hazard ratio. See appendices C and D for additional guidance.
2.4 Probability of Future Events.
The stakeholder(s) should compare the probability of HL 3 and higher events occurring in the future to the long-term acceptable risk guidelines outlined in table 1 of this AC, as well as the applicable design standards to determine whether an unsafe condition exists. An unsafe condition exists if the probability of occurrence does not meet both of these guidelines and design standards.
2.5 Immediate Action is Warranted.
When the stakeholder identifies an unsafe condition, the FAA should decide if immediate action is warranted following the guidelines detailed in paragraph 4.3. If 2-1
AC 39-8A immediate action is warranted, such action should reflect the severity of the condition and the likelihood of additional events occurring before the implementation of a longterm solution.
2.5.1 It is possible that the likelihood of a recurrence of an unsafe condition cannot be adequately estimated immediately following a potentially severe event. If it is possible to take immediate and practical action which the FAA has a basis to believe will address that safety risk, while the agency’s initial assessment to fully determine the cause and likelihood of recurrence is being made, the FAA should consider that action.
For example, an accident caused by an inflight thrust reverser deployment resulted in the pilot losing control of the airplane. Neither the cause of failures, which led to the thrust reverser deploying inflight, nor the reasons why the inflight deployment resulted in the pilot losing control of the airplane, could be readily determined with certainty.
However, it was possible to take reasonable immediate action by “locking-out” the thrust reversers. The FAA would allow the thrust reversers to be “unlocked” when it could assure system integrity by requiring periodic checks of the entire thrust reverser system, including its fault indication features. The final corrective action was to incorporate system modifications to ensure that subsequent inflight deployments are not anticipated to occur within the fleet life of the airplane type.
2.5.2 If no practical or readily implemented corrective action is possible to ensure continued operational safety, or if such action is not known to be sufficient in and of itself, then every effort should be made to objectively evaluate the appropriate level of response to the identified unsafe condition. When factual data is sparse, this review should be based on sound engineering judgment. The intent in either situation is for consistent and objective responses to unsafe conditions.
2.6 Responses to Unsafe Conditions.
Responses to identified unsafe conditions can vary from an immediate mitigating reaction to an extensively considered final response. The control programs for most unsafe conditions may include initial, interim, and final actions. The CAAM processes assist with the evaluation of those actions and help to verify that the actions are appropriate and timely to mitigate the unsafe condition.
2.7 Risk to Persons Outside the Airplane.
Throughout the CAAM risk assessment processes, the stakeholders should consider the threat to persons both inside and outside the airplane when determining how to mitigate the unsafe condition. However, certain causes of serious injury to persons outside the airplane, such as those caused by human negligence or recklessness, cannot be wholly prevented by practicable changes to the type design. Examples of this include personnel that have walked into turning propellers or been ingested into engines due to a disregard for cautions, warning markings, or manual instructions. Many other risks to persons outside the airplane, such as the shedding of small components from the airplane or engine were statistically insignificant from the perspective of CAAM risk assessment, in comparison with the airplane-level risk, due to the low probability of persons being 2-2
AC 39-8A injured. Thus, if the threat to persons outside the airplane is insignificant compared to the threat to persons inside the airplane, or to the applicable risk guidelines, then the stakeholder does not need to consider the marginal additional risk to persons outside the airplane when developing corrective action.
2-3
AC 39-8A CHAPTER 3. RISK ASSESSMENT
3.1 Risk Assessment.
It may be necessary for the FAA to work closely with the design approval holder and, if appropriate, the operators to complete the steps outlined within the CAAM risk assessment process. This cooperative effort may take the form of the FAA engineer overseeing the work of the manufacturer or designer, the FAA engineer and manufacturer working in concert, or the FAA engineer performing the analysis with input from the manufacturer.
3.2 Assumptions.
Any risk analysis, whether qualitative or quantitative, is only as accurate as the assumptions, data, and analytical techniques used. Therefore, all assumptions should be justified to ensure that the conclusions of the analysis are valid. The justification of the assumptions should be an integral part of the analysis. Assumptions can be validated by using experience with identical or similar systems or components, with due allowance made for differences of design, duty cycle, and environment. When it is not possible to adequately justify the critical elements of the analysis, conservatism should be built into the initial assumptions. However, it is important to maintain consistency with the observed physics of the event and failure mode when adding conservatism.
Alternatively, the uncertainty in the data and assumptions should be evaluated to the degree necessary to demonstrate that the conclusions are relatively insensitive to that uncertainty. If a quantitative method is used, it is essential that the analysis calibrates with the experience to date. A quantitative risk analysis cannot be expected to credibly predict into the future if it does not calibrate to actual experience.
3.3 Probability of Event Prediction.
There are various methods used to classify the severity and predict the probability of event occurrence. The intent, in any case, is to perform a realistic structured assessment to determine whether an unsafe condition exists. This ensures that an unsafe condition with a greater risk of occurrence receives a higher level of attention and resources for its resolution than does one that represents a lower risk. The actual process used may vary depending on the nature of the potentially unsafe condition, the level of experience with similar unsafe conditions, the level of risk assessment, and prioritization of information needed to support effective decision-making. However, the stakeholder should perform a process equivalent to that described in this AC as fully as possible. The steps outlined in paragraphs 3.3.1 – 3.3.12 may be performed using either quantitative or qualitative methods, or both.
3.3.1 Estimate the Number of Airplanes Exposed.
Determine the number of airplanes for which the unsafe condition may exist, or be expected to develop, if no corrective action is taken. For example, estimate the number of airplanes with engine parts that fall within a certain serial number range or airplanes with installed engines with a certain total cycles or total hours. A failure condition may 3-1
AC 39-8A apply to the fleet of airplanes installed with the affected engines, or it may be limited to a particular, identifiable subpopulation. Alternatively, the full fleet may consist of many subpopulations that are all at risk of the failure condition, but at different rates or probabilities. Subpopulations may include different engine models, operators, types of operations, location of operations, manufacturing heats, production lots, or serial number ranges. To divide the full fleet and separately assess the risk of subpopulations, the individual components, engines, or airplanes should be able to be identified as belonging to the at-risk subpopulation(s). For example, if the model of each engine is known, the heat of material or serial number for each part is known, or the type of operation for flights can be identified.
3.3.2 Estimate the Uncorrected Risk Factor.
3.3.2.1 Use analytical techniques to estimate the uncorrected risk factor. This step estimates how many events will occur within the exposed population.
While the event of interest is usually the occurrence of the identified unsafe condition, events of lesser or greater severity may also be analyzed.
Risk factors for HLs 3, 4, and 5 are also calculated to allow for comparison to the risk guidelines outlined in table 1 of this AC. Risk factors for higher-level events are obtained by multiplying the event risk factor by the applicable hazard ratio.
3.3.2.2 Failure of components is most often a function of component cycles, airplane flights, or hours of operation. However, it may be a function of calendar time or other parameters. Failure rate data expressed as failures per flight hour, cycle, or flight may also be dependent on the frequency with which the actual stresses that cause failure to occur within the exposure period. For example, a component failure mode may be predominantly a function of the number of times electrical power is applied to it. If so, the average power applications per flight hour should be the same between the source of the failure rate data and the subject application of that data, or an appropriate correction should be applied.
Note: Exposure means the possibility of occurrence, not the certainty of it.
If multiple airplane types are exposed to the same unsafe condition, then the estimate should include all affected airplanes rather than assessing the risk to each airplane type separately.
3.3.2.3 For uncorrected risk factor, a 20-year fleet life may be assumed, or another reasonable estimate of the actual fleet life may be used. The fleet life should be converted to flights or flight hours, as appropriate, per airplane.
Additionally, the risk factor should be converted to risk per flight or flight hour, if applicable, to facilitate comparing risks on a common exposure basis. Converting the risk per flight or flight hour is done by dividing the risk factor by the total number of flights or flight hours within the assumed exposure period. Whatever exposure basis is used, flights or flight hours, it should be used consistently to allow the various risks being managed simultaneously to be readily compared to each other and the risk 3-2
AC 39-8A guidelines. The uncorrected risk factor(s) should be compared to the risk guidelines outlined in table 1 of this AC to help establish the relative threat posed by the unsafe condition.
3.4 Mitigating and Corrective Actions.
3.4.1 Identify Options for Mitigating Action.
Some types of actions that have proven to be both practical and beneficial for immediate responses are inspections, placards, revisions, supplements to the airplane flight manual, staggering engines to obtain mixed life engines on a given airplane (for infant-mortality problems), and pre-flight checks. These same actions are appropriate for follow-on actions, along with repair or replacement of the suspect components.
3.4.2 Estimate the Effects of Candidate Actions.
The stakeholder should evaluate candidate actions considered after identification of an unsafe condition with the appropriate manufacturer, designer, or operators. This evaluation considers the action’s capacity to reduce the future risk to acceptable levels.
From a technical perspective, several candidate actions may be available. The selected action(s) should consider such issues as confidence in the effectiveness of the corrective action, availability of the resources necessary to support the corrective action, the ability of operators to incorporate the corrective action expediently and adequately, and potential implementation risk for operators to rework a particular design. For acceptable short-term risk, see the risk guidelines outlined in table 1 of this AC.
3.4.3 Estimate Potential Risk Reduction.
Once the candidate actions have been identified, the risk factor for the proposed control program should be estimated using the process described above. This process should be performed for all candidate actions under consideration, thereby allowing the effects of different programs to be compared. The objective is to keep the risks to the affected fleet below the applicable guidelines until final action can be taken to bring the product back to the level of safety intended by the product's original basis of certification. If none of the immediate corrective action programs can achieve the needed risk reductions, more aggressive action, including grounding, should be considered.
Note: Aircraft operation with an unsafe condition may be acceptable, provided short term guidelines in table 1 are met during the corrective action.
3.4.4 Estimate Impact on Resources.
Resources are generally considered to be time, material (parts and inspection equipment), and labor. However, there may be additional considerations such as shop capacity, parts distribution, and operational disruptions. The extent of these resources should be estimated to quantify the impact of the AD or other corrective action, allow for timely provisioning, and aid in the determination of desirable tradeoffs between resources and risk. Depending on the analysis that has been performed, the number of replacement parts, shop visits, inspections, etc., may be available as output parameters.
However, the results from the steps used to establish the risk factor could likewise be 3-3
AC 39-8A used to estimate an impact on resources. Data from the manufacturer(s), operators, or both will often be used in this process.
3.4.5 Rank Practical Candidate Actions.
Given that several candidate actions provide an equivalent reduction in risk, they can be ranked regarding the impact on resources. Small increases in risk may be acceptable when a candidate action with lower risk is of much greater difficulty to effectively implement or is much more burdensome than a slightly riskier option. Each candidate action should be evaluated against its effectiveness (meaning its relative reduction of risk), availability of resources (such as shop visit capacity, material availability, personnel, etc.), and how quickly the candidate action can be implemented, how easy it is to implement, and its relative cost. Candidate actions include such items as:
• Manufacturing, maintenance, or operational procedural changes;
• On-wing or in-shop inspections;
• Limitations on time-limited dispatch (TLD); and
• Part repairs, replacement, or modifications.
The number of cycles or hours between initial and repetitive actions should also be evaluated. The ideal action would be inexpensive, easy to perform, possible to begin immediately, and completely effective. The real situation often requires trading off these characteristics. For example, developing an accurate inspection tool and method that can be used for engines on-wing may mean inspection does not begin immediately.
3.4.6 Develop and Implement Appropriate Responses.
The objective of all continued operational safety decisions is to maintain an acceptable level of safety by reducing the risks posed by future events. Selection of actions, including taking no specific action, should be based on the specific circumstances and an assessment of the risk of future occurrences of the unsafe condition. Prohibition of engine operation based on an observed unsafe condition, pending determination of the root causes and appropriate corrective action, should be reserved for situations where immediate action is necessary, and no other option would be effective in addressing the unsafe condition. See paragraph 4.3 for risk guidelines for immediate action.
Note: If the FAA decides not to implement a particular candidate corrective action, the decision and its justification must be filed by the involved stakeholders in the project folder for future reference, following FAA policies related to office retention length.
Closure documentation should include justification and reasons for the determination of non-implementation of the corrective action.
3.4.7 Verify Results of Corrective Actions.
Initial corrective actions, whether immediate reactions or responses initially considered
(e.g., the issuance of an immediately adopted AD), may not represent the final action needed to address the unsafe condition. To that end, service experience and any other data gathered during the action implementation should be carefully reviewed to validate the analytical process and the estimated risks.
3-4
AC 39-8A
3.4.8 Monitor Implementation and Impacts of the Corrective Actions Taken.
When feasible, the rate of incorporation of the corrective action(s) should be tracked by all involved stakeholders to verify that the action is being implemented within the timeline prescribed by the AD. If the action includes inspection, the stakeholder(s) should analyze the inspection results to help quantify incipient failures and aid assessment of the extent of the problem. Service experience should be tracked to ensure that the rate of occurrence of the referenced event is reduced. However, a rate of occurrence may not be applicable in cases of rare events or small exposures. The stakeholder(s) should evaluate the service experience and inspection results against expectations developed because of any quantitative or qualitative analysis performed as part of the action. The inclusion of reporting requirements within the body of the AD will ensure that the operators report inspection findings. This inclusion should be considered for those high risk (as defined in paragraph 4.4.1) unsafe conditions where the inspection findings are necessary to evaluate the adequacy of actions. FAA principal aviation safety inspectors or the manufacturer’s field service representatives can also provide direct insights into the impact of mandated corrective actions independent of any reporting requirements.
3.4.9 Verify Corrective Actions Were Effective.
Any experience that deviates significantly from expectations or assumptions is grounds to revise the risk assessment. Continue monitoring field experience and inspection results to ensure that any interim action continues to validate the assumptions and predictions of the risk assessment. Final action carries with it an assumption that the causal factors, which lead to an unsafe condition, have been effectively eliminated or mitigated. The stakeholder should track the field experience to validate this assumption.
Care should be taken to ensure that any unforeseen adverse impacts of corrective actions taken are identified and evaluated.
3.4.10 Follow-On Assessments and Responses.
The initial actions taken in response to an unsafe condition may be insufficient to mitigate the risk to acceptable levels. A complete understanding of the problems and contributing factors to the unsafe condition will most likely be available at a later point.
Therefore, the stakeholder(s) should consider follow-on responses and actions. The stakeholder(s) should apply the risk assessment process described in this paragraph to the decisions involved in the use of actions, whether initial or follow-on. The FAA response to an unsafe condition should be based on a technical understanding of the problem and should require an appropriate implementation schedule that is consistent with the risk assessment. When performing a follow-on assessment, the fact that exposure time has elapsed between the initial and follow-on assessments should not be used to justify extending the duration of the control program. The exposure time for evaluating the risk for follow-on actions is the amount of time needed to complete the entire corrective action program, such as any initial, interim, and final actions. The objective is to keep the risks below the applicable guidelines until the product is brought back to the level of safety intended by the product’s original basis of certification.
3-5
AC 39-8A
3.5 Assessment Model Considerations.
As mentioned above, quantitative assessments of potential unsafe conditions are the preferred method since they provide measurements that enhance the oversight and viability of potential corrective actions and prioritization. Often, the need to reduce the risk of an unsafe condition may not be supported by an adequate quantitative assessment. However, manufacturers should be able to acquire or develop data by experience, test, and analysis as needed for quantitative assessments. The stakeholders can use these assessments to judge the adequacy of control programs and validate that immediate or initial corrective actions provide sufficient risk reduction. Quantitative assessments, therefore, should be a goal for assessing any potential unsafe condition. A structured approach in performing quantitative assessments is essential for ensuring credible results. Important controls include anchoring the model to known facts, reviewing the input data, assumptions and judgment, calibrating the model to actual experience, reviewing all critical elements, establishing a consistent set of ground rules, and utilizing engineering judgment.
Note: The utilized safety and risk assessment methods are different for certification and continued airworthiness. Consequently, to avoid confusion, this AC does not focus on certification classifications and should not be used to support product certification.
3.5.1 Anchoring Model to Known Facts.
There will be many irrefutable facts within a problem. A problem can refer to an abnormality that might lead to an unsafe condition. For instance, these facts may include the number of parts failed, found cracked, and not cracked. The assessment model should not contradict any known information.
3.5.2 Reviewing Input Data, Assumptions, and Judgment.
All critical elements, input data, assumptions, and judgment require careful review and validation. A safety team approach in reviewing all critical elements, input data, assumptions, and judgment is most effective. The safety team should consist of experts from appropriate disciplines, such as stress analysis, fracture mechanics, reliability engineering, airworthiness, product support, inspection methods, etc. Agreement by consensus on critical assessment model inputs is essential, including validation of operational data by operators. Some examples of critical inputs include hazard ratio, inspection reliabilities, crack initiation and propagation data, failure distributions, part utilization, affected population definition, shop visit rates, material defect distributions, and rate of incorporating corrective actions. For example, realistic assessments of hazard ratio will often necessitate the involvement of the installer. The assessment should also consider the possibility of individual airplanes operating in a master minimum equipment list (MMEL) or other configuration that could adversely affect the risk associated with the unsafe condition. Although, it is neither necessary nor appropriate to assume that the entire fleet is operating in such a configuration.
Consideration should be made as to whether it is appropriate to allow continued dispatch in all previously acceptable configurations or missions.
Note: The risk assessment should be conducted objectively. This evaluation includes the severity and the conditional probability of an outcome given a failure condition. For 3-6
AC 39-8A example, whether one starts with the assessment that disk bursts are catastrophic or observes that there has been an HL 3 disk burst in service, the risk assessment for a given outcome should be equivalent.
3.5.3 Calibrating Model to Actual Experience.
It is essential that the model be capable of calibration to what has already happened. If the model does not calibrate, further team review by the stakeholders is necessary to determine which model assumptions may be in error. The model will not predict reliably if it cannot calibrate to the actual experience.
3.5.4 Reviewing All Critical Elements.
Subject to ex parte restrictions, it is important that all critical elements, input data, assumptions, judgment, etc., of the model be made available to the affected parties with data accessibility privileges for detailed review. Review by the responsible FAA engineers is essential for ensuring regulatory safety goals are met. Information, such as current service experience, from operators, installers, and the involved FAA inspectors is important for ensuring accurate data and viable corrective actions.
3.5.5 Establishing a Consistent Set of Ground Rules.
A consistent set of ground rules for constructing numerical assessments is necessary to ensure valid comparisons. Examples of areas where consistent ground rules are necessary include the determination of flight exposures, event HLs, hazard ratios, and event probability for each flight. There may be subtle yet significant differences in the ground rules used by different manufacturers in performing quantitative assessments.
Therefore, it is important to refrain from comparing assessment results from different manufacturers, unless it can be verified that the assessments were performed using the same ground rules. The desire to have a consistent set of ground rules for comparative purposes factors into the recommended use of a 20-year fleet exposure for uncorrected risk. It also factors into the use of hazard ratios to predict the risk of events of equivalent severity, such as HLs 3, 4, or 5.
3.5.6 Utilizing Engineering Judgment.
Regardless of how much engineering data is gathered to describe a problem mathematically, engineering judgment will always be necessary. Engineering judgment is, however, a potential source of subjectivity that can introduce uncertainty into the assessment. In assessing the need for action and the adequacy of a control program, it may be helpful to assess the potential variation in the major assumptions. This variation will result in a range of results. Acquiring additional data on model inputs will reduce the uncertainty and, therefore, reduce the range of possible results. This judgment and any other assumptions in the assessment need to be documented and validated to the greatest extent possible. The stakeholder should consider the amount of judgment and the level of confidence in the associated validations when determining the appropriate response to the problem.
3-7