• Determine the severity of the consequences, for a realized hazard;
• Express a combination of the likelihood and severity of harm as “risk;” and
• Evaluate the appropriate response to the identified risk.
(a) Your CASS should take into account four principal potential sources of hazards:
1. Personnel (hiring, capabilities, competence, interaction, and human factors),
2. Equipment (design, maintenance, logistics, and technology),
3. Workplace (environment, sanitation, and user friendly), and
4. Organization (safety culture, authority, responsibility, standards, procedures, and controls).
(b) There are a number of quantitative and graphical tools that exist to help you determine the gradations of a risk (high, medium, low) based on the likelihood of an unwanted event occurring and the severity of the consequences if it does occur. In the initial steps of your CASS process, the appropriate response involves setting surveillance priorities based on risk assessments aimed at maintaining compliance and safety in maintenance activities. Your CASS risk assessment, through the feedback loop, helps you to set the audit and data collection priorities that enhance the focus of surveillance. An interdisciplinary team, guided by your CASS management but involving representatives from relevant technical areas, can best accomplish this process.
(c) To identify the areas to audit and in order to set priorities, you should consider factors in outside reports. These could include results of inspections, reports, special studies, or audits conducted by outside entities such as the FAA, DOD, Department of Transportation (DOT), DOT Office of the Inspector General (OIG), or National Transportation Safety Board (NTSB). Outside reports may address:
• Information specific to you or your maintenance providers,
• Information related to the industry as a whole and of interest to you, or
• Relevant information about an accident, incident, or equipment type.
(4) Audit Materials. You should provide your CASS auditors with checklists to ensure consistency and completeness of audits. However, you should permit your auditors that flexibility necessary to ask questions not contained on the checklist if he or she finds an area that requires further investigation.
(5) Areas Requiring Auditing. The CASS monitors all 10 elements of your maintenance program. The following are examples of some, but not all, of the areas within each element of your maintenance program that the CASS looks at. You should note that all of these items are predisposed to surveillance of the maintenance program performance function of the CASS, accomplished through audits.
(a) Airworthiness Responsibility.
Page 18 Par 5-2
1. As an air carrier, you are primarily responsible for the airworthiness of your aircraft as well as the performance of maintenance, including work done by maintenance providers, on your aircraft. You must ensure that all completed maintenance, including work done by outside persons, is in accordance with your maintenance program and maintenance manual.
2. Your air carrier certificate makes you a maintenance entity. Each person who accomplishes maintenance on your aircraft does so as an agent on your behalf. Consistent with the privileges and limitations of your air carrier certificate, you must, through your maintenance organization, execute operational control over any maintenance activities anyone accomplishes on your aircraft or any of its component parts. Such activities include determining when maintenance is required, what maintenance is required, accomplishing the maintenance, determining if the maintenance was done satisfactorily, and approving your aircraft for RTS.
Consistent with the regulations, you may not delegate this responsibility to anyone else, including persons that you use for any of your maintenance, preventive maintenance, or alterations.
(b) Manuals.
1. The content of all your manuals, including maintenance manuals and technical content, is your responsibility. You may base your manuals on the Original Equipment Manufacturer (OEM) manuals or other information, but regulations require you to use your own manual, not the OEM manuals.
2. You should ensure that all of your manuals, publications, and forms are useable, current, correct, and readily available to all personnel required to use them.
3. You should ensure that each person who is required to comply with your manual must have access to it during performance of normal duties.
(c) Maintenance Organization.
1. Consistent with the responsibility described above, you must have a maintenance organization that is able to effectively exercise and maintain operational control over all persons performing, supervising, managing, and amending your maintenance program.
Your maintenance organization must be able to manage and guide your maintenance personnel and provide the direction necessary to achieve your overall maintenance program objectives.
2. The individual with overall maintenance program authority and responsibility is your Director of Maintenance (DOM), who should function as the accountable manager for the maintenance program. Your organization must clearly identify this individual, and he or she must be qualified in accordance with 14 CFR part 119, §§ 119.65 and 119.67(c) or §§ 119.69 and 119.71(e), as appropriate. While retaining overall authority and responsibility, your accountable manager may delegate direct responsibility for elements of the program as appropriate for the size and structure of your maintenance organization.
3. Your air carrier manual must contain a chart or a description of your maintenance organization showing clear authority and responsibility, including delegated Par 5-2 Page 19
responsibility, for the overall maintenance program and all of its elements and functions. The regulations also require you to include a description in your maintenance manual of the duties and responsibilities for each position in your maintenance organization so that you do not end up with a fragmented organizational system with high risk of confusion over who is responsible for a given task.
4. In order to be effective, an adequate maintenance organization must be able to demonstrate the following four organizational obligations:
• The obligation to define the environment within which individuals conduct their tasks,
• The obligation to define the policies and procedures that individuals must follow and respect,
• The obligation to allocate the resources that individuals need in order to achieve your safety and production goals, and
• The obligation to investigate system failures and take all needed remedial action to avoid a repetition.
5. Your maintenance organization will be predisposed to failure if you permit the following failures to occur:
• Failure to understand the effect of people on the safety and reliability of aircraft maintenance operations,
• Failure to organize your employees’ work,
• Failure to monitor your employees’ work effectively, and
• Failure to implement appropriate and effective corrective actions.
6. You must organizationally separate the performance of your RII function(s) from the performance of your other maintenance (including inspection), preventive maintenance, and alteration functions. This organizational separation must be below the level of the individual who has primary responsibility for your RII function, as well as your other maintenance, preventive maintenance, and alterations functions. In simple terms, this means that the part of the maintenance organization that accomplishes your maintenance, preventive maintenance, and alterations functions cannot be the same part of your maintenance organization that accomplishes your RII function. When implementing the organizational separation, do not confuse your maintenance department with your maintenance organization. Your maintenance department and your inspection department are both a part of your maintenance organization. The organizational separation has to occur below the individual in your organization who has primary responsibility for both the maintenance function and the required inspection function. This individual must be qualified to make an airworthiness determination.
(d) Maintenance Schedule. Your maintenance schedule sets out the item, task, and interval of your scheduled maintenance effort. You should ensure that you accomplish your maintenance schedule as stated. The maintenance schedule should be task-based and appropriately modified in accordance with the CASS data collection and analysis findings. Your audits should find that your initial selection and the continuous validation of each scheduled maintenance task and its associated interval is accomplished according to well-defined criteria Page 20 Par 5-2
throughout the service life of the item, system, or structure. Your maintenance schedule should be primarily proactive and designed to permit the item, system, or structure to do what it is designed to do.
(e) RIIs.
1. You must have specific procedures, standards, and limits necessary for the acceptance or rejection of each your RIIs and for periodic inspection and calibration of precision tools, measuring devices, and test equipment. You should note that the OEM’s manuals and procedures do not contain RII procedures, standards, and limits. You must develop and document these aspects yourself.
2. You must ensure that anyone who you authorize to accomplish RII inspections is properly trained and qualified for each RII task that you authorize him or her to perform.
3. You must ensure that any RII inspector understands that if he or she performs an item of work, he or she may not perform a required inspection on that item.
4. You must ensure that the structure of your maintenance organization is designed to organizationally separate the accomplishment of the required inspection function from the accomplishment of the other maintenance, including other inspection, preventive maintenance, and alteration functions.
5. You must ensure that your manual contains procedures to ensure that only supervisory personnel of an inspection unit or the individual who has overall responsibility for the RII function as well as the other maintenance, preventive maintenance, and alteration functions may countermand the decision of any RII inspector regarding an RII.
(f) Maintenance Providers. You must ensure that all of your maintenance providers and your suppliers are qualified and provide services and products according to your maintenance program and manual. Other than location, there should be no difference between the way your own employees and any of your maintenance providers complete work. Before you identify any maintenance provider in your maintenance provider listing as an essential maintenance provider, you should conduct an onsite audit that will permit you to determine that each proposed essential maintenance provider has an adequate organization and provides competent personnel and adequate facilities and equipment for the essential maintenance intended.
(g) Personnel Training.
1. You must have means that you can use to determine that all of your maintenance personnel, including maintenance provider personnel, are competent to accomplish their duties.
2. You must have a training program for maintenance personnel (including inspection personnel and maintenance provider personnel) who determine the adequacy of accomplished maintenance.
Par 5-2 Page 21
3. Part of your maintenance program should ensure that your maintenance personnel are competent to perform their duties.
(h) Accomplishment and Approval of Maintenance.
1. All of your maintenance facilities and equipment, including base and line stations as well as all of your maintenance providers’ facilities and equipment, are adequate to perform the maintenance. Other than scope and location, there should be no difference in the standards for facilities and equipment between you and your maintenance providers.
• You properly store, dispense, identify, and handle parts and components.
• You and your maintenance provider properly calibrate your tools and equipment.
• You identify requirements for specialized tools or training and provide the tools and training.
2. You or your maintenance provider perform(s) maintenance and alterations according to the methods, standards, and techniques specified in your manuals.
3. If you use an onsite representative to monitor work at a maintenance provider, you should ensure that you have clear procedures for this individual to report and document deviations from your manual or any of your methods, standards, and techniques specified in your manuals.
4. You or other persons properly document work interruptions and deferred maintenance in shift turnover records and accomplish them according to applicable procedures.
5. Major repairs and major alterations are properly classified, consistent with the 14 CFR part 1, § 1.1 meaning of major alteration or repair, and accomplished in accordance with FAA-approved technical data.
NOTE: During 1953, the Civil Aeronautics Authority (CAA) published a list of repairs to specific parts, as well as specific types of repairs that were considered major repairs in Civil Aeronautics Manual 18. This major repair list was later adopted, unchanged, as part of 14 CFR part 43 appendix A. If you rely exclusively on this standardized list of major repairs to make your major/minor classification, you will end up classifying some minor repairs as major and some major repairs as minor simply because the list has not been updated to include evolving airplane design and construction techniques such as composite structures, damage tolerant design, and the high speed pressurized jet transport that did not exist in 1953.
6. Appropriately certificated mechanics or repairmen whom you have authorized to do so, execute log entries and Airworthiness Release Forms.
7. You, or those you have authorized to do so, complete log entries and Airworthiness Release Forms according to your written policies and procedures.
Page 22 Par 5-2
(i) Maintenance Recordkeeping System.
1. You or other persons generate and retain maintenance records and current status records in accordance with your manual procedures.
2. Maintenance records and current status records are complete and correct.
3. You or other persons appropriately evaluate, accomplish, and track Airworthiness Directives (AD).
4. ADs with an alternative method of compliance (AMOC) and ADs that have been reconfigured (undone) are appropriately evaluated for accomplishment and for other modifications in the affected area (Supplemental Type Certificate (STC), major repairs, and alterations), documentation, and tracked.
NOTE: The current edition of AC 39-9, Airworthiness Directives Management Processes, provides you with additional information and guidance on developing a management process to help you comply with part 39, Airworthiness Directives.
5. The maintenance recordkeeping system identifies life-limited parts and tracks the current status time in service.
(j) CASS.
1. Your CASS ensures that you or other persons properly execute all elements of your maintenance program and that they are consistently effective by design rather than by chance.
2. Senior management reviews CASS issues on a regularly scheduled basis.
They hold meetings of CASS or maintenance management committees or boards on a regular basis to discuss findings, analysis, and the progress of corrective actions. These meetings may address events, as well as statistical data and trends.
(6) Objective of CASS Audits. Most of your CASS audits should be primarily proactive, searching out potential problem areas in program execution before they result in undesirable events. However, your CASS procedures may also be reactive when they address how to direct audits in response to events or a series of events. For example, an analysis of rejected takeoffs, unscheduled landings, in-flight shutdowns, or accidents or incidents may indicate the need for special audits or surveillance. One of the activities of a CASS identifies indications of program weaknesses or deficiencies through an analysis of trends. For example, your CASS auditors would not necessarily investigate a single maintenance-related rejected takeoff. A CASS would, however, consider whether that instance, coupled with other instances, indicated a need to focus audits on a particular issue.
(7) Informal Communications Within a CASS. Your auditors and analysts should maintain informal lines of communication with your personnel in the various maintenance departments, so that your maintenance personnel can discuss concerns they may have. Through Par 5-2 Page 23
this informal communications process, you can learn about potential hazards in your system. For example, you may learn about an event that might have occurred but, because of some intervention, did not. Your shop personnel know of this event, but it is otherwise difficult or impossible to detect in routine audits. If you keep informal lines of communication open to your shop personnel, your CASS may detect this near event. The FAA suggests that your CASS procedures address how to encourage this type of communication and interaction.
b. Analysis of Audits.
(1) RCA. A risk assessment process tells you where to allocate resources and helps you to understand what it finds. Your audit results should undergo risk assessment and preliminary RCA to identify a deficiency or potential deficiency in any element of your maintenance program. This preliminary analysis helps your CASS personnel determine the priority level that the issue merits and what type of additional technical expertise it may need to complete the RCA and to evaluate corrective action options.
(a) RCA treats errors as defects in the system rather than in an individual. RCA looks beyond the symptom to find the organizational defect that permitted an error to occur. The more thorough the analysis, the greater the likelihood you will uncover why the system deficiency could occur, and how your organization can respond definitively. The RCA process starts during the early part of an audit, as your auditors must collect information conducive to later analysis. If your CASS is to uncover a procedural weakness, for example, your auditors must collect information about the procedure. This should be factual and objective information, not premature judgment about root cause. RCA is key to any complete CASS, even though procedures may vary in complexity from certificate holder to certificate holder.
(b) An effective RCA can be as simple as asking and answering a question (five times) about why something happened. For example, if there is a human error in a maintenance process, the scenario could be as follows:
Question Answer Why was the error made? The individual involved was not competent to do the work.
Why? The training department did not identify a training need for this individual.
Why? The training needs identification process is very weak.
Why? The company has not identified the weakness in the training needs process.
Why? The training department lost two experienced analysts last year and has not replaced them.
(c) The above RCA leads to a training department workload problem. A superficial analysis might have lead to disciplinary action against one individual, which is indicative of a blame culture, and would, most likely, lead to a recurrence of the same error by a different individual.
Page 24 Par 5-2
(d) A CASS should reflect your overall management philosophy and it should avoid or discourage a “blame culture” within your maintenance organization. A blame culture:
• Fixes blame and moves on.
• Focuses on who made the error and their punishment.
• Encourages you to stop short of identifying:
• Systemic problems, or
• Root causes.
• Never permits you to fix the problem.
• Allows mishaps and mistakes to recur.
(e) RCA processes can range from the simple “five whys” above to the use of complex commercially available RCA software.
(2) Objective of Audit Analysis.
(a) Analysis Process. The objective of this analysis is to allow you to address the problem in such a way as to avoid a recurrence of the deficiencies. To the extent possible, you should set forth the analysis process in the CASS documentation. Your analysis process should be as objective as possible to avoid any tendency to promote individual or commercial interests.
You should also place priority on finding the systemic or root cause of a program deficiency over seeking to assign personal blame, at any level of the organization, for an error. There is a difference between accountability and culpability.
(b) System Deficiencies. While you should design your audits mainly to verify that you are performing maintenance in accordance with your manual, the regulations, and applicable requirements, your auditors and analysts should also be alert for system deficiencies. That is, there may be procedures in your manual that you correctly follow, but which have become outdated, conflict with other manual procedures, or for some other reason are in need of change.
You should encourage your auditors and analysts to be inquisitive and to think in terms of “what if?” so that your CASS functions proactively, detecting problem areas or trends before they lead to an accident, incident, or infraction of regulations. For example, what if event x occurred in conjunction with observed condition y? This inquisitive type approach should permeate your CASS organization, from determining audit priorities and scheduling, through auditing and analyzing, and including monitoring and evaluating corrective actions.
(3) Managing Data from Audit Analysis. Your audit analysis process is not typically as oriented toward quantitative analysis as the operational data analysis discussed below.
However, you may find it useful to manage the data through database or quantitative applications. The FAA emphasizes that this approach does not have to be complicated or costly.
The level of formality and sophistication should match that of your organization.
5-3. VERIFYING THE EFFECTIVENESS OF YOUR MAINTENANCE PROGRAM.
a. Surveillance of the Effectiveness of Your Maintenance Program.
Par 5-2 Page 25
(1) Collecting Operational Data. The main tool for determining whether your maintenance program is effective is to collect and analyze operational data that shows the availability of the equipment, which should be one of the objectives of your maintenance program. You should collect data that measures the output of your maintenance program. A primary function or output of your air carrier maintenance program is to permit your equipment (aircraft) to continue to do what it is supposed to do. Therefore, any activity (data) that shows that your equipment is not available to do what it is supposed to do would be a prime indicator of the level of maintenance program effectiveness. The level of unscheduled maintenance is such an indicator. Other indicators that may be reflected in a requirement for unscheduled maintenance are maintenance personnel not following the methods, techniques, and procedures in your manual; facilities and equipment not being adequate for the work that is being performed; and an aircraft being released to service when it is not airworthy.
(2) Types of Operational Data. Consistent with the performance-based CASS regulation, the FAA does not intend to mandate the specific data that you should collect.
However, the FAA does expect you to have a process to ensure that the data that you do collect is adequate to meet the intent of the CASS requirement and is useful. The FAA expects you to use an effective selection and periodic review process. The FAA does not want you to collect specific data elements that you do not use and that may not fit your particular situation. You can collect operational data from two different activities: routine or unplanned (non-routine) maintenance. However, you should keep any operational data that you collect from any unscheduled work occurring because of planned/routine work separate from the data that is not a result of planned/routine activities. Non-routine/unscheduled maintenance occurring as a result of scheduled, planned maintenance activities is a normal activity. This is the time when you should find discrepancies, and when you should accomplish the identified maintenance, not when your aircraft are in an operational status carrying passengers and/or cargo.
(a) However, you should not consider unscheduled maintenance activities occurring when your airplane is in an operational status carrying passengers and/or cargo as a normal and acceptable activity. After all, a primary function of your scheduled maintenance program is to permit your airplane to be available for operations in air transportation. Logically, a reduction of availability of your aircraft to carry passengers and/or cargo as a result of unscheduled maintenance activities is a primary indicator of the level of effectiveness of your maintenance program.
(b) Some examples of routine operational data are:
• Corrosion prevention and control program findings,
• Repair assessment findings,
• Engine Condition Monitoring (ECM) information,
• Scheduled structural inspection findings,
• Repairs accomplished as a result of scheduled structural inspection findings,
• Scheduled maintenance findings,
• Repairs accomplished as a result of scheduled maintenance findings,
• Vendor or maintenance provider information,
• Component teardown reports, and Page 26 Par 5-3
• Individual component failure rates.
(c) Examples of non-routine data are:
• Pilot reports, including maintenance deferred in accordance with the minimum equipment list (MEL)/Configuration Deviation List (CDL);
• “Chronic” systems that have repeat write-ups in a specified time period (for example, 10 to 15 days);
• Flight delays and cancellations related to maintenance issues;
• Service Difficulty Reports (SDR); and
• Unscheduled parts replacement or unscheduled maintenance.
(d) Operational data also includes reactive data collection and analysis responding to such non-routine events as:
• Accidents and incidents;
• High-load events such as hard landings, turbulence, etc.;
• In-flight engine and propeller separations and uncontained engine failures;
• In-flight engine shutdowns;
• In-flight propeller featherings;
• Lightning strikes;
• Unscheduled landings caused by mechanical difficulties or malfunctions;
and
• Rejected takeoffs caused by mechanical difficulties or malfunctions.
(e) As with reactive audit surveillance, your CASS should generally approach these types of problems from the analytical, systems perspective. For example, in response to one or more rejected takeoffs, a CASS might focus the operational data collection and analysis to determine if a pattern in rejected takeoffs was evident, or if it could examine other types of data or information in relation to the rejected takeoff circumstance.
(f) The above data sets are only examples. Although the data sets focus on equipment, this area of your CASS may also collect other types of data, such as information on types of maintenance errors that you may have experienced.
(3) CASS Documentation. What you should include in your CASS documentation regarding collection of operational data. Your CASS documentation should include a means of identifying data that is relevant and useful for you to use in monitoring the effectiveness of your specific and unique maintenance program. You should periodically review and reevaluate the usefulness of the data that you collect and analyze to accomplish this portion of your CASS.
b. Analysis of Operational Data. Your CASS procedures should:
(1) Provide your analysts with an understanding of the potential significance of each data set and how to process the data to understand its significance. This may require:
Par 5-3 Page 27
• Statistical analysis, such as comparing the frequency of certain events or equipment failures with a determined norm; or
• Qualitative analysis, to evaluate reports of certain types of events.
NOTE: This process is not necessarily the same as that used in an FAA-approved reliability program.
(2) Emphasize that analysis of operational data should consider root causes of negative trends or anomalies. This preliminary RCA, including human factors, may require collaboration with technical personnel in the affected areas or specialists in engineering and reliability departments.
(3) Delineate the roles of your CASS analysts as well as other departments or personnel in the analysis of operational data. You may decide to select a system that uses alerts or warnings if results of your analysis exceed certain predetermined parameters. However, your CASS should not rely completely on such alerts to the exclusion of your analysts’ judgment. In addition, the use of alerts has historically been associated with parts and their failure rates, while current maintenance methodology focuses on systems and the loss of function, not part failure rates. The FAA’s expectation of your CASS in this regard is that you have a complete, written procedure to review and analyze the operational data collected and to determine when further review is necessary.
5-4. FINAL RCA AND CORRECTIVE ACTION. While the surveillance and analysis steps differ for the verification of performance of your maintenance program versus verification of the effectiveness of your program, the process merges when you respond to CASS findings. The two types of analysis identify potential deficiencies in your maintenance program. In responding to these findings and analyses, the objective of your CASS is to determine the root causes of your program deficiencies and address them appropriately, regardless of the perspective from which you found the deficiencies. Generally, the organizational area responsible for analysis will present their results to your technical or production area with a preliminary analysis of the collected information and, in some cases, possible underlying causes of the problem. Personnel in your technical or production areas complete the RCA (if necessary) and develop proposed corrective action alternatives.
a. Final RCA.
(1) Preparing for RCA.
(a) Analysis of audit findings or operational data requires you to evaluate mechanical, human performance, or other data generated by your CASS in order to determine the effectiveness of a process, maintenance practice, or equipment. In the case of operational data, analysis begins with comparison of the data to a standard representing acceptable performance.
The standard may be in the form of an average or other means of calculating a reference. The standard may be set by the FAA, industry common practice, or by you, the certificate holder, as appropriate.
Page 28 Par 5-3
(b) Your CASS procedures should note that in performing RCA, you should consider all relevant areas, including the role of your maintenance organization, your senior management, and your policies, procedures, and communications.
(2) Applicability of RCA. RCA applies to both audit findings and operational data analysis. For example, either audits or operational data analysis may point to maintenance errors committed because of inadequate training or a confusing work card. Analysis should not stop with simply correcting the work card or determining which mechanics were inadequately trained and then training them. Rather, the analysis should determine why the work card got that way, or why the training breach occurred and consider looking at other areas (including the role of your maintenance organization and senior management), such as communications, staffing, scheduling, or training program design that may be involved.
(3) Principles and Considerations of RCA. Principles and considerations of RCA are closely related to those of risk assessment, particularly in terms of the thoroughness of the analysis. Both processes consider not simply the person involved in an issue (for example, the mechanic made a mistake), but all aspects of the organization and each individual in that organization where that person works. This approach has the premise that human error is a consequence rather than a deliberate action and, as such, proactive measures and continuous reform of different aspects of the processes and organization can address latent conditions in the system and increase the system’s resistance to operational hazards. The term “latent conditions” refers to flawed procedures or organizational characteristics capable of creating hazards if the right conditions or actions occur.
(a) Your RCA should consider two major areas:
1. Systems analysis plays an increasingly important role in CASS because of the increasing complexity and variety of air carrier operations, equipment, and organizations.
Systems analysis emphasizes a harmonized approach to an enterprise, including specific written procedures and planning for all activities, clearly established authority and responsibilities, communications processes, and methods of measuring results, detecting system errors, and preventing recurrence. This harmonized approach recognizes the wide range of interrelated issues that are potentially associated with a problem in the system, such as management direction and policies, communications, and pilot technique, in addition to the maintenance activities themselves.
2. Human factors analysis begins with your maintenance organization itself. As stated earlier in this AC, your maintenance organization:
• Defines the environment where your employees conduct their tasks.
• Defines the policies and procedures that your employees must follow and respect.
• Allocates the resources that your employees need to achieve your safety and production goals.
• Investigates system failures and takes all needed remedial action to avoid a recurrence.
Par 5-4 Page 29