Page 24 Par 5-2 undesirable events. However, your CASS procedures may also be reactive when they address how to direct audits in response to events or a series of events. For example, an analysis of rejected takeoffs, unscheduled landings, in-flight shutdowns, or accidents or incidents may indicate the need for special audits or surveillance. One of the activities of a CASS identifies indications of program weaknesses or deficiencies through an analysis of trends. For example, your CASS auditors would not necessarily investigate a single maintenance-related rejected takeoff. A CASS would, however, consider whether that instance, coupled with other instances, indicated a need to focus audits on a particular issue.
(7) Informal Communications Within a CASS. Your auditors and analysts should maintain informal lines of communication with your personnel in the various maintenance departments, so that your maintenance personnel can discuss concerns they may have. Through this informal communications process, you can learn about potential hazards in your system. For example, you may learn about an event that might have occurred but, because of some intervention, did not. This event is known to your shop personnel but is otherwise difficult or impossible to detect in routine audits. If you keep informal lines of communication open to your shop personnel, your CASS may detect this near event. The FAA suggests that your CASS procedures address how to encourage this type of communication and interaction.
b. Analysis of Audits.
(1) RCA. A risk assessment process tells you where to allocate resources and helps you to understand what it finds. Your audit results should undergo risk assessment and preliminary RCA to identify a deficiency or potential deficiency in any element of your maintenance program. This preliminary analysis helps your CASS personnel determine the priority level that the issue merits and what type of additional technical expertise it may need to complete the RCA and to evaluate corrective action options.
(a) RCA treats errors as defects in the system rather than in an individual. RCA looks beyond the symptom to find the organizational defect that permitted an error to occur. The more thorough the analysis, the greater the likelihood you will uncover why the system deficiency could occur, and how your organization can respond definitively. The RCA process starts during the early part of an audit, as your auditors must collect information conducive to later analysis. If your CASS is to uncover a procedural weakness, for example, your auditors must collect information about the procedure. This should be factual and objective information, not premature judgment about root cause. RCA is key to any complete CASS, even though procedures may vary in complexity from certificate holder to certificate holder.
(b) An effective RCA can be as simple as asking and answering a question (five times) about why something happened. For example, if there is a human error in a maintenance process, the scenario could be as follows:
Question Answer Why was the error made? The individual invo lved was not competent to do the work.
Why? The training department did not identify a training need for this individual.
Why? The training needs identi fication process is very weak.
Par 5-2 Page 25
Why? The company has not identif ied the weakness in the training needs process.
Why? The training department lost two experienced analysts last year and has not replaced them.
(c) The above RCA leads to a training department workload problem. A superficial analysis might have lead to disciplinary action against one individual, which is indicative of a blame culture, and would, most likely, lead to a recurrence of the same error by a different individual.
(d) A CASS should reflect your overall management philosophy and it should avoid or discourage a “blame culture” within your maintenance organization. A blame culture:
• Fixes blame and moves on.
• Focuses on who made the error and their punishment.
• Encourages you to stop short of identifying:
• Systemic problems, or
• Root causes.
• Never permits you to fix the problem.
• Allows mishaps and mistakes to recur.
(e) RCA processes can range from the simple “five whys” above to the use of complex commercially available RCA software.
(2) Objective of Audit Analysis.
(a) Analysis Process. The objective of this analysis is to allow you to address the problem in such a way as to avoid a recurrence of the deficiencies. To the extent possible, you should set forth the analysis process in the CASS documentation. Your analysis process should be as objective as possible to avoid any tendency to promote individual or commercial interests.
You should also place priority on finding the systemic or root cause of a program deficiency over seeking to assign personal blame, at any level of the organization, for an error. There is a difference between accountability and culpability.
(b) System Deficiencies. While you should design your audits mainly to verify that you are performing maintenance in accordance with your manual, the regulations, and applicable requirements, your auditors and analysts should also be alert for system deficiencies. That is, there may be procedures in your manual that you correctly follow, but which have become outdated, conflict with other manual procedures, or for some other reason are in need of change.
You should encourage your auditors and analysts to be inquisitive and to think in terms of “what if?” so that your CASS functions proactively, detecting problem areas or trends before
Page 26 Par 5-2 they lead to an accident, incident, or infraction of regulations. For example, what if event x occurred in conjunction with observed condition y? This inquisitive type approach should permeate your CASS organization, from determining audit priorities and scheduling, through auditing and analyzing, and including monitoring and evaluating corrective actions.
(3) Managing Data from Audit Analysis. Your audit analysis process is not typically as oriented toward quantitative analysis as the operational data analysis discussed below.
However, you may find it useful to manage the data through database or quantitative applications. The FAA emphasizes that this approach does not have to be complicated or costly.
The level of formality and sophistication should match that of your organization.
5-3. VERIFYING THE EFFECTIVENESS OF YOUR MAINTENANCE PROGRAM.
a. Surveillance of the Effectiveness of Your Maintenance Program.
(1) Collecting Operational Data. The main tool for determining whether your maintenance program is effective is to collect and analyze operational data that shows the availability of the equipment, which should be one of the objectives of your maintenance program. You should collect data that measures the output of your maintenance program. A primary function or output of your air carrier maintenance program is to permit your equipment (aircraft) to continue to do what it is supposed to do. Therefore, any activity (data) that shows that your equipment is not available to do what it is supposed to do would be a prime indicator of the level of maintenance program effectiveness. The level of unscheduled maintenance is such an indicator. Other indicators that may be reflected in a requirement for unscheduled maintenance are maintenance personnel not following the methods, techniques, and procedures in your manual; facilities and equipment not being adequate for the work that is being performed; and an aircraft being released to service when it is not airworthy.
(2) Types of Operational Data. Consistent with the performance-based CASS regulation, the FAA does not intend to mandate the specific data that you should collect.
However, the FAA does expect you to have a process to ensure that the data that you do collect is adequate to meet the intent of the CASS requirement and is useful. The FAA expects you to use an effective selection and periodic review process. The FAA does not want you to collect specific data elements that you do not use and that may not fit your particular situation. You can collect operational data from two different activities: routine or unplanned (non-routine) maintenance. However, you should keep any operational data that you collect from any unscheduled work occurring because of planned/routine work separate from the data that is not a result of planned/routine activities. Non-routine/unscheduled maintenance occurring as a result of scheduled, planned maintenance activities is a normal activity. This is the time when you should find discrepancies, and when you should accomplish the identified maintenance, not when your aircraft are in an operational status carrying passengers and/or cargo.
(a) However, you should not consider unscheduled maintenance activities occurring when your airplane is in an operational status carrying passengers and/or cargo as a normal and acceptable activity. After all, a primary function of your scheduled maintenance program is to permit your airplane to be available for operations in air transportation. Logically, a reduction of availability of your aircraft to carry passengers and/or cargo as a result of unscheduled
Par 5-3 Page 27 maintenance activities is a primary indicator of the level of effectiveness of your maintenance program.
(b) Some examples of routine operational data are:
• Corrosion prevention and control program findings,
• Repair assessment findings,
• Engine Condition Monitoring (ECM) information,
• Scheduled structural inspection findings,
• Repairs accomplished as a result of scheduled structural inspection findings,
• Scheduled maintenance findings,
• Repairs accomplished as a result of scheduled maintenance findings,
• Vendor or maintenance provider information,
• Component teardown reports, and
• Individual component failure rates.
(c) Examples of non-routine data are:
• Pilot reports, including maintenance deferred in accordance with the minimum equipment list (MEL)/Configuration Deviation List (CDL);
• “Chronic” systems that have repeat write-ups in a specified time period (for example, 10 to 15 days);
• Flight delays and cancellations related to maintenance issues;
• Service Difficulty Reports (SDR); and
• Unscheduled parts replacement or unscheduled maintenance.
(d) Operational data also includes reactive data collection and analysis responding to such non-routine events as:
• Accidents and incidents;
• High-load events such as hard landings, turbulence, etc.;
• In-flight engine and propeller separations and uncontained engine failures;
Page 28 Par 5-3
• In-flight engine shutdowns;
• In-flight propeller featherings;
• Lightning strikes;
• Unscheduled landings caused by mechanical difficulties or malfunctions;
and
• Rejected takeoffs caused by mechanical difficulties or malfunctions.
(e) As with reactive audit surveillance, your CASS should generally approach these types of problems from the analytical, systems perspective. For example, in response to one or more rejected takeoffs, a CASS might focus the operational data collection and analysis to determine if a pattern in rejected takeoffs was evident, or if it could examine other types of data or information in relation to the rejected takeoff circumstance.
(f) The above data sets are only examples. Although the data sets focus on equipment, this area of your CASS may also collect other types of data, such as information on types of maintenance errors that you may have experienced.
(3) CASS Documentation. What you should include in your CASS documentation regarding collection of operational data. Your CASS documentation should include a means of identifying data that is relevant and useful for you to use in monitoring the effectiveness of your specific and unique maintenance program. You should periodically review and reevaluate the usefulness of the data that you collect and analyze to accomplish this portion of your CASS.
b. Analysis of Operational Data. Your CASS procedures should:
(1) Provide your analysts with an understanding of the potential significance of each data set and how to process the data to understand its significance. This may require:
• Statistical analysis, such as comparing the frequency of certain events or equipment failures with a determined norm; or
• Qualitative analysis, to evaluate reports of certain types of events.
NOTE: This process is not necessarily the same as that used in an FAA-approved reliability program.
(2) Emphasize that analysis of operational data should consider root causes of negative trends or anomalies. This preliminary RCA, including human factors, may require collaboration with technical personnel in the affected areas or specialists in engineering and reliability departments.
(3) Delineate the roles of your CASS analysts as well as other departments or personnel in the analysis of operational data. You may decide to select a system that uses alerts or warnings if results of your analysis exceed certain predetermined parameters. However, your CASS should
Par 5-3 Page 29 not rely completely on such alerts to the exclusion of your analysts’ judgment. In addition, the use of alerts has historically been associated with parts and their failure rates, while current maintenance methodology focuses on systems and the loss of function, not part failure rates. The FAA’s expectation of your CASS in this regard is that you have a complete, written procedure to review and analyze the operational data collected and to determine when further review is necessary.
5-4. FINAL RCA AND CORRECTIVE ACTION. While the surveillance and analysis steps differ for the verification of performance of your maintenance program versus verification of the effectiveness of your program, the process merges when you respond to CASS findings. The two types of analysis identify potential deficiencies in your maintenance program. In responding to these findings and analyses, the objective of your CASS is to determine the root causes of your program deficiencies and address them appropriately, regardless of the perspective from which you found the deficiencies. Generally, the organizational area responsible for analysis will present their results to your technical or production area with a preliminary analysis of the collected information and, in some cases, possible underlying causes of the problem. Personnel in your technical or production areas complete the RCA (if necessary) and develop proposed corrective action alternatives.
a. Final RCA.
(1) Preparing for RCA.
(a) Analysis of audit findings or operational data requires you to evaluate mechanical, human performance, or other data generated by your CASS in order to determine the effectiveness of a process, maintenance practice, or equipment. In the case of operational data, analysis begins with comparison of the data to a standard representing acceptable performance.
The standard may be in the form of an average or other means of calculating a reference. The standard may be set by the FAA, industry common practice, or by you, the certificate holder, as appropriate.
(b) Your CASS procedures should note that in performing RCA, you should consider all relevant areas, including the role of your maintenance organization, your senior management, and your policies, procedures, and communications.
(2) Applicability of RCA. RCA applies to both audit findings and operational data analysis. For example, either audits or operational data analysis may point to maintenance errors committed because of inadequate training or a confusing work card. Analysis should not stop with simply correcting the work card or determining which mechanics were inadequately trained and then training them. Rather, the analysis should determine why the work card got that way, or why the training breach occurred and consider looking at other areas (including the role of your maintenance organization and senior management), such as communications, staffing, scheduling, or training program design that may be involved.
(3) Principles and Considerations of RCA. Principles and considerations of RCA are closely related to those of risk assessment, particularly in terms of the thoroughness of the analysis. Both processes consider not simply the person involved in an issue
Page 30 Par 5-4 (for example, the mechanic made a mistake), but all aspects of the organization and each individual in that organization where that person works. This approach has the premise that human error is a consequence rather than a deliberate action and, as such, proactive measures and continuous reform of different aspects of the processes and organization can address latent conditions in the system and increase the system’s resistance to operational hazards. The term “latent conditions” refers to flawed procedures or organizational characteristics capable of creating hazards if the right conditions or actions occur.
(a) Your RCA should consider two major areas:
1. Systems analysis plays an increasingly important role in CASS because of the increasing complexity and variety of air carrier operations, equipment, and organizations.
Systems analysis emphasizes a harmonized approach to an enterprise, including specific written procedures and planning for all activities, clearly established authority and responsibilities, communications processes, and methods of measuring results, detecting system errors, and preventing recurrence. This harmonized approach recognizes the wide range of interrelated issues that are potentially associated with a problem in the system, such as management direction and policies, communications, and pilot technique, in addition to the maintenance activities themselves.
2. Human factors analysis begins with your maintenance organization itself. As stated earlier in this AC, your maintenance organization:
• Defines the environment where your employees conduct their tasks.
• Defines the policies and procedures that your employees must follow and respect.
• Allocates the resources that your employees need to achieve your safety and production goals.
• Investigates system failures and takes all needed remedial action to avoid a recurrence.
(b) Within your maintenance organization, human factors analysis looks at how your employees communicate and perform in the work environment and then seeks to incorporate that knowledge into the design of equipment, processes, and organizations. This enhances safety and maximizes the human contribution, partly by designing systems to anticipate the inevitability of human error. Human factors includes basic issues that can be addressed in audit checklists, such as whether there is adequate lighting for maintenance personnel to perform their work and whether schedules permit personnel to be properly rested. However, the human factors discipline addresses a wider range of issues affecting how people interface with technology and the operational system; how people learn about new or changed equipment, technology, and documentation; and how people adapt to the general workplace environment.
(c) You, as an air carrier, should be aware that knowledge gained from human factors can help you avoid maintenance errors, ensure that individuals’ initial skill sets match
Par 5-4 Page 31 task requirements, ensure that individuals maintain and improve their skills, and enhance the work environment. This knowledge can help your CASS analysts perform RCA. Continuing with the previous example of inadequate training, with insufficient awareness of human factors issues, you might trace a maintenance error to a mechanic or technician who appears to have insufficient training for the task, and determine that the solution is more technical training. However, further analysis may reveal that there are contributing flaws in equipment design, job cards, manuals, work environment, or such organizational procedures as shift turnover reports that more training will not satisfactorily overcome. Or, it may turn out that a different kind of training, perhaps involving decision-making skills, is called for.
(d) The FAA is involved in cooperative efforts with the industry and academia in promoting human factors in aviation. This field is rapidly evolving, particularly in its application to aviation maintenance. According to a study conducted for the FAA, maintenance error contributes to a significant portion of air carrier accidents, with shift-turnover errors and work interruptions standing out as leading types of underlying causes. Based on human factors’ growing importance and the human factors information available to the industry, the FAA expects that you will apply concepts of human factors to your CASS surveillance and analysis.
(e) Your CASS surveillance should ensure that an RCA that considers human factors is part of the investigation of individual events by any personnel designated by you to respond to maintenance-related events, such as rejected takeoffs. Otherwise, data reviewed in a CASS could be incomplete.
(f) One challenge presented by the increasing emphasis on human factors is how to balance two seemingly contradictory purposes. On the one hand, the FAA and industry need to encourage personnel to cooperate in addressing system organization and design issues without inhibitions caused by fear of discipline or enforcement. On the other hand, in some cases, individual employees or even senior management may bear a degree of culpability (for example, in deliberately bypassing important controls or committing a serious regulatory infraction in the commission of a maintenance error). In some instances, disciplinary action or even FAA administrative or legal enforcement may be indicated. This is a common issue in the industry. FAA programs designed to promote the greater good of the system, instead of the blame culture described earlier, have become available. These programs encourage voluntary reporting of errors and infractions by aviation personnel and operators without threat of disciplinary action or penalty. Your specific CASS, in any event, should be concerned specifically with identifying and correcting deficiencies in your maintenance program. You should design your CASS with that objective in mind, rather than specific event resolution, even if your CASS analysts research specific events.
b. Analytical Tools and Processes. While it is not necessary that you implement any specific externally developed system, specific analytical tools or processes are available to assist you in your analysis process. Examples of these follow:
(1) Maintenance Error Decision Aid (MEDA). Developed by the Boeing Human Factors Engineering group in collaboration with the FAA, airlines, and the International Association of Machinists, MEDA analyzes human performance issues related to maintenance errors and trends. MEDA is used to track events, investigate and prevent maintenance errors, and
Page 32 Par 5-4 identify contributing factors, corrective actions, and prevention strategies. The group of collaborators has also developed a software analysis package to work with this aid and facilitate analysis of systemic issues.
(2) Managing Engineering Safety Health. Developed by the University of Manchester in collaboration with British Airways (BA) Engineering. It is a system geared toward researching the workplace and organizational environment in aircraft maintenance to find the issues with greatest potential to contribute to human factors problems. The system uses software, diagnostic, and sampling tools. Managing Engineering Safety Health conducts anonymous, survey-like assessments among personnel at the work location, which are then analyzed. This is a more structured, data-intensive approach to determining and monitoring personnel attitudes toward the system than the interview process discussed earlier. The industry has far less practical experience with Managing Engineering Safety Health than with MEDA.
(3) Human Factors Accident Classification System Maintenance Extension.
Developed by the U.S. Naval Safety Center in collaboration with the FAA for use in the air carrier industry as well as naval aviation, this comprehensive system incorporates a number of analytical tools and has profiled maintenance errors and contributing conditions, permitting development of potential prevention measures. While the Human Factors Accident Classification System Maintenance Extension may be more sophisticated than many operators would need, it demonstrates principles and techniques of software-aided analysis that could apply to a CASS.
c. Corrective Action Options.
(1) Determining Whether or Not to Proceed with a Corrective Action. Once your CASS auditors and analysts have identified a problem or deficiency, you must determine if a corrective action is warranted and, if so, the details of the corrective action
(i.e., the corrective action plan).
(2) CASS Procedures. CASS procedures regarding the determination of whether you should proceed with a corrective action. Your CASS procedures should outline:
• How such a determination will be made,
• Who will make the determination, and
• What levels of review, if any, you will perform. A review is a system safety control.
(3) Developing a Corrective Action Proposal. Your technical area personnel should have primary responsibility for developing the corrective action proposal, as they would be most familiar with the technical workings of the area in question and would be sensitive to the possibility of creating new problems as a result of the corrective action. Your CASS procedures should emphasize a team approach. Team members should include your CASS auditors or analysts, your technical area personnel in the affected maintenance disciplines, and perhaps other affected areas such as training or flight operations.
Par 5-4 Page 33
(4) Types of Corrective Actions. There are several possible types of general corrective actions or responses, depending on the outcome of your risk assessment:
(a) Prevent recurrence through engineering or system changes designed to eliminate the risk.
(b) Accept the underlying cause of a trend or discrepancy, but reduce the risk through implementing controls or countermeasures. Examples are training, policy or procedure revisions, or warning devices. Other countermeasures might be modifying or introducing new equipment or technology.
(c) Accept that under certain conditions a discrepancy may occur, but be prepared to contain or mitigate the results of that situation. Your CASS does not necessarily have to implement corrective actions for every apparently negative trend or finding. Analysis of findings or trends may identify problem areas that do not present safety hazards that you are willing to accept, in accordance with your risk assessment process. For example, you might find that a higher than average number of component removals occur at a particular location with “no fault found” as a consequence of a non-safety-related failure. You might determine that the reason for this situation is that your aircraft spends insufficient time on the ground for line maintenance to completely isolate the fault. You might prefer to continue the brief turn times and simply change components. This would be a business decision for you to make. However, more comprehensive corrective actions would be mandatory if your CASS detects that your maintenance program lacks adequate procedures and standards to meet the requirements of part 121 or 135, as applicable.
d. Written Procedures for Developing and Implementing Corrective Actions. Your CASS should provide relatively detailed written procedures for developing and implementing corrective action. Your written procedures should:
(1) Result in a specific corrective action plan that addresses the following:
• Development and proposal of the corrective action.
• Analysis and final approval level of the corrective action, including who is responsible for approval of the corrective action.
• Who will implement the corrective action.
• How the responsible person will implement the corrective action.
• When the corrective action completion due date is.
• Who will evaluate the outcome and how, including identification of data requiring collection, awareness of the possibility of unintended consequences, and events that should trigger a response.
• Who will monitor the status of the corrective action and how.
Page 34 Par 5-4
• Reporting the status of the corrective action (to whom, with what frequency).
(2) Maintain the appropriate role of auditors in developing responses to findings, so that they continue to remain independent from the corrective actions they may subsequently audit.
(3) Distinguish clearly between your technical area personnel’s responsibility for developing and implementing corrective actions, and your CASS personnel’s responsibility for producing the findings and analysis and making sure the technical area involved develops and implements appropriate corrective actions.
(4) Designate the position or organization responsible for evaluating and approving proposed corrective actions. Your CASS director or other designated manager may appoint a corrective action team to design and propose a corrective action. The team—which typically represents a cross section of the departments involved in audits, operational data collection, analysis, and production—oversees the implementation of the corrective action. Technical and reliability control boards are most often used in conjunction with FAA-approved reliability programs; however, a similar concept applies to a CASS, even if no FAA-approved reliability program exists.
e. Corrective Action Risk Assessment.
(1) CASS Procedures Regarding Risk Assessment. Your CASS procedures should:
(a) Specify that your personnel will analyze a proposed corrective action carefully before its selection and implementation to ensure that corrective action is necessary and will actually fix the problem and not lead to unintended, negative consequences.
(b) Remind both CASS and technical area personnel of the need to consider the impact of the proposed corrective action on other aspects of your operation. This includes other areas of your maintenance program, such as manuals. The corrective action may require coordination with other areas, such as flight operations, that it could affect.
(2) Personnel Involved in Risk Assessment.
(a) Your technical area personnel play the key role in risk assessment, but the process should include your CASS analysts, who will act as resources in support of your technical area managers and bring risk assessment and systems analysis techniques to the process. Your auditor and analyst should be qualified (through training or experience) in systems analysis and can contribute to the evaluation of a proposed corrective action by determining if the basic system elements have been considered. However, your technical personnel have the expertise to actually develop and implement the corrective action, and to evaluate it in practical terms. Thus, the corrective action is a result of cooperation between your technical personnel and your CASS personnel.
(b) Personnel working on the proposed corrective actions should ensure they consider issues of a timetable for the corrective action implementation, and the safety attributes of authority, responsibility, procedures, controls, process measurement, and interfaces.
Par 5-4 Page 35
f. Corrective Action Plan.
(1) With the RCA complete, corrective action options identified, and risk assessment performed as appropriate, you can make a final decision on the proposed corrective action plan.
The corrective action plan should address all relevant issues, including a timetable for completion of the action, with milestones if appropriate. The appropriate technical department and, if the corrective action goes beyond your maintenance organization, other departments (such as flight operations) should then implement the plan.
(2) Your CASS procedures should identify:
• How this plan will be approved and at what level of your company.
• The parties responsible for implementing, monitoring, and ensuring that all affected parties are notified, both within your maintenance organization and externally, if necessary.
5-5. FOLLOWUP.
a. Corrective Action Followup. Your CASS procedures should:
(1) Specify how you will evaluate your corrective actions for effectiveness. That is, did your corrective action do what you wanted it to do? This may require the following:
• Followup audits of a specific area,
• Regular communication from the affected technical area about the effects of the corrective action and,
• Other forms of verification by your auditors or analysts tracking the implementation.
(2) Identify the person or entity (such as a CASS board) responsible for determining if any changes in the status of a corrective action are acceptable. Your CASS auditors or analysts have the duty of ensuring that the person implementing the corrective action did so in accordance with the established timetable, or, if not, determining why the timetable has changed.
(3) Include responsibilities and guidelines for:
• Tracking the implementation of corrective actions in accordance with the timeline.
• The role of your auditors, managers, management committees, and senior management.
• How your CASS will use automation or computerized systems.