Page 38 Par 6-2
(2) Be able to understand and interpret the data they see, as well as evaluate facilities, equipment, and processes they observe. While they are unlikely to have specialized knowledge in all of the areas over which they conduct surveillance, a foundation of technical expertise is important.
b. Surveillance and Analysis. Your auditors and analysts should have training and experience in the functions they are responsible for looking at and analyzing. However, it is also essential that they have training and experience in the following areas:
• Systems analysis,
• Auditing techniques,
• Risk assessment and risk management (RM),
• RCA, and
• Human factors.
c. Quality Processes and Systems. Additionally, you may seek specialized training in specific quality processes or systems for your CASS personnel, such as:
(1) The International Organization for Standardization (ISO) developed ISO 9000, which is a quality system set of standards that seeks to standardize processes into organized and documented systems.
(2) Six Sigma, which is process-oriented, but from an intensively data-oriented, statistical approach.
d. Technical. Persons who collect and analyze operational data may require specialized technical backgrounds. This will depend on the level of complexity of the operational data that you collect. These personnel may work in the unit conducting an FAA-approved reliability program or in an independent data collection and analysis system. You may also wish to consider auditor certification. There are a number of auditor certifications available such as those under ISO 9001 and the RABQSA International Certification AS9110, Aerospace Auditor.
e. Summary of Experience and Training for Your CASS Personnel. Your CASS document should reflect that you have considered the type of experience and training, initial and recurrent, appropriate to your auditors and analysts in your operation. Areas to consider include those listed in the following table:
Par 6-2 Page 39 (and 40) Table 6-1. Subject Training Subject Area Auditors Audit Analysts Operational Data Analysts Title 14 of the Code of Federal Regulations part 121/135 (as applicable);
operations specifications.
Initial and recurrent. Initial and recurrent. Initial and recurrent.
Systems analysis training.
General training in quality standards.
General training in quality standards, statistics, and/or training in widely used industry courses such as ISO 9000.
General training in quality standards, statistics, and/or training in widely used industry courses such as ISO 9000.
Audit training. Init ial and recurrent.
Coordinating Agencies for Supplier’s Evaluation (C.A.S.E.) training if applicable.
Experience or training in conducting and reporting results of audits.
Statistical investigation experience or training.
Risk assessment training.
Initial and recurrent. Initial and recurrent. Initial and recurrent.
Root cause analysis training, including human factors.
Initial and recurrent. Initial and recurrent. Initial and recurrent.
Technical competence.
FAA mechanic certificate, engineering, or other maintenance background.
FAA mechanic certificate, engineering, or other maintenance background.
FAA mechanic certificate, engineering, or other maintenance background.
Educational background.
Related education or training may partially fulfill similar qualification requirements set by you.
Related education or training may partially fulfill similar qualification requirements set by you.
Related education or training may partially fulfill similar qualification requirements set by you.
Par 7-1 Page 41 CHAPTER 7. COMMUNICATIONS BETWEEN YOUR CASS PERSONNEL AND OTHER
DEPARTMENTS
7-1. COMMUNICATING SPECIFIC CASS RESULTS AND ACTIONS. The procedures that you would use for communicating CASS information and results to interested parties, internally and, as applicable, externally (for example, vendors, or the FAA) varies depending on factors such as the size and nature of your operation, the level of automation, and your CASS procedures themselves. The number and complexity of the standardized communications processes, such as forms or electronic mail messages with standard distribution, should be appropriate to the overall size and scope of your operation and CASS.
a. Standard Communication Processes. You should develop appropriate standard communication processes for all aspects of your CASS to assist in standardizing procedures, including the following:
(1) Audit checklists and results.
(2) Analysis procedures and results.
(3) Records of audit/analysis findings: internal.
(4) Records of audit/analysis findings: external.
(5) Corrective action forms and corrective action plans. These forms should address system considerations to ensure that there is a clear understanding of when the person implementing the corrective action will do so, who is responsible, and what the impact will be on written procedures.
(6) Information for monitoring and followup of corrective action. The processes should also assist in tracking the implementation of corrective actions once underway.
(7) Periodic status reports to senior management and to the FAA.
b. CASS Description. Your CASS description should address issues such as:
(1) Who is responsible for keeping these standard communication processes up-to-date and available.
(2) Who is responsible for completing the standard communication processes.
(3) Where it sends communications, who must respond, and how it tracks responses.
(4) How, where, and for how long it retains completed records.
7-2. EDUCATING PERSONNEL ON CASS. Your CASS should include procedures and responsibility to create some form of communication between the area responsible for the CASS, other areas of the company, and the FAA. You may accomplish this through training, newsletters, bulletins, meetings, or another format that you determine is suitable. One purpose of
Page 42 Par 7-2 such communication is to educate your mechanics and other departments that feed information and data into the CASS about why these data are necessary, what you do with the data, and how this process benefits the operation.
7-3. COMMUNICATIONS WITH PERSONNEL OUTSIDE THE CASS. The FAA expects a good communication system to meet the objectives in this section. You must determine which system is best for your particular operation.
a. Outside Resources. Your CASS should provide for regular, structured communications within the CASS structure and between your CASS and any other resources involved in decision-making for you. Examples of these other resources include:
(1) Avionics and other shops,
(2) Cabin safety organization,
(3) Engineering department and FAA-approved reliability program department,
(4) FAA certificate management office (CMO) or PI,
(5) Flight operations,
(6) Ground operations,
(7) Inspection department,
(8) Internal Evaluation Program (IEP),
(9) Maintenance control,
(10) Maintenance operations,
(11) Manufacturers’ technical representatives,
(12) Purchasing,
(13) QA,
(14) Receiving inspection,
(15) Maintenance recordkeeping department,
(16) Safety program,
(17) Senior management,
(18) Stores department, and
(19) Training departments.
Par 7-3 Page 43 (and 44)
b. Feedback Loop. The communications mechanisms should include a feedback loop designed to ensure that any changes implemented as a result of corrective actions are functioning as intended and improving the process. You may accomplish this level of communications through a variety of means, including the following:
(1) Periodic (weekly, monthly, quarterly) statistical and narrative CASS reports on trends, findings, and the status of corrective actions.
(2) Periodic CASS meetings to discuss trends or specific problem areas. If you are a smaller certificate holder, where your relevant managers work in close proximity, such meetings might be informal but frequent, or if you are a larger certificate holder using more formal and structured meetings, you may designate specific boards or committees.
(3) CASS board or committee meetings, including senior management, possibly on a monthly or bimonthly basis. Even if meetings are somewhat informal, you should keep minutes.
c. Monthly Report. Typically, if you use a program incorporating statistical performance standards (alert-type programs), you develop a periodic (monthly) report with appropriate data displays, summarizing the previous month’s activity. To help evaluate the effectiveness of the total maintenance program, the report should cover all aircraft systems controlled by the FAA-approved reliability program. A certificate holder without an FAA-approved reliability program may find that using a similar report can enhance its CASS.
Par 8-1 Page 45 CHAPTER 8. HOW YOUR CASS DIFFERS FROM AND RELATES TO OTHER PROGRAMS
8-1. SUMMARY OF OTHER PROGRAMS. Your description of your CASS should identify other related programs where you participate and explain how your CASS relates to those programs or differs from them. Experience has shown that certain other programs are potential sources of information for your CASS. Some programs have been mistakenly assumed to be so similar to a CASS that you might neglect an important aspect of your CASS. Therefore, your CASS documentation should describe the relationship between your CASS and programs, such as the:
• FAA-approved reliability program,
• IEP,
• Safety program,
• Voluntary Disclosure Reporting Program (VDRP),
• Coordinating Agencies for Supplier’s Evaluation (C.A.S.E.),
• Aviation Safety Action Program (ASAP), and
• Aviation Safety Reporting Program.
8-2. DISCUSSION OF INDIVIDUAL PROGRAMS.
a. FAA-Approved Reliability Program. According to the current edition of AC 120-17, Maintenance Control by Reliability Methods, the FAA developed the concept of reliability control to maintain an acceptable level of reliability. The concept evolved based on FAA and airline efforts to develop more responsive methods of controlling scheduled maintenance without sacrificing safety or FAA regulatory responsibility. An FAA-approved reliability program includes systems for data collection and analysis, corrective action, statistical performance standards, data display and reporting, scheduled maintenance adjustments, and process changes.
AC 120-17 defines an acceptable level of reliability as maintaining failure rates below a predetermined value. However, AC 120-17 does not address modern-day, task-based scheduled maintenance programs.
(1) Typically, if you are a larger certificate holder, you will have an FAA-approved reliability program, but the operational statistical data collection and analysis requirements of a program that conforms to AC 120-17 usually exceed the resources or requirements of smaller and even most medium-sized operators, and generally are greater than what would be necessary for those operators’ CASS. However, if you do have an approved reliability program, you should incorporate it into your CASS as the means of performing operational data collection and analysis to monitor the effectiveness of your maintenance program. Your CASS procedures should describe how you will integrate the approved reliability program into your CASS. You should be aware that an FAA-approved reliability program cannot substitute for a CASS because the reliability program addresses only the scheduled maintenance element and does not include
Page 46 Par 8-2 the broader auditing surveillance and analysis of the full range of all 10 elements of your maintenance program, nor the complete processes for developing and implementing corrective actions. An FAA-approved reliability program should be a part of your CASS; however, it cannot replace or substitute for a CASS.
(2) This AC is not intended to describe FAA-approved reliability programs. However, CASS operational data collection needs are typically similar to, but much more comprehensive than, those of an approved reliability program. You should, within your CASS, establish a program similar to an FAA-approved reliability program for the purpose of collecting and analyzing operational data. However, you must ensure its operational data collection program meets the needs of your CASS.
(3) Within the aviation community, it is common to refer to “reliability,” in a generic sense, as meaning dispatch availability of equipment or in relation to equipment failure rates. If your CASS manual or document uses this terminology, you should distinguish whether the reference is to an FAA-approved reliability program or generic reliability.
b. IEP.
(1) An IEP is a voluntary program that you can use to provide measurement of your internal processes and procedures to assess whether they are adequate and functioning properly.
An IEP should be independent of all of your other programs and systems, and could be a useful tool to evaluate your CASS, as well as other systems or programs, such as your safety program.
An IEP is a very high-level review to provide information to your senior management as to how well critical programs, such as your CASS, are working. It is not a substitute for a CASS. An IEP is a broader, system evaluation program and is less “audit-oriented” than your CASS, although both use a system evaluation approach. An IEP would not focus on evaluating the maintenance program, because such a focus would duplicate what the CASS is supposed to do. If management believes they need to have additional evaluation of your maintenance program or certain elements of your maintenance program, they should direct the CASS to accomplish that work. An IEP should produce information to determine if the CASS is being performed and is effective.
(2) The current edition of AC 120-59, Air Carrier Internal Evaluation Programs, describes the IEP. You should not misunderstand the IEP as a program that replaces existing regulatory auditing requirements such as a CASS. Audits are a very minor part of an effective IEP.
c. Safety Program.
(1) According to 14 CFR part 119, § 119.65, if you are conducting operations under 14 CFR part 121, you must have a full-time Director of Safety (DOS) or equivalent position unless you ask for, and the FAA authorizes, a deviation from the management position requirements. Consistent with the objectives of part 119, the DOS is responsible for keeping your highest management officials fully informed about the safety status of your entire operation.
Your DOS should oversee a safety program that addresses the wide range of risks involved in operations in air transportation, including flight operations, maintenance operations, and ground
Par 8-2 Page 47 operations. The safety program should be a comprehensive program with a variety of elements, such as investigations of and a reporting system for accidents and incidents, safety audits and inspections, operational risk assessment, and trend analysis.
(2) If you conduct operations under 14 CFR part 135, the regulations do not require you to have a DOS position. Nonetheless, the FAA encourages you to designate a company management official or manager to monitor and evaluate your flight, maintenance, and ground safety practices, procedures, and programs and to keep your highest management official fully informed about the safety status of your entire operation.
d. VDRP. The current edition of AC 00-58, Voluntary Disclosure Reporting Program, provides guidance for you on procedures to use when voluntarily disclosing to the FAA violations of Federal aviation regulations. Your participation in the program may reveal important information regarding maintenance issues and lead to the development of comprehensive fixes relevant to the maintenance program that your CASS oversees.
(1) Under this program, you may voluntarily report violations of regulations that you discover and avoid certain enforcement consequences.
(2) While the regulations do not require that your CASS addresses disclosures made under the VDRP, the FAA recommends that you consider whether to include information from voluntary disclosures in your CASS in any fashion. For example, your CASS personnel may be the same personnel as those who handle voluntary disclosures. They may therefore be able to use “de-identified” information from voluntary disclosures to point to areas where additional auditing may be necessary. Your CASS personnel should be aware of comprehensive fixes developed in conjunction with the VDRP. These are, after all, precisely the types of systems or procedural modifications that an effective CASS is seeking to avoid adverse audit findings or unwanted operational performance.
e. C.A.S.E.
(1) C.A.S.E. is a cooperative effort within the airline industry to audit suppliers and vendors and to analyze, control, and determine the acceptability of vendors supplying parts and maintenance services to participating airlines. According to FAA Order 8900.1, Flight Standards Information Management System (FSIMS), the FAA has determined that the use of C.A.S.E.
audits satisfies some of the requirements of §§ 121.373 and 135.431. Operations specifications (OpSpecs) must authorize the use of the C.A.S.E. program to satisfy some of these requirements.
(2) If you participate in C.A.S.E., your CASS procedures should address whether or how the CASS will use C.A.S.E. audits and the basis for that decision. If the functions being audited depend on the specifics of your program, a C.A.S.E. audit would probably not suffice.
The important point is that you have a written policy and procedures for when and how to use C.A.S.E. audit results in your CASS.
f. ASAPs. The current edition of AC 120-66, Aviation Safety Action Program (ASAP), describes this program for the voluntary reporting of safety issues and events by employees, such as crewmembers and mechanics. ASAPs involve the collection, analysis, and retention of safety data that would otherwise be unobtainable. Such data can be important input to your CASS.
Page 48 Par 8-2
g. Aviation Safety Reporting Program. The current edition of AC 00-46, Aviation Safety Reporting Program, describes the Aviation Safety Reporting Program, which uses the National Aeronautics and Space Administration (NASA) as a third party to receive aviation safety reports. The Aviation Safety Reporting Program invites crewmembers, maintenance personnel, and others to report to NASA actual or potential discrepancies and deficiencies involving aviation safety. NASA designed and administers the Aviation Safety Reporting System to facilitate the program. These reports may help your CASS personnel identify areas of potential concern within their own company based on industry-wide trends or experiences identified by NASA.
Par 9-1 Page 49 CHAPTER 9. HOW TO DETERMINE IF YOUR CASS IS WORKING PROPERLY
9-1. WHY YOU SHOULD EVALUATE YOUR CASS.
a. Accomplishing its Function. As with any system or program, you should evaluate your CASS (that is, you should accomplish a process measurement) so that any personnel responsible for overseeing your CASS, such as your top management, may be confident that your CASS is accomplishing its function. Verifying that your CASS is working as intended is also a primary oversight task of your FAA PI.
b. Effective by Design. A common misconception is that a certificate holder can evaluate its CASS solely on the basis of the results of its maintenance program. That is, it is common to assume that if your aircraft are consistently airworthy, then your CASS must be doing its job.
However, this favorable result may occur for other reasons, such as the extraordinary diligence or memory of a few individuals. The purpose of your CASS is to ensure, with a system-oriented, structured approach that your maintenance program is functioning properly and is effective, consistently and by design rather than good fortune. You should not assume that good maintenance is synonymous with your CASS working properly.
c. Functioning Properly. Thus, personnel with CASS oversight responsibilities (including the FAA) require a different approach to determining if your CASS is indeed working properly. They need to know that you have complete CASS policies and procedures to monitor and evaluate your maintenance program, that it is carrying out these policies and procedures, and that they work. For example, to ensure that your CASS is functioning properly, a senior manager would not analyze component removal rates, but rather verify that your CASS was analyzing component removal rates, detecting trends as appropriate, and implementing corrective actions when necessary. You should have procedures—either in your CASS manual, referenced in your CASS manual, or contained in another document such as its IEP manual—for evaluating your CASS and informing top management as to the effectiveness of your CASS. The regulations not only require a maintenance program to meet many specific standards; they also require a system (CASS) to monitor that program.
9-2. STEPS TO EVALUATE YOUR CASS. The FAA expects you to develop your own method of evaluating that your CASS is working properly, including how you intend to measure whether you have allocated sufficient staffing and resources to the CASS.
a. System Safety Attributes. Determine that your CASS addresses applicable system safety attributes (responsibility, authority, procedures, controls, process measurement, and interfaces). If you have an IEP that follows this format, it would provide your senior management with an appropriate means of evaluating your CASS. That would be one way, but not the only way, to evaluate your CASS.
b. Indicators. The following questions may be useful in indicating whether you properly designed your CASS or that it is working as intended, although you may identify other indicators:
(1) Are your CASS personnel sufficiently independent of the areas they audit? Are they trained specifically in their CASS responsibilities?
Page 50 Par 9-2
(2) Have you allocated resources to your CASS sufficient to permit timely analysis of audits and data, as well as followup to corrective actions? Or are there delays in responding to findings and implementing corrective actions?
(3) Are your CASS personnel able to perform their duties in accordance with reasonable schedules?
(4) How many findings does your CASS produce, and what are the trends?
NOTE: CASS is supposed to produce findings, so absolute numbers—even high numbers of findings—are not necessarily a negative outcome; if combined with effective corrective actions and followup action, numerous CASS findings could be a positive indicator that the CASS is doing its job of detecting deficiencies and yielding appropriate, well-analyzed corrective actions. Trends are important, however. The same types of findings should not recur often once the CASS has addressed those findings.
(5) Have an unusually large number of unplanned maintenance events occurred within a specified time; for example, 21 days after a substantial inspection or other maintenance task? If so, does an investigation indicate that there are deficiencies in your maintenance program that your CASS should have identified and averted, or can the investigation attribute the anomaly to other factors?
(6) Does analysis indicate recurring problems in areas that you previously thought corrective actions had addressed?
(7) Are new problem areas coming to light? (This would be indicative of your CASS working to detect new issues.)
(8) Are your CASS corrective actions resulting in new problem areas, reflecting insufficient root cause, risk, or system analysis before implementing these corrective actions?
(9) How do CASS results compare with outside audit results, such as those conducted by the DOD or the FAA?
(10) Have regulatory violations occurred that your CASS might have averted?
(11) Does your senior management understand and support the CASS?
(12) Are your CASS auditors and analysts encouraged to consider all possible aspects of an issue, including the role of your senior management, when developing corrective actions?
(13) Has your CASS evolved into a punitive process (the blame culture) with the effect of discouraging open participation of company personnel, or do your personnel cooperate actively and offer input to the CASS?
(14) Are all areas of your maintenance program undergoing CASS audits in accordance with a schedule based on a process of risk assessment and prioritization?
Par 9-2 Page 51 (and 52)
(15) Do the depth and quality of the audit reports and analysis reflect that your personnel have sufficient time and resources?
c. Senior Management Review. Your senior management should review CASS issues on a monthly or bimonthly basis. You may hold meetings of this sort, possibly of CASS or maintenance management committees or boards, to discuss findings, analysis, and progress of corrective actions. These meetings may address statistical data and trends, depending on your size, type of operation, and capability of producing comprehensive statistical reports.