Par 8-1 Page 45 CHAPTER 8. HOW YOUR CASS DIFFERS FROM AND RELATES TO OTHER PROGRAMS
8-1. SUMMARY OF OTHER PROGRAMS. Your description of your CASS should identify other related programs where you participate and explain how your CASS relates to those programs or differs from them. Experience has shown that certain other programs are potential sources of information for your CASS. Some programs have been mistakenly assumed to be so similar to a CASS that you might neglect an important aspect of your CASS. Therefore, your CASS documentation should describe the relationship between your CASS and programs, such as the:
• FAA-approved reliability program,
• IEP,
• Safety program,
• Voluntary Disclosure Reporting Program (VDRP),
• Coordinating Agencies for Supplier’s Evaluation (C.A.S.E.),
• Aviation Safety Action Program (ASAP), and
• Aviation Safety Reporting Program.
8-2. DISCUSSION OF INDIVIDUAL PROGRAMS.
a. FAA-Approved Reliability Program. According to the current edition of AC 120-17, Maintenance Control by Reliability Methods, the FAA developed the concept of reliability control to maintain an acceptable level of reliability. The concept evolved based on FAA and airline efforts to develop more responsive methods of controlling scheduled maintenance without sacrificing safety or FAA regulatory responsibility. An FAA-approved reliability program includes systems for data collection and analysis, corrective action, statistical performance standards, data display and reporting, scheduled maintenance adjustments, and process changes.
AC 120-17 defines an acceptable level of reliability as maintaining failure rates below a predetermined value. However, AC 120-17 does not address modern-day, task-based scheduled maintenance programs.
(1) Typically, if you are a larger certificate holder, you will have an FAA-approved reliability program, but the operational statistical data collection and analysis requirements of a program that conforms to AC 120-17 usually exceed the resources or requirements of smaller and even most medium-sized operators, and generally are greater than what would be necessary for those operators’ CASS. However, if you do have an approved reliability program, you should incorporate it into your CASS as the means of performing operational data collection and analysis to monitor the effectiveness of your maintenance program. Your CASS procedures should describe how you will integrate the approved reliability program into your CASS. You should be aware that an FAA-approved reliability program cannot substitute for a CASS because the reliability program addresses only the scheduled maintenance element and does not include
Page 46 Par 8-2 the broader auditing surveillance and analysis of the full range of all 10 elements of your maintenance program, nor the complete processes for developing and implementing corrective actions. An FAA-approved reliability program should be a part of your CASS; however, it cannot replace or substitute for a CASS.
(2) This AC is not intended to describe FAA-approved reliability programs. However, CASS operational data collection needs are typically similar to, but much more comprehensive than, those of an approved reliability program. You should, within your CASS, establish a program similar to an FAA-approved reliability program for the purpose of collecting and analyzing operational data. However, you must ensure its operational data collection program meets the needs of your CASS.
(3) Within the aviation community, it is common to refer to “reliability,” in a generic sense, as meaning dispatch availability of equipment or in relation to equipment failure rates. If your CASS manual or document uses this terminology, you should distinguish whether the reference is to an FAA-approved reliability program or generic reliability.
b. IEP.
(1) An IEP is a voluntary program that you can use to provide measurement of your internal processes and procedures to assess whether they are adequate and functioning properly.
An IEP should be independent of all of your other programs and systems, and could be a useful tool to evaluate your CASS, as well as other systems or programs, such as your safety program.
An IEP is a very high-level review to provide information to your senior management as to how well critical programs, such as your CASS, are working. It is not a substitute for a CASS. An IEP is a broader, system evaluation program and is less “audit-oriented” than your CASS, although both use a system evaluation approach. An IEP would not focus on evaluating the maintenance program, because such a focus would duplicate what the CASS is supposed to do. If management believes they need to have additional evaluation of your maintenance program or certain elements of your maintenance program, they should direct the CASS to accomplish that work. An IEP should produce information to determine if the CASS is being performed and is effective.
(2) The current edition of AC 120-59, Air Carrier Internal Evaluation Programs, describes the IEP. You should not misunderstand the IEP as a program that replaces existing regulatory auditing requirements such as a CASS. Audits are a very minor part of an effective IEP.
c. Safety Program.
(1) According to 14 CFR part 119, § 119.65, if you are conducting operations under 14 CFR part 121, you must have a full-time Director of Safety (DOS) or equivalent position unless you ask for, and the FAA authorizes, a deviation from the management position requirements. Consistent with the objectives of part 119, the DOS is responsible for keeping your highest management officials fully informed about the safety status of your entire operation.
Your DOS should oversee a safety program that addresses the wide range of risks involved in operations in air transportation, including flight operations, maintenance operations, and ground
Par 8-2 Page 47 operations. The safety program should be a comprehensive program with a variety of elements, such as investigations of and a reporting system for accidents and incidents, safety audits and inspections, operational risk assessment, and trend analysis.
(2) If you conduct operations under 14 CFR part 135, the regulations do not require you to have a DOS position. Nonetheless, the FAA encourages you to designate a company management official or manager to monitor and evaluate your flight, maintenance, and ground safety practices, procedures, and programs and to keep your highest management official fully informed about the safety status of your entire operation.
d. VDRP. The current edition of AC 00-58, Voluntary Disclosure Reporting Program, provides guidance for you on procedures to use when voluntarily disclosing to the FAA violations of Federal aviation regulations. Your participation in the program may reveal important information regarding maintenance issues and lead to the development of comprehensive fixes relevant to the maintenance program that your CASS oversees.
(1) Under this program, you may voluntarily report violations of regulations that you discover and avoid certain enforcement consequences.
(2) While the regulations do not require that your CASS addresses disclosures made under the VDRP, the FAA recommends that you consider whether to include information from voluntary disclosures in your CASS in any fashion. For example, your CASS personnel may be the same personnel as those who handle voluntary disclosures. They may therefore be able to use “de-identified” information from voluntary disclosures to point to areas where additional auditing may be necessary. Your CASS personnel should be aware of comprehensive fixes developed in conjunction with the VDRP. These are, after all, precisely the types of systems or procedural modifications that an effective CASS is seeking to avoid adverse audit findings or unwanted operational performance.
e. C.A.S.E.
(1) C.A.S.E. is a cooperative effort within the airline industry to audit suppliers and vendors and to analyze, control, and determine the acceptability of vendors supplying parts and maintenance services to participating airlines. According to FAA Order 8900.1, Flight Standards Information Management System (FSIMS), the FAA has determined that the use of C.A.S.E.
audits satisfies some of the requirements of §§ 121.373 and 135.431. Operations specifications (OpSpecs) must authorize the use of the C.A.S.E. program to satisfy some of these requirements.
(2) If you participate in C.A.S.E., your CASS procedures should address whether or how the CASS will use C.A.S.E. audits and the basis for that decision. If the functions being audited depend on the specifics of your program, a C.A.S.E. audit would probably not suffice.
The important point is that you have a written policy and procedures for when and how to use C.A.S.E. audit results in your CASS.
f. ASAPs. The current edition of AC 120-66, Aviation Safety Action Program (ASAP), describes this program for the voluntary reporting of safety issues and events by employees, such as crewmembers and mechanics. ASAPs involve the collection, analysis, and retention of safety data that would otherwise be unobtainable. Such data can be important input to your CASS.
Page 48 Par 8-2
g. Aviation Safety Reporting Program. The current edition of AC 00-46, Aviation Safety Reporting Program, describes the Aviation Safety Reporting Program, which uses the National Aeronautics and Space Administration (NASA) as a third party to receive aviation safety reports. The Aviation Safety Reporting Program invites crewmembers, maintenance personnel, and others to report to NASA actual or potential discrepancies and deficiencies involving aviation safety. NASA designed and administers the Aviation Safety Reporting System to facilitate the program. These reports may help your CASS personnel identify areas of potential concern within their own company based on industry-wide trends or experiences identified by NASA.
Par 9-1 Page 49 CHAPTER 9. HOW TO DETERMINE IF YOUR CASS IS WORKING PROPERLY
9-1. WHY YOU SHOULD EVALUATE YOUR CASS.
a. Accomplishing its Function. As with any system or program, you should evaluate your CASS (that is, you should accomplish a process measurement) so that any personnel responsible for overseeing your CASS, such as your top management, may be confident that your CASS is accomplishing its function. Verifying that your CASS is working as intended is also a primary oversight task of your FAA PI.
b. Effective by Design. A common misconception is that a certificate holder can evaluate its CASS solely on the basis of the results of its maintenance program. That is, it is common to assume that if your aircraft are consistently airworthy, then your CASS must be doing its job.
However, this favorable result may occur for other reasons, such as the extraordinary diligence or memory of a few individuals. The purpose of your CASS is to ensure, with a system-oriented, structured approach that your maintenance program is functioning properly and is effective, consistently and by design rather than good fortune. You should not assume that good maintenance is synonymous with your CASS working properly.
c. Functioning Properly. Thus, personnel with CASS oversight responsibilities (including the FAA) require a different approach to determining if your CASS is indeed working properly. They need to know that you have complete CASS policies and procedures to monitor and evaluate your maintenance program, that it is carrying out these policies and procedures, and that they work. For example, to ensure that your CASS is functioning properly, a senior manager would not analyze component removal rates, but rather verify that your CASS was analyzing component removal rates, detecting trends as appropriate, and implementing corrective actions when necessary. You should have procedures—either in your CASS manual, referenced in your CASS manual, or contained in another document such as its IEP manual—for evaluating your CASS and informing top management as to the effectiveness of your CASS. The regulations not only require a maintenance program to meet many specific standards; they also require a system (CASS) to monitor that program.
9-2. STEPS TO EVALUATE YOUR CASS. The FAA expects you to develop your own method of evaluating that your CASS is working properly, including how you intend to measure whether you have allocated sufficient staffing and resources to the CASS.
a. System Safety Attributes. Determine that your CASS addresses applicable system safety attributes (responsibility, authority, procedures, controls, process measurement, and interfaces). If you have an IEP that follows this format, it would provide your senior management with an appropriate means of evaluating your CASS. That would be one way, but not the only way, to evaluate your CASS.
b. Indicators. The following questions may be useful in indicating whether you properly designed your CASS or that it is working as intended, although you may identify other indicators:
(1) Are your CASS personnel sufficiently independent of the areas they audit? Are they trained specifically in their CASS responsibilities?
Page 50 Par 9-2
(2) Have you allocated resources to your CASS sufficient to permit timely analysis of audits and data, as well as followup to corrective actions? Or are there delays in responding to findings and implementing corrective actions?
(3) Are your CASS personnel able to perform their duties in accordance with reasonable schedules?
(4) How many findings does your CASS produce, and what are the trends?
NOTE: CASS is supposed to produce findings, so absolute numbers—even high numbers of findings—are not necessarily a negative outcome; if combined with effective corrective actions and followup action, numerous CASS findings could be a positive indicator that the CASS is doing its job of detecting deficiencies and yielding appropriate, well-analyzed corrective actions. Trends are important, however. The same types of findings should not recur often once the CASS has addressed those findings.
(5) Have an unusually large number of unplanned maintenance events occurred within a specified time; for example, 21 days after a substantial inspection or other maintenance task? If so, does an investigation indicate that there are deficiencies in your maintenance program that your CASS should have identified and averted, or can the investigation attribute the anomaly to other factors?
(6) Does analysis indicate recurring problems in areas that you previously thought corrective actions had addressed?
(7) Are new problem areas coming to light? (This would be indicative of your CASS working to detect new issues.)
(8) Are your CASS corrective actions resulting in new problem areas, reflecting insufficient root cause, risk, or system analysis before implementing these corrective actions?
(9) How do CASS results compare with outside audit results, such as those conducted by the DOD or the FAA?
(10) Have regulatory violations occurred that your CASS might have averted?
(11) Does your senior management understand and support the CASS?
(12) Are your CASS auditors and analysts encouraged to consider all possible aspects of an issue, including the role of your senior management, when developing corrective actions?
(13) Has your CASS evolved into a punitive process (the blame culture) with the effect of discouraging open participation of company personnel, or do your personnel cooperate actively and offer input to the CASS?
(14) Are all areas of your maintenance program undergoing CASS audits in accordance with a schedule based on a process of risk assessment and prioritization?
Par 9-2 Page 51 (and 52)
(15) Do the depth and quality of the audit reports and analysis reflect that your personnel have sufficient time and resources?
c. Senior Management Review. Your senior management should review CASS issues on a monthly or bimonthly basis. You may hold meetings of this sort, possibly of CASS or maintenance management committees or boards, to discuss findings, analysis, and progress of corrective actions. These meetings may address statistical data and trends, depending on your size, type of operation, and capability of producing comprehensive statistical reports.
Par 10-1 Page 53 (and 54) CHAPTER 10. THE ROLE OF THE FAA IN RELATION TO YOUR CASS
10-1. THE FAA’S ROLE, IN GENERAL. As with any applicable aviation regulation, you must understand that you hold the primary responsibility for regulatory compliance, not the FAA. The FAA’s role is not to design the CASS for each certificate holder, but to ensure that you have satisfactory policies and procedures in place. For example, the FAA will not provide you with an extensive list of data that you should collect and analyze because of the wide variation in the nature and scope of airline operations. However, the FAA expects you to demonstrate that your CASS includes a process for selecting and periodically reevaluating data sets appropriate for your operation and appropriate for monitoring your own unique maintenance program. The FAA also expects you to have a logical and current reason for selecting the data set it collects.
10-2. THE FAA PI’S ROLE. The term FAA PI, as used in this AC, is generally intended to mean the principal maintenance inspector (PMI). However, the principal avionics inspector (PAI) also plays an important role in the oversight of the certificate holder’s CASS and shares many of the same responsibilities as the PMI. The FAA PI:
a. Assists in Developing Your CASS. Works with you in developing your CASS, providing guidance and ensuring that your CASS meets the requirements of the regulations.
b. Reviews CASS Records. Reviews your CASS records, such as results of audits and analysis, corrective action, and followup. Therefore, it would be useful for you and your PI to have a common understanding of how long you will retain these records, not only in terms of usefulness to your CASS, but also to facilitate review by your PI to help him or her determine that you are properly executing your CASS.
c. Meets with Maintenance and QA Management. Meets on a regular basis with managers in your maintenance and QA areas, particularly with the person responsible for your CASS. Your CASS should provide one of the best barometers of the overall status of your maintenance program, how you or others are using it, whether it is effective, and whether a change is being effected as necessary.
d. Meets with Senior Management. Meets with your senior management to determine how well they understand and support the CASS.
Par 11-1 Page 55 (and 56) CHAPTER 11. ADMINISTRATIVE INFORMATION
11-1. HOW TO OBTAIN THIS AND OTHER FAA PUBLICATIONS.
a. AC 00.2 and AC 00-44. The current edition of AC 00.2, Advisory Circular Checklist and Status of Other FAA Publications, contains a listing of all ACs. The current edition of AC 00-44, Status of Federal Aviation Regulations, contains a listing of the Code of Federal Regulations (CFR) and current prices.
b. ACs Online. You can get copies of this and other ACs online at http://www.faa.gov/regulations_policies/advisory_circulars.
c. Regulations Online. You can also obtain a copy of current regulations online at http://www.gpoaccess.gov/ecfr.
d. CFR and ACs by Mail. You can obtain the CFR and ACs for a fee from the following address:
Superintendent of Documents P.O. Box 371954 Pittsburgh, PA 15250-7954
e. Mailing List. For placement on our mailing list of free ACs, contact:
U.S. Department of Transportation Subsequent Distribution Office SVC-121.23 Ardmore East Business Center 3341Q 75th Avenue Landover, MD 20785
f. FAA Web Site. Our Web site is located at http://www.faa.gov. The FSIMS Web site is located at http://fsims.faa.gov.
Appendix 1 APPENDIX 1. SAMPLE CASS FOR A LARGE CERTIFICATE HOLDER Type of Certificate Holder Fleet composition 250 turbojet airplanes; B-737, B-757, A-320.
Number of maintenance base and line stations Base station and 25 line stations.
Proportion of maintenance contracted to third parties All letter checks, overhauls, and major maintenance performed in-house. Some line maintenance, parts work, and off-wing engine work is contracted.
Scheduled or on-demand Scheduled (Title 14 of the Code of Federal Regulations (14 CFR) part 121).
Size and structure of the maintenance organization Engineering and maintenance organizations include extensive engineering capability, quality assurance (QA) department, full range of shops for support, components, electronics, engines, etc.
a. Continuing Analysis and Surveillance System (CASS) Management and Planning.
b. General Priority. This certificate holder prioritizes in the following general manner:
(1) Safe operations (air and ground).
(2) Detect and prevent noncompliance.
(3) Improve operating efficiency.
c. CASS Written Procedures (A System Safety Attribute: Procedures). A separate CASS manual contains the CASS written procedures. The CASS appendix is detailed, including specific procedures for root cause and systems analysis and discussion of how to address human factors.
d. CASS in the Certificate Holder Organization. The senior vice president of engineering and maintenance (14 CFR part 119, § 119.65 Director of Maintenance (DOM)) has overall authority and responsibility for this air carrier’s maintenance program, including the CASS. The CASS specifically appears on the functional organizational flowchart.
e. Authority for CASS (A System Safety Attribute: Authority). The director of QA holds the direct authority for the CASS.
f. Responsibility for CASS (A System Safety Attribute: Responsibility). The CASS board has direct responsibility for the CASS. The manager of the CASS chairs the board and reports to the director of QA.
Appendix 1
g. Policy for CASS Auditor/Analyst Independence from Production (A System Safety Attribute: Controls). CASS auditors/analysts are in a separate department under QA.
Operational data collection and analysis are assigned to the reliability group (within the certificate holder’s Federal Aviation Administration (FAA)-approved reliability program) in the engineering department. The reliability group reports CASS information directly to the CASS board.
h. Policy Regarding Personnel Actions Resulting from CASS Findings/Results. The certificate holder’s policy is consistent with its participation in Aviation Safety Action Programs (ASAP) and the Voluntary Disclosure Reporting Program (VDRP). Inadvertent errors do not lead to disciplinary action or FAA enforcement action. Full reporting and disclosure are encouraged to facilitate system corrections.
i. Surveillance and Analysis of Performance of Maintenance Program.
Audits Responsibility Manager of CASS.
Prioritization CASS board develops a surveillance plan based on risk assessment.
Cycles The CASS board assigned each area an onsite audit accomplished semiannually to every 5 years, depending on the risk assessment. Also depending on the risk assessment, some areas never receive an onsite audit assignment. The CASS board updates the annual audit plan quarterly.
Scope CASS covers all internal and third-party areas of maintenance.
Coordinating Agencies for Supplier’s Evaluation (C.A.S.E.) audits may indicate the need for followup audits or may be integrated into the CASS.
Process Full-time CASS auditors use checklists and conduct annual interviews of personnel at all levels of maintenance to uncover concerns or latent problems. The CASS department receives reports of all maintenance-related events, such as rejected takeoffs, for analysis and use in risk assessment for audits. Schedule special audits as needed.
The CASS also reviews self-audits from all departments. Some vendors conduct their audits by document reviews, written questionnaires, telephone followup, or combinations of these. Followup is required if preliminary results raise concerns.
Flow Report the initial results to the manager of CASS. The CASS board reviews summaries of all findings/initial analyses and details of issues as deemed necessary by the manager of CASS.
Appendix 1 Analysis Responsibility Auditors conduct preliminary analysis based on specific experience and training and internally developed guidelines that channel analysis to system root causes.
Perform preliminary root cause analysis Auditor/analyst.
Classify hazards/ perform risk assessment CASS board and analysts.
Flow The CASS board transmits results to the affected department, which assigns personnel to participate on the corrective action team under direction of the CASS board.
j. Surveillance and Analysis of Effectiveness of Maintenance Program.
Operational Data Collection Responsibility Manager of CASS.
Prioritization The reliability group determines priority based on risk assessment.
Scope Extensive. Ranges from pilot reports, Engine Condition Monitoring (ECM), mechanical delays, and teardown reports to data from special authorizations such as category II/III, extended range operation with two-engine airplanes, and operations in Reduced Vertical Separation Minimum (RVSM) airspace and minimum navigation performance specifications (MNPS) airspace.
Process In accordance with the certificate holder’s FAA-approved reliability program. Data collection is oriented toward detecting trends, positive or negative, before the occurrence of events. However, data collection (and analysis) may vary based on maintenance-related events. The CASS board formally reviews the list of operational data sets collected by the CASS board every 2 years to determine if it needs to be adjusted. The initial list is based on the current edition of Advisory Circular (AC) 120-17, Maintenance Control by Reliability Methods, and CASS board determinations.
Flow The reliability group, although located within the engineering department, has a direct reporting relationship to the manager of CASS.
Analysis of Operational Data Responsibility Reliability group.
Prioritization Reliability board.
Process Technical experts within the reliability group perform the analysis and make preliminary determination of possible root causes.
Flow Results are reported to the manager of CASS and the CASS board.