Par 9-1 Page 49 CHAPTER 9. HOW TO DETERMINE IF YOUR CASS IS WORKING PROPERLY
9-1. WHY YOU SHOULD EVALUATE YOUR CASS.
a. Accomplishing its Function. As with any system or program, you should evaluate your CASS (that is, you should accomplish a process measurement) so that any personnel responsible for overseeing your CASS, such as your top management, may be confident that your CASS is accomplishing its function. Verifying that your CASS is working as intended is also a primary oversight task of your FAA PI.
b. Effective by Design. A common misconception is that a certificate holder can evaluate its CASS solely on the basis of the results of its maintenance program. That is, it is common to assume that if your aircraft are consistently airworthy, then your CASS must be doing its job.
However, this favorable result may occur for other reasons, such as the extraordinary diligence or memory of a few individuals. The purpose of your CASS is to ensure, with a system-oriented, structured approach that your maintenance program is functioning properly and is effective, consistently and by design rather than good fortune. You should not assume that good maintenance is synonymous with your CASS working properly.
c. Functioning Properly. Thus, personnel with CASS oversight responsibilities (including the FAA) require a different approach to determining if your CASS is indeed working properly. They need to know that you have complete CASS policies and procedures to monitor and evaluate your maintenance program, that it is carrying out these policies and procedures, and that they work. For example, to ensure that your CASS is functioning properly, a senior manager would not analyze component removal rates, but rather verify that your CASS was analyzing component removal rates, detecting trends as appropriate, and implementing corrective actions when necessary. You should have procedures—either in your CASS manual, referenced in your CASS manual, or contained in another document such as its IEP manual—for evaluating your CASS and informing top management as to the effectiveness of your CASS. The regulations not only require a maintenance program to meet many specific standards; they also require a system (CASS) to monitor that program.
9-2. STEPS TO EVALUATE YOUR CASS. The FAA expects you to develop your own method of evaluating that your CASS is working properly, including how you intend to measure whether you have allocated sufficient staffing and resources to the CASS.
a. System Safety Attributes. Determine that your CASS addresses applicable system safety attributes (responsibility, authority, procedures, controls, process measurement, and interfaces). If you have an IEP that follows this format, it would provide your senior management with an appropriate means of evaluating your CASS. That would be one way, but not the only way, to evaluate your CASS.
b. Indicators. The following questions may be useful in indicating whether you properly designed your CASS or that it is working as intended, although you may identify other indicators:
(1) Are your CASS personnel sufficiently independent of the areas they audit? Are they trained specifically in their CASS responsibilities?
Page 50 Par 9-2
(2) Have you allocated resources to your CASS sufficient to permit timely analysis of audits and data, as well as followup to corrective actions? Or are there delays in responding to findings and implementing corrective actions?
(3) Are your CASS personnel able to perform their duties in accordance with reasonable schedules?
(4) How many findings does your CASS produce, and what are the trends?
NOTE: CASS is supposed to produce findings, so absolute numbers—even high numbers of findings—are not necessarily a negative outcome; if combined with effective corrective actions and followup action, numerous CASS findings could be a positive indicator that the CASS is doing its job of detecting deficiencies and yielding appropriate, well-analyzed corrective actions. Trends are important, however. The same types of findings should not recur often once the CASS has addressed those findings.
(5) Have an unusually large number of unplanned maintenance events occurred within a specified time; for example, 21 days after a substantial inspection or other maintenance task? If so, does an investigation indicate that there are deficiencies in your maintenance program that your CASS should have identified and averted, or can the investigation attribute the anomaly to other factors?
(6) Does analysis indicate recurring problems in areas that you previously thought corrective actions had addressed?
(7) Are new problem areas coming to light? (This would be indicative of your CASS working to detect new issues.)
(8) Are your CASS corrective actions resulting in new problem areas, reflecting insufficient root cause, risk, or system analysis before implementing these corrective actions?
(9) How do CASS results compare with outside audit results, such as those conducted by the DOD or the FAA?
(10) Have regulatory violations occurred that your CASS might have averted?
(11) Does your senior management understand and support the CASS?
(12) Are your CASS auditors and analysts encouraged to consider all possible aspects of an issue, including the role of your senior management, when developing corrective actions?
(13) Has your CASS evolved into a punitive process (the blame culture) with the effect of discouraging open participation of company personnel, or do your personnel cooperate actively and offer input to the CASS?
(14) Are all areas of your maintenance program undergoing CASS audits in accordance with a schedule based on a process of risk assessment and prioritization?
Par 9-2 Page 51 (and 52)
(15) Do the depth and quality of the audit reports and analysis reflect that your personnel have sufficient time and resources?
c. Senior Management Review. Your senior management should review CASS issues on a monthly or bimonthly basis. You may hold meetings of this sort, possibly of CASS or maintenance management committees or boards, to discuss findings, analysis, and progress of corrective actions. These meetings may address statistical data and trends, depending on your size, type of operation, and capability of producing comprehensive statistical reports.
Par 10-1 Page 53 (and 54) CHAPTER 10. THE ROLE OF THE FAA IN RELATION TO YOUR CASS
10-1. THE FAA’S ROLE, IN GENERAL. As with any applicable aviation regulation, you must understand that you hold the primary responsibility for regulatory compliance, not the FAA. The FAA’s role is not to design the CASS for each certificate holder, but to ensure that you have satisfactory policies and procedures in place. For example, the FAA will not provide you with an extensive list of data that you should collect and analyze because of the wide variation in the nature and scope of airline operations. However, the FAA expects you to demonstrate that your CASS includes a process for selecting and periodically reevaluating data sets appropriate for your operation and appropriate for monitoring your own unique maintenance program. The FAA also expects you to have a logical and current reason for selecting the data set it collects.
10-2. THE FAA PI’S ROLE. The term FAA PI, as used in this AC, is generally intended to mean the principal maintenance inspector (PMI). However, the principal avionics inspector (PAI) also plays an important role in the oversight of the certificate holder’s CASS and shares many of the same responsibilities as the PMI. The FAA PI:
a. Assists in Developing Your CASS. Works with you in developing your CASS, providing guidance and ensuring that your CASS meets the requirements of the regulations.
b. Reviews CASS Records. Reviews your CASS records, such as results of audits and analysis, corrective action, and followup. Therefore, it would be useful for you and your PI to have a common understanding of how long you will retain these records, not only in terms of usefulness to your CASS, but also to facilitate review by your PI to help him or her determine that you are properly executing your CASS.
c. Meets with Maintenance and QA Management. Meets on a regular basis with managers in your maintenance and QA areas, particularly with the person responsible for your CASS. Your CASS should provide one of the best barometers of the overall status of your maintenance program, how you or others are using it, whether it is effective, and whether a change is being effected as necessary.
d. Meets with Senior Management. Meets with your senior management to determine how well they understand and support the CASS.
Par 11-1 Page 55 (and 56) CHAPTER 11. ADMINISTRATIVE INFORMATION
11-1. HOW TO OBTAIN THIS AND OTHER FAA PUBLICATIONS.
a. AC 00.2 and AC 00-44. The current edition of AC 00.2, Advisory Circular Checklist and Status of Other FAA Publications, contains a listing of all ACs. The current edition of AC 00-44, Status of Federal Aviation Regulations, contains a listing of the Code of Federal Regulations (CFR) and current prices.
b. ACs Online. You can get copies of this and other ACs online at http://www.faa.gov/regulations_policies/advisory_circulars.
c. Regulations Online. You can also obtain a copy of current regulations online at http://www.gpoaccess.gov/ecfr.
d. CFR and ACs by Mail. You can obtain the CFR and ACs for a fee from the following address:
Superintendent of Documents P.O. Box 371954 Pittsburgh, PA 15250-7954
e. Mailing List. For placement on our mailing list of free ACs, contact:
U.S. Department of Transportation Subsequent Distribution Office SVC-121.23 Ardmore East Business Center 3341Q 75th Avenue Landover, MD 20785
f. FAA Web Site. Our Web site is located at http://www.faa.gov. The FSIMS Web site is located at http://fsims.faa.gov.
Appendix 1 APPENDIX 1. SAMPLE CASS FOR A LARGE CERTIFICATE HOLDER Type of Certificate Holder Fleet composition 250 turbojet airplanes; B-737, B-757, A-320.
Number of maintenance base and line stations Base station and 25 line stations.
Proportion of maintenance contracted to third parties All letter checks, overhauls, and major maintenance performed in-house. Some line maintenance, parts work, and off-wing engine work is contracted.
Scheduled or on-demand Scheduled (Title 14 of the Code of Federal Regulations (14 CFR) part 121).
Size and structure of the maintenance organization Engineering and maintenance organizations include extensive engineering capability, quality assurance (QA) department, full range of shops for support, components, electronics, engines, etc.
a. Continuing Analysis and Surveillance System (CASS) Management and Planning.
b. General Priority. This certificate holder prioritizes in the following general manner:
(1) Safe operations (air and ground).
(2) Detect and prevent noncompliance.
(3) Improve operating efficiency.
c. CASS Written Procedures (A System Safety Attribute: Procedures). A separate CASS manual contains the CASS written procedures. The CASS appendix is detailed, including specific procedures for root cause and systems analysis and discussion of how to address human factors.
d. CASS in the Certificate Holder Organization. The senior vice president of engineering and maintenance (14 CFR part 119, § 119.65 Director of Maintenance (DOM)) has overall authority and responsibility for this air carrier’s maintenance program, including the CASS. The CASS specifically appears on the functional organizational flowchart.
e. Authority for CASS (A System Safety Attribute: Authority). The director of QA holds the direct authority for the CASS.
f. Responsibility for CASS (A System Safety Attribute: Responsibility). The CASS board has direct responsibility for the CASS. The manager of the CASS chairs the board and reports to the director of QA.
Appendix 1
g. Policy for CASS Auditor/Analyst Independence from Production (A System Safety Attribute: Controls). CASS auditors/analysts are in a separate department under QA.
Operational data collection and analysis are assigned to the reliability group (within the certificate holder’s Federal Aviation Administration (FAA)-approved reliability program) in the engineering department. The reliability group reports CASS information directly to the CASS board.
h. Policy Regarding Personnel Actions Resulting from CASS Findings/Results. The certificate holder’s policy is consistent with its participation in Aviation Safety Action Programs (ASAP) and the Voluntary Disclosure Reporting Program (VDRP). Inadvertent errors do not lead to disciplinary action or FAA enforcement action. Full reporting and disclosure are encouraged to facilitate system corrections.
i. Surveillance and Analysis of Performance of Maintenance Program.
Audits Responsibility Manager of CASS.
Prioritization CASS board develops a surveillance plan based on risk assessment.
Cycles The CASS board assigned each area an onsite audit accomplished semiannually to every 5 years, depending on the risk assessment. Also depending on the risk assessment, some areas never receive an onsite audit assignment. The CASS board updates the annual audit plan quarterly.
Scope CASS covers all internal and third-party areas of maintenance.
Coordinating Agencies for Supplier’s Evaluation (C.A.S.E.) audits may indicate the need for followup audits or may be integrated into the CASS.
Process Full-time CASS auditors use checklists and conduct annual interviews of personnel at all levels of maintenance to uncover concerns or latent problems. The CASS department receives reports of all maintenance-related events, such as rejected takeoffs, for analysis and use in risk assessment for audits. Schedule special audits as needed.
The CASS also reviews self-audits from all departments. Some vendors conduct their audits by document reviews, written questionnaires, telephone followup, or combinations of these. Followup is required if preliminary results raise concerns.
Flow Report the initial results to the manager of CASS. The CASS board reviews summaries of all findings/initial analyses and details of issues as deemed necessary by the manager of CASS.
Appendix 1 Analysis Responsibility Auditors conduct preliminary analysis based on specific experience and training and internally developed guidelines that channel analysis to system root causes.
Perform preliminary root cause analysis Auditor/analyst.
Classify hazards/ perform risk assessment CASS board and analysts.
Flow The CASS board transmits results to the affected department, which assigns personnel to participate on the corrective action team under direction of the CASS board.
j. Surveillance and Analysis of Effectiveness of Maintenance Program.
Operational Data Collection Responsibility Manager of CASS.
Prioritization The reliability group determines priority based on risk assessment.
Scope Extensive. Ranges from pilot reports, Engine Condition Monitoring (ECM), mechanical delays, and teardown reports to data from special authorizations such as category II/III, extended range operation with two-engine airplanes, and operations in Reduced Vertical Separation Minimum (RVSM) airspace and minimum navigation performance specifications (MNPS) airspace.
Process In accordance with the certificate holder’s FAA-approved reliability program. Data collection is oriented toward detecting trends, positive or negative, before the occurrence of events. However, data collection (and analysis) may vary based on maintenance-related events. The CASS board formally reviews the list of operational data sets collected by the CASS board every 2 years to determine if it needs to be adjusted. The initial list is based on the current edition of Advisory Circular (AC) 120-17, Maintenance Control by Reliability Methods, and CASS board determinations.
Flow The reliability group, although located within the engineering department, has a direct reporting relationship to the manager of CASS.
Analysis of Operational Data Responsibility Reliability group.
Prioritization Reliability board.
Process Technical experts within the reliability group perform the analysis and make preliminary determination of possible root causes.
Flow Results are reported to the manager of CASS and the CASS board.
Appendix 1
k. Corrective Action.
Final Root Cause Analysis (RCA) Responsibility Manager(s) of technical area(s) affected.
Procedures The director of CASS transmits preliminary analysis results to the manager of the affected department, who designates technical personnel to coordinate final RCA with the auditor/analyst. The CASS auditor/analyst oversees the process and ensures that the formal RCA process, including human factors and systems analysis, is followed.
Use of specific analytical systems Internally developed analytical process and industry tools.
Flow The auditor/analyst and technical department develop a joint final report and submit it to their respective managers.
Determination of Corrective Action Options Responsibility The CASS board appoints a corrective action team to include the CASS auditor/analyst, reliability group, technical area(s) affected, and related areas potentially affected (for example, flight operations).
Procedures The team leader is from the technical area affected. Perform risk assessment of the problem and develop corrective action options. The CASS auditor/analyst or reliability group representative does not propose corrective actions but reviews options for systems considerations and relevance to RCA.
Flow The manager(s) of the affected area(s), working with the team, transmits options to the CASS board.
Selection of Corrective Action and Corrective Action Plan Responsibility CASS board.
Procedures Decision based on a priority for safety and regulatory compliance. Risk assessment is the basic tool to support the decision. CASS personnel do not participate in the development of the corrective action but review for systems considerations and relevance to RCA.
Flow Depends on the level of the problem and the corrective action. Routine issues may be resolved at the team level with direct implementation by the affected area manager; the CASS board is then advised of this action. More significant program changes may require prior review and concurrence from the CASS board. The CASS board may raise highest level decisions to the director of QA or senior vice president of engineering and maintenance.
Appendix 1 Followup Responsibility Manager of CASS.
Procedures CASS auditor/analyst or reliability group, as applicable, assigned to develop a followup plan based on the seriousness of the problem.
Followup may include communications from technical area verifying effectiveness, followup audits or data collection, and/or a followup evaluation.
Flow The technical area reports implementation status to the manager of the CASS, who informs the CASS board. The CASS board may inform the director of QA if the problem is sufficiently serious or the implementation plan is not followed.
NOTE: The above tables provide many examples of the system safety attributes: controls and procedures.
l. Communications Between CASS and Other Personnel (A System Safety Attribute:
Interfaces).
Communication of Specific CASS Results and Actions Responsibility CASS board.
Procedures Audits based on updated checklists. The CASS collects and stores operational data in computer systems, and some analysis and alerting features are automated. Audit and analysis results communicated through company electronic mail system, with acknowledgements.
Corrective action tracking through computerized database system.
Flow Electronic mail and standard electronic reports of information flow among CASS board, corrective action teams, technical areas, and director of QA, when applicable.
Communications with Maintenance Personnel Responsibility CASS board.
Procedures The CASS initial and recurrent training included for all personnel, including lectures from CASS personnel. Monthly newsletter summary to production areas regarding CASS activities.
Flow The CASS department works with the training and company communications department, and receives feedback from managers and supervisors, particularly in the maintenance areas.
Appendix 1 Interfaces (A System Safety Attribute: Interfaces) Responsibility CASS board.
Procedures Reliability group provides regular reports on analysis results, trends, and concerns.
Auditors/analysts provide regular reports on findings, analyses, trends, and concerns.
VDRP manager provides summaries of disclosures and proposed comprehensive fixes for CASS review and input.
Copies of C.A.S.E. audit results.
To CASS Board CASS auditors review reports from company ASAPs and at least annually review maintenance-related Aviation Safety Reporting Program reports for consideration in setting audit and operational data collection priorities.
Feedback to technical areas regarding findings, trends, concerns, and followup results.
Feedback to VDRP manager regarding proposed comprehensive fixes;
coordination with manager of CASS.
Monthly reliability analysis summaries and other CASS summaries for distribution to the vice president of engineering and maintenance, director of QA and other senior management, department managers in maintenance, flight and ground operations, Internal Evaluation Program (IEP), safety office, and FAA principal inspector (PI).
CASS reports reflect VDRP comprehensive fixes without detailing the initiating circumstances.
Semiannual summary report to chief executive officer (CEO).
Copies of reliability reports and CASS summaries to the FAA PI. The FAA PI has online access to CASS reports and documents such as summaries, analysis, trends, and corrective action tracking.
From CASS Board CASS board meeting minutes.
Flow Communications channeled through manager of CASS.
m. Personnel Who Perform CASS Functions.
(1) Full-time auditors and analysts; in some cases, an auditor may also be an analyst.
(2) All members of the CASS board who have not participated in specific CASS training receive a total of 16 hours initial training covering CASS, root cause and systems analysis, and human factors.
(3) Company has hired a human factors specialist to address issues across all departments and to participate on the CASS board.
Appendix 1 Page 7 (and 8) How the Certificate Holder Evaluates Its CASS (A System Safety Attribute: Process Measurement) Responsibility Senior vice president of engineering and maintenance.
Procedures IEP evaluates CASS annually.
Flow The CASS transmits IEP reports directly to the CEO and to the senior vice president of engineering and maintenance.