9.9 Auditor Training and Qualifications. If feasible, the aviation organization should specify that IEP auditors have training and/or experience in recognized quality management auditing, systems analysis, RCA, and risk assessment, as well as evaluation principles and techniques. Any one or combination of the following could accomplish training:
1. In-house prepared courses.
2. College courses.
3. Home study course materials.
4. Industry seminars and workshops.
5. Selected FAA courses.
9.10 Determination of Resources. To determine the resources needed by an IEP, senior management should make the scope of oversight and the extent of involvement for the IEP explicit. In most cases, the size of the organization and budgetary considerations will be the principal defining factors.
10 EXAMPLE IEP MANUAL. An example of an IEP manual that incorporates elements discussed within this AC is provided in Appendix B. An aviation organization’s IEP plan should be documented and may be a standalone manual or a portion of some larger department’s manual (e.g., the safety department’s manual). The example included in Appendix B may be more appropriate for a large part 121 air carrier. However, it can be scaled down and modified to fit other aviation organization’s needs.
11 AC FEEDBACK FORM. For your convenience, the AC Feedback Form is the last page of this AC. Note any deficiencies found, clarifications needed, or suggested improvements regarding the contents of this AC on the Feedback Form.
ORIGINAL SIGNED by /s/ Rick Domingo Executive Director, Flight Standards Service 11
Appendix A APPENDIX A. DEFINITIONS A.1 Audit. A methodical, planned review which builds on the principles of inspection.
An audit is used to determine how business is being conducted and compares results with how business should have been conducted in accordance with established procedures.
The most frequently used audit categories are (quality) system audits, service/product audits, and process audits.
A.2 Audit Scope. The operational disciplines and/or operational areas that are assessed during the conduct of an audit.
A.3 Auditor. An individual who has satisfied defined experience prerequisites and is successfully qualified under a defined training program to conduct audits.
A.4 Authority. As a safety attribute, a clearly identifiable, qualified, and knowledgeable person who has the authority to set up and change a process.
A.5 Concern. A conclusion by an auditor, supported by objective evidence, that is seen as a potential problem, trend, or inefficiency that may become a finding.
A.6 Continual. A close, prolonged succession or recurrence, infinite in time, without interruption.
A.7 Control. Key procedure, responsibility, or decision-making position within an organization, department, division, or functional area. Checks and restraints are designed into a process to ensure a desired result. Comprehensive evaluations (system audits) will focus on verifying and testing the controls within the organization.
A.8 Corrective Action. The action(s) taken to correct a deficiency with the intent to preclude recurrence of the finding or noncompliance of an approved standard.
A.9 Corrective Action Plan (CAP). The total plan of an aviation organization to close all findings through implementation of comprehensive corrective action. This plan should include the changes in policy and/or procedures that will ensure continued compliance.
A.10 Evaluation. A functionally independent review of company policies, procedures, and systems. If accomplished by the company itself, the evaluation should be done by an element of the company other than the one performing the function being evaluated.
The evaluation process builds on the concepts of audit and inspection. An evaluation is an anticipatory process, and is designed to identify and correct potential findings before they occur. An evaluation is synonymous with the term “systems audit.”
A.11 Evaluation Standard. Specific criteria on which basis a functional area will be evaluated in terms of compliance or conformance.
A.12 Evidence. A documented statement of fact, prepared by an aviation organization, which may be quantitative or qualitative, and is based on observations, inspections, measurements, or tests that can be verified. For the purpose of internal evaluation, A-1
Appendix A evidence should generally be written documentation that supports an IEP’s analysis and review. This data is necessary to substantiate findings or concerns and to enable management or evaluators to determine the root causes of any reported findings.
A.13 Finding. A determination as a result of an audit that compliance or conformance with an evaluation standard is not being achieved.
A.14 Followup. A process involving monitoring of CAPs to verify timely and effective implementation designed to eliminate the underlying (root) cause of the deficiency.
A.15 Inspection. The act of observing a particular event or action to ensure that correct procedures and requirements are followed during the accomplishment of that event or action. The primary purpose of an inspection is to verify that established standards are followed during an observed event or action.
Note: The term “inspection” is defined in this AC within the context of quality auditing principles. It does not address or define FAA inspections.
A.16 International Air Transport Association (IATA) Operational Safety Audit (IOSA).
An internationally recognized evaluation system designed to assess the operational management and control systems of a part 121 air carrier, which is based on ICAO standards.
A.17 Policy. A high-level overall plan embracing the general goals and acceptable practices of a group. Policies state how goals will be achieved.
A.18 Procedures. Written or unwritten methods (regulatory or nonregulatory) that an aviation organization uses to accomplish a particular process.
A.19 Process. A set of interrelated resources and activities that transform inputs to outputs.
Resources may include personnel, finance, facilities, equipment, techniques, and methods.
A.20 Process Audit. A documented activity that assesses the effectiveness and efficiency of a series of related or sequential work activities.
A.21 Quality Assurance (QA). The independent activity of providing the evidence needed to establish confidence, among all concerned, that the quality function is being performed effectively. This activity “assures quality” through independent evaluation of established processes, procedures, and documentation.
A.22 Quality Control. The determination of the quality of a product by inspection and testing to determine compliance with standards. This activity “controls quality” through establishment of effective controls, documentation, and procedures within specific functional areas.
A-2
Appendix A A.23 Quality Management System. A defined organizational structure, written management responsibilities, and associated system of processes, procedures, and detail documents to ensure compliance with internal standards or regulatory requirements.
A.24 Quality System Audit. A documented activity performed to verify by examination and evaluation of objective evidence that applicable elements of the quality management system are documented and effectively implemented in accordance with specific requirements.
A.25 Responsibility. As a safety attribute, a clearly identifiable, qualified, and knowledgeable person who is accountable for the quality of a process.
A.26 Root Cause Analysis (RCA). Determination of what caused a finding. The identification of the root cause is the key to the implementation of an effective corrective action.
Items to consider when determining the root cause should include deficiencies in training, materials, procedures, empowerment, or management oversight.
A.27 Senior Management. The highest level of management within an organization that has the authority and responsibility for setting policy, demonstrating commitment, meeting requirements, approving resources, setting objectives, implementing processes, and achieving desired outcomes.
A.28 Service/Product Audit. An objective and structured assessment of conformance to required service-level or product-level performance characteristics. It may be qualitative or quantitative, as appropriate.
A-3
Appendix B APPENDIX B. SAMPLE INTERNAL EVALUATION PROGRAM MANUAL (14 CFR PART 121 AIR CARRIER) B.1 Overview.
B.1.1 The IEP will provide operational departments a comprehensive method to continually monitor internal processes, programs, and procedures to ensure that each operating department remains in compliance with appropriate company policies and procedures and applicable Federal, state, and local regulations.
B.1.2 The IEP is mandatory within this carrier for flight operations, In-Flight Services (IFS), Maintenance and Engineering (M&E), and customer service. Other departments are encouraged to develop similar programs to monitor departmental performance and compliance.
B.2 Program.
B.2.1 Key Responsible Personnel.
B.2.1.1 Director of Safety (DOS). The DOS has overall responsibility to develop and implement a comprehensive safety program.
B.2.1.1.1 The duties and responsibilities of the DOS include:
1. Monitoring and reporting to senior management on all air carrier activities that may have an impact on safety.
2. Establishing a reporting system that provides for a timely and free flow of safety-related information.
3. Developing and maintaining a database of incident/accident information to monitor and analyze trends.
4. Monitoring and evaluating the various safety and malfunction-reporting systems to ensure appropriate integration and evaluation of data.
5. Investigating and reporting incidents/accidents and making recommendations to preclude a recurrence.
6. Conducting safety audits and inspections.
7. Soliciting and processing safety improvement suggestions.
8. Developing and maintaining a safety awareness program.
9. Reviewing and evaluating the adequacy of the emergency response plan.
10. Monitoring industry safety concerns that may have an impact on operations.
B-1
Appendix B
11. Maintaining close liaison with the FAA, National Transportation Safety Board (NTSB), and industry safety organizations and associations.
12. Ensuring that the necessary safety program elements have been developed, properly integrated, and coordinated throughout the air carrier. These elements include:
• A safety incident/accident reporting system.
• Accident/incident investigation.
• Safety audits and inspections.
• IEP.
• Operational risk assessment program.
• Open reporting systems.
• Routine monitoring and trend analysis programs.
• Review of external evaluation programs.
• Safety committee(s).
13. Discharging duties as required to meet applicable legal requirements and to maintain safe operations in accordance with part 119, § 119.65.
B.2.1.1.2 DOS Qualifications. The DOS should meet the qualification requirements as outlined in FAA Order 8900.1, Volume 2, Chapter 2, Section 3, Safety Assurance System: Evaluate Part 121/135 Management Personnel. The DOS should have extensive operational experience and professional qualifications in aviation. This includes the knowledge and understanding of the following:
1. Aviation safety programs.
2. Aviation safety standards.
3. Safe aviation operating practices.
B.2.1.1.3 DOS Expertise. The DOS should have established professional qualifications.
These qualifications may be any of the following:
1. An FAA Commercial Pilot or Airline Transport Pilot (ATP) Certificate.
2. An FAA Mechanic Certificate.
3. An FAA Aircraft Dispatcher Certificate.
4. Three years of supervisory experience in any combination of the following:
• Supervisory position with a 14 CFR part 121 or a scheduled part 135 air carrier;
• Supervisory position comparable to the above in the U.S. military;
B-2
Appendix B
• Supervisory position with a U.S. Government department, board, or agency that deals directly with aviation matters; or
• Supervisory position with a part 91 or 125 operations/aviation department.
B.2.1.1.4 DOS Knowledge. The DOS should have a full understanding of the following materials with respect to the airline’s operation:
1. Aviation safety standards and safe operating practices.
2. Title 14 CFR parts 1–199.
3. The airline’s operations specifications (OpSpecs).
4. The manual required by part 121, § 121.133.
5. All appropriate maintenance and airworthiness requirements of parts 1–199.
B.2.1.1.5 DOS Authority. The DOS has the authority to establish and modify the policies and procedures associated with the airline’s IEP. Proposals for modifications to the IEP may be submitted by any manager or employee.
Proposals will be submitted through the Director of Internal Evaluation (DIE), who will seek and obtain approval from the DOS prior to implementation.
Such modifications should be documented within the IEP database (see paragraph B.2.3 concerning IEP data management).
B.2.1.2 Director of Internal Evaluation (DIE). The DIE has overall responsibility for the day-to-day management of the IEP. This person will serve as the chairman of the Internal Evaluation Review Board (IERB) with regard to IEP management and control. The DIE will develop and manage the IEP to ensure compliance with the written program as outlined in this manual and will seek to continually improve and enhance the program.
B.2.1.2.1 The duties and responsibilities of the DIE include:
1. Developing, implementing, and maintaining an IEP in accordance with the guidance contained in the current edition of FAA Advisory Circular (AC) 120-59, Internal Evaluation Programs.
2. Developing an annual evaluation schedule through coordination with the IERB.
3. Ensuring all evaluations are conducted on schedule (planned start date plus or minus days).
4. Reviewing all evaluation information and preparing a summary of all evaluations for the IERB.
B-3
Appendix B
5. Preparing finding reports for submission to each operating department with evaluation findings.
6. Reviewing departmental corrective action plans (CAP) for accuracy and effectiveness.
7. Tracking both findings and corrective actions to ensure followup and completion.
8. Preparing a monthly summary of evaluation findings and CAPs for the DOS.
9. Preparing quarterly and annual reports of the IEP for senior management and the Board of Directors (BOD) Safety Committee.
10. Maintaining the electronic database of evaluation information and audit findings.
11. Assuming the duties of the DOS position in the event that the incumbent is unavailable to perform those duties.
B.2.1.2.2 DIE Qualifications.
1. Have a full understanding of the airline’s aviation safety standards and safe operating practices; parts 1–199; the certificate holder’s OpSpecs; and the manual(s) required by § 121.133.
2. Have supervisory or managerial experience within flight operations, M&E, station management, customer service, or equivalent experience in the transportation industry.
3. Have successfully completed a recognized comprehensive safety certificate program (this qualification is preferred).
4. Be familiar with the Occupational Safety and Health Administration (OSHA), the Environmental Protection Agency (EPA), and other Federal safety regulations.
5. Be familiar with accepted auditing techniques.
B.2.1.3 Manager of IEPs. The Manager of IEPs reports to the DIE. The Manager of IEPs will oversee and coordinate the efforts of the Supervisors of Internal Evaluation (SIE) for the performance of evaluations and audits of systems, processes, operations, stations, and facilities.
B.2.1.3.1 The responsibilities of the Manager of IEPs include:
1. Scheduling audits and evaluations in accordance with the IEP guidelines.
2. Conducting audits and evaluations of stations and facilities.
3. Conducting annual oversight evaluations of operating departments.
B-4
Appendix B
4. Managing the subordinate staff within the department.
5. Compiling and analyzing audit and evaluation findings for root cause identification.
6. Recommending corrective actions pursuant to findings.
7. Tracking corrective actions to completion.
8. Maintaining a database of audit materials, checklists, findings, root causes, corrective actions, and analysis.
9. Preparing reports for the DIE, the DOS, the IERB, and senior management.
10. Managing maintenance of the IEP Company Manual.
B.2.1.3.2 Manager of IEPs Qualifications.
1. Have a full understanding of the airline’s aviation safety standards and safe operating practices; parts 1–199; the certificate holder’s OpSpecs; and the manual(s) required by § 121.133.
2. Have a minimum of 2 years’ experience in the airline industry in station or operations management.
3. Be familiar with accepted auditing techniques.
4. Be familiar with database management.
B.2.1.4 Supervisors of Internal Evaluation (SIE). The SIEs report to the Manager of IEPs. These positions will perform the inspections, audits, and evaluations of the airline’s stations, facilities, and operating departments in accordance with the IEP guidelines.
B.2.1.4.1 The responsibilities of the SIEs include:
1. Conducting scheduled and unscheduled inspections, audits, and evaluations as directed by the Manager of IEPs.
2. Reporting findings related to inspections, audits, and evaluations.
3. Tracking corrective actions (related to findings) to completion.
4. Maintaining files, records, and data related to the IEP.
5. Analyzing findings and deficiencies for causal factor and root cause identification.
B.2.1.4.2 SIE Qualifications.
1. Have an understanding of the airline’s aviation safety standards and safe operating practices; parts 1–199; the certificate holder’s OpSpecs; and the manual(s) required by § 121.133.
B-5
Appendix B
2. Have knowledge of airport operations, cargo, security, and hazmat.
3. Have a minimum of 2 years’ experience in the airline industry.
4. Have a current or past qualification as a Ground Security Coordinator (GSC).
B.2.2 Guidance. The IEP will be established and maintained in accordance with § 119.65 and the current edition of AC 120-59.
B.2.3 Data Management. The airline has purchased and installed a database system to serve as a repository for all data related to the IEP. This is a networked system accessible to specified employees within the Safety Division who have been granted permission to perform certain operations within the system. Written documentation regarding the operation and functionality of the system is maintained within the IEP department by the DIE, and is stored electronically within the database itself. Training related to the operation and maintenance of the system is provided by the manufacturer, and a recurrent user seminar is conducted annually. Functions and capabilities of the system include, but are not limited to:
1. Evaluation and audit schedules.
2. Organization structure.
3. Tracking of staff assigned to audits and evaluations.
4. Tracking of time spent by staff on audits and evaluations.
5. Tracking by user identification of all actions related to the conduct of audits and evaluations.
6. Storage and tracking of all findings related to audits and evaluations.
7. Audit and evaluation scope, objective, and frequency.
8. Assignment of responsibility and timeline for responses to findings.
9. Forwarding (via email) of findings to the designated party responsible for corrective action.
10. Identification of overdue audits and evaluations.
11. Identification of overdue finding responses.
12. Requirement that all findings are resolved before audit/evaluation status can be changed to “closed.”
13. Automatic scheduling of followup audit/evaluation upon closure.
14. Automatic generation of audit/evaluation/finding reports.
15. Tools for classification of audit/evaluation results with regard to risk, quality, conformity, and effectiveness.
16. Tools for causal factor analysis of findings.
B-6
Appendix B
17. Tools for trend analysis of findings.
18. Storage of files related to audit/evaluation accomplishment, findings, corrective actions, analysis, trending, root cause identification/correction, audit/evaluation review, and closure.
B.2.4 Document Control. In accordance with the provisions of the Management Policies and Procedures Manual, audit checklists and work papers will be retained on file for a period of 2 years. Audit reports will be retained in active files for a period of 2 years, then archived for 8 years. In order to enable long-term analysis, derivative data, findings, corrective actions, causal factors, Root Cause Analysis (RCA), and trending, information will be retained within the database indefinitely.
B.2.5 Audit Objectives and Scope. A statement of the objective and scope of each audit is contained in the checklist and the audit report. The following are areas of ongoing oversight:
B.2.5.1 Station evaluations will be performed throughout the year, in addition to the annual oversight evaluations of operating departments. These evaluations will include checks of manuals, training, cargo, hazmat, security, loading, and records. Master checklists are on file in the IEP department. Internal Evaluation personnel will conduct these evaluations.
B.2.5.2 Maintenance operations evaluations will be conducted throughout the year by the Quality Assurance (QA) Department. These evaluations will be reviewed during and included in the annual oversight evaluation of M&E. Maintenance QA audits and records will also be reviewed during the annual oversight evaluation.
B.2.5.3 There are classifications of findings and tools for quality, conformity, effectiveness, and risk assessment. Findings will be entered under the following classifications:
1. NCP—Noncompliance (with regulations).
2. NCF—Nonconformance (with documented procedures).
3. SRC—Safety-Related Concern (currently in compliance and conformance, but the problem may have safety implications).
4. QRC—Quality-Related Concern (currently in compliance and conformance, but the problem indicates a weakness in the quality system).
5. OBS—Observation (comment).
B.2.5.4 Figure B-1, Risk Assessment Matrix, will be used to determine the risk factor associated with a given finding. An NCP finding should be brought by the DOS to the immediate attention of the Vice President of the responsible operating department in order to enable timely submission of a voluntary B-7
Appendix B disclosure to the FAA. NCP and NCF findings will require a written CAP to be issued no later than 15 days from the presentation of the finding report to the audited entity. OBS, SRC, and QRC findings that have a risk factor of 2 will require a written CAP to be issued no later than 30 days from the presentation of the finding report to the audited entity. Any finding that has a risk factor of 1 will require immediate corrective action and notification of the operating department Vice President.
FIGURE B-1. Risk Assessment Matrix
RISK ASSESSMENT MATRIX Severity Likelihood Catastrophic Critical Marginal Negligible Frequent 1 1 2 2 Probable 1 1 2 3 Occasional 1 2 2 3 Remote 2 2 3 3
Severity Scale Definitions Catastrophic Accident with serious injuries and/or fatalities. Loss (or breakdown) of an entire system or subsystem.
Critical Accident or serious incident with injuries and/or moderate damage to aircraft.
Partial breakdown of a system or subsystem.
Marginal Accident or incident with minor injury and/or minor aircraft damage. System deficiencies leading to poor air carrier performance or disruption to air carrier schedules.
Negligible Less than minor injury and/or less than minor system damage. Little or no effect on system or subsystem.
Likelihood Scale Definitions Frequent Will be continuously experienced unless action is taken to change events.
Probable Will occur often if events follow normal pattern.
Occasional Potential for infrequent occurrence.
Remote Not likely to happen (but could).
B-8