Appendix A evidence should generally be written documentation that supports an IEP’s analysis and review. This data is necessary to substantiate findings or concerns and to enable management or evaluators to determine the root causes of any reported findings.
A.13 Finding. A determination as a result of an audit that compliance or conformance with an evaluation standard is not being achieved.
A.14 Followup. A process involving monitoring of CAPs to verify timely and effective implementation designed to eliminate the underlying (root) cause of the deficiency.
A.15 Inspection. The act of observing a particular event or action to ensure that correct procedures and requirements are followed during the accomplishment of that event or action. The primary purpose of an inspection is to verify that established standards are followed during an observed event or action.
Note: The term “inspection” is defined in this AC within the context of quality auditing principles. It does not address or define FAA inspections.
A.16 International Air Transport Association (IATA) Operational Safety Audit (IOSA).
An internationally recognized evaluation system designed to assess the operational management and control systems of a part 121 air carrier, which is based on ICAO standards.
A.17 Policy. A high-level overall plan embracing the general goals and acceptable practices of a group. Policies state how goals will be achieved.
A.18 Procedures. Written or unwritten methods (regulatory or nonregulatory) that an aviation organization uses to accomplish a particular process.
A.19 Process. A set of interrelated resources and activities that transform inputs to outputs.
Resources may include personnel, finance, facilities, equipment, techniques, and methods.
A.20 Process Audit. A documented activity that assesses the effectiveness and efficiency of a series of related or sequential work activities.
A.21 Quality Assurance (QA). The independent activity of providing the evidence needed to establish confidence, among all concerned, that the quality function is being performed effectively. This activity “assures quality” through independent evaluation of established processes, procedures, and documentation.
A.22 Quality Control. The determination of the quality of a product by inspection and testing to determine compliance with standards. This activity “controls quality” through establishment of effective controls, documentation, and procedures within specific functional areas.
A-2
Appendix A A.23 Quality Management System. A defined organizational structure, written management responsibilities, and associated system of processes, procedures, and detail documents to ensure compliance with internal standards or regulatory requirements.
A.24 Quality System Audit. A documented activity performed to verify by examination and evaluation of objective evidence that applicable elements of the quality management system are documented and effectively implemented in accordance with specific requirements.
A.25 Responsibility. As a safety attribute, a clearly identifiable, qualified, and knowledgeable person who is accountable for the quality of a process.
A.26 Root Cause Analysis (RCA). Determination of what caused a finding. The identification of the root cause is the key to the implementation of an effective corrective action.
Items to consider when determining the root cause should include deficiencies in training, materials, procedures, empowerment, or management oversight.
A.27 Senior Management. The highest level of management within an organization that has the authority and responsibility for setting policy, demonstrating commitment, meeting requirements, approving resources, setting objectives, implementing processes, and achieving desired outcomes.
A.28 Service/Product Audit. An objective and structured assessment of conformance to required service-level or product-level performance characteristics. It may be qualitative or quantitative, as appropriate.
A-3
Appendix B APPENDIX B. SAMPLE INTERNAL EVALUATION PROGRAM MANUAL (14 CFR PART 121 AIR CARRIER) B.1 Overview.
B.1.1 The IEP will provide operational departments a comprehensive method to continually monitor internal processes, programs, and procedures to ensure that each operating department remains in compliance with appropriate company policies and procedures and applicable Federal, state, and local regulations.
B.1.2 The IEP is mandatory within this carrier for flight operations, In-Flight Services (IFS), Maintenance and Engineering (M&E), and customer service. Other departments are encouraged to develop similar programs to monitor departmental performance and compliance.
B.2 Program.
B.2.1 Key Responsible Personnel.
B.2.1.1 Director of Safety (DOS). The DOS has overall responsibility to develop and implement a comprehensive safety program.
B.2.1.1.1 The duties and responsibilities of the DOS include:
1. Monitoring and reporting to senior management on all air carrier activities that may have an impact on safety.
2. Establishing a reporting system that provides for a timely and free flow of safety-related information.
3. Developing and maintaining a database of incident/accident information to monitor and analyze trends.
4. Monitoring and evaluating the various safety and malfunction-reporting systems to ensure appropriate integration and evaluation of data.
5. Investigating and reporting incidents/accidents and making recommendations to preclude a recurrence.
6. Conducting safety audits and inspections.
7. Soliciting and processing safety improvement suggestions.
8. Developing and maintaining a safety awareness program.
9. Reviewing and evaluating the adequacy of the emergency response plan.
10. Monitoring industry safety concerns that may have an impact on operations.
B-1
Appendix B
11. Maintaining close liaison with the FAA, National Transportation Safety Board (NTSB), and industry safety organizations and associations.
12. Ensuring that the necessary safety program elements have been developed, properly integrated, and coordinated throughout the air carrier. These elements include:
• A safety incident/accident reporting system.
• Accident/incident investigation.
• Safety audits and inspections.
• IEP.
• Operational risk assessment program.
• Open reporting systems.
• Routine monitoring and trend analysis programs.
• Review of external evaluation programs.
• Safety committee(s).
13. Discharging duties as required to meet applicable legal requirements and to maintain safe operations in accordance with part 119, § 119.65.
B.2.1.1.2 DOS Qualifications. The DOS should meet the qualification requirements as outlined in FAA Order 8900.1, Volume 2, Chapter 2, Section 3, Safety Assurance System: Evaluate Part 121/135 Management Personnel. The DOS should have extensive operational experience and professional qualifications in aviation. This includes the knowledge and understanding of the following:
1. Aviation safety programs.
2. Aviation safety standards.
3. Safe aviation operating practices.
B.2.1.1.3 DOS Expertise. The DOS should have established professional qualifications.
These qualifications may be any of the following:
1. An FAA Commercial Pilot or Airline Transport Pilot (ATP) Certificate.
2. An FAA Mechanic Certificate.
3. An FAA Aircraft Dispatcher Certificate.
4. Three years of supervisory experience in any combination of the following:
• Supervisory position with a 14 CFR part 121 or a scheduled part 135 air carrier;
• Supervisory position comparable to the above in the U.S. military;
B-2
Appendix B
• Supervisory position with a U.S. Government department, board, or agency that deals directly with aviation matters; or
• Supervisory position with a part 91 or 125 operations/aviation department.
B.2.1.1.4 DOS Knowledge. The DOS should have a full understanding of the following materials with respect to the airline’s operation:
1. Aviation safety standards and safe operating practices.
2. Title 14 CFR parts 1–199.
3. The airline’s operations specifications (OpSpecs).
4. The manual required by part 121, § 121.133.
5. All appropriate maintenance and airworthiness requirements of parts 1–199.
B.2.1.1.5 DOS Authority. The DOS has the authority to establish and modify the policies and procedures associated with the airline’s IEP. Proposals for modifications to the IEP may be submitted by any manager or employee.
Proposals will be submitted through the Director of Internal Evaluation (DIE), who will seek and obtain approval from the DOS prior to implementation.
Such modifications should be documented within the IEP database (see paragraph B.2.3 concerning IEP data management).
B.2.1.2 Director of Internal Evaluation (DIE). The DIE has overall responsibility for the day-to-day management of the IEP. This person will serve as the chairman of the Internal Evaluation Review Board (IERB) with regard to IEP management and control. The DIE will develop and manage the IEP to ensure compliance with the written program as outlined in this manual and will seek to continually improve and enhance the program.
B.2.1.2.1 The duties and responsibilities of the DIE include:
1. Developing, implementing, and maintaining an IEP in accordance with the guidance contained in the current edition of FAA Advisory Circular (AC) 120-59, Internal Evaluation Programs.
2. Developing an annual evaluation schedule through coordination with the IERB.
3. Ensuring all evaluations are conducted on schedule (planned start date plus or minus days).
4. Reviewing all evaluation information and preparing a summary of all evaluations for the IERB.
B-3
Appendix B
5. Preparing finding reports for submission to each operating department with evaluation findings.
6. Reviewing departmental corrective action plans (CAP) for accuracy and effectiveness.
7. Tracking both findings and corrective actions to ensure followup and completion.
8. Preparing a monthly summary of evaluation findings and CAPs for the DOS.
9. Preparing quarterly and annual reports of the IEP for senior management and the Board of Directors (BOD) Safety Committee.
10. Maintaining the electronic database of evaluation information and audit findings.
11. Assuming the duties of the DOS position in the event that the incumbent is unavailable to perform those duties.
B.2.1.2.2 DIE Qualifications.
1. Have a full understanding of the airline’s aviation safety standards and safe operating practices; parts 1–199; the certificate holder’s OpSpecs; and the manual(s) required by § 121.133.
2. Have supervisory or managerial experience within flight operations, M&E, station management, customer service, or equivalent experience in the transportation industry.
3. Have successfully completed a recognized comprehensive safety certificate program (this qualification is preferred).
4. Be familiar with the Occupational Safety and Health Administration (OSHA), the Environmental Protection Agency (EPA), and other Federal safety regulations.
5. Be familiar with accepted auditing techniques.
B.2.1.3 Manager of IEPs. The Manager of IEPs reports to the DIE. The Manager of IEPs will oversee and coordinate the efforts of the Supervisors of Internal Evaluation (SIE) for the performance of evaluations and audits of systems, processes, operations, stations, and facilities.
B.2.1.3.1 The responsibilities of the Manager of IEPs include:
1. Scheduling audits and evaluations in accordance with the IEP guidelines.
2. Conducting audits and evaluations of stations and facilities.
3. Conducting annual oversight evaluations of operating departments.
B-4
Appendix B
4. Managing the subordinate staff within the department.
5. Compiling and analyzing audit and evaluation findings for root cause identification.
6. Recommending corrective actions pursuant to findings.
7. Tracking corrective actions to completion.
8. Maintaining a database of audit materials, checklists, findings, root causes, corrective actions, and analysis.
9. Preparing reports for the DIE, the DOS, the IERB, and senior management.
10. Managing maintenance of the IEP Company Manual.
B.2.1.3.2 Manager of IEPs Qualifications.
1. Have a full understanding of the airline’s aviation safety standards and safe operating practices; parts 1–199; the certificate holder’s OpSpecs; and the manual(s) required by § 121.133.
2. Have a minimum of 2 years’ experience in the airline industry in station or operations management.
3. Be familiar with accepted auditing techniques.
4. Be familiar with database management.
B.2.1.4 Supervisors of Internal Evaluation (SIE). The SIEs report to the Manager of IEPs. These positions will perform the inspections, audits, and evaluations of the airline’s stations, facilities, and operating departments in accordance with the IEP guidelines.
B.2.1.4.1 The responsibilities of the SIEs include:
1. Conducting scheduled and unscheduled inspections, audits, and evaluations as directed by the Manager of IEPs.
2. Reporting findings related to inspections, audits, and evaluations.
3. Tracking corrective actions (related to findings) to completion.
4. Maintaining files, records, and data related to the IEP.
5. Analyzing findings and deficiencies for causal factor and root cause identification.
B.2.1.4.2 SIE Qualifications.
1. Have an understanding of the airline’s aviation safety standards and safe operating practices; parts 1–199; the certificate holder’s OpSpecs; and the manual(s) required by § 121.133.
B-5
Appendix B
2. Have knowledge of airport operations, cargo, security, and hazmat.
3. Have a minimum of 2 years’ experience in the airline industry.
4. Have a current or past qualification as a Ground Security Coordinator (GSC).
B.2.2 Guidance. The IEP will be established and maintained in accordance with § 119.65 and the current edition of AC 120-59.
B.2.3 Data Management. The airline has purchased and installed a database system to serve as a repository for all data related to the IEP. This is a networked system accessible to specified employees within the Safety Division who have been granted permission to perform certain operations within the system. Written documentation regarding the operation and functionality of the system is maintained within the IEP department by the DIE, and is stored electronically within the database itself. Training related to the operation and maintenance of the system is provided by the manufacturer, and a recurrent user seminar is conducted annually. Functions and capabilities of the system include, but are not limited to:
1. Evaluation and audit schedules.
2. Organization structure.
3. Tracking of staff assigned to audits and evaluations.
4. Tracking of time spent by staff on audits and evaluations.
5. Tracking by user identification of all actions related to the conduct of audits and evaluations.
6. Storage and tracking of all findings related to audits and evaluations.
7. Audit and evaluation scope, objective, and frequency.
8. Assignment of responsibility and timeline for responses to findings.
9. Forwarding (via email) of findings to the designated party responsible for corrective action.
10. Identification of overdue audits and evaluations.
11. Identification of overdue finding responses.
12. Requirement that all findings are resolved before audit/evaluation status can be changed to “closed.”
13. Automatic scheduling of followup audit/evaluation upon closure.
14. Automatic generation of audit/evaluation/finding reports.
15. Tools for classification of audit/evaluation results with regard to risk, quality, conformity, and effectiveness.
16. Tools for causal factor analysis of findings.
B-6
Appendix B
17. Tools for trend analysis of findings.
18. Storage of files related to audit/evaluation accomplishment, findings, corrective actions, analysis, trending, root cause identification/correction, audit/evaluation review, and closure.
B.2.4 Document Control. In accordance with the provisions of the Management Policies and Procedures Manual, audit checklists and work papers will be retained on file for a period of 2 years. Audit reports will be retained in active files for a period of 2 years, then archived for 8 years. In order to enable long-term analysis, derivative data, findings, corrective actions, causal factors, Root Cause Analysis (RCA), and trending, information will be retained within the database indefinitely.
B.2.5 Audit Objectives and Scope. A statement of the objective and scope of each audit is contained in the checklist and the audit report. The following are areas of ongoing oversight:
B.2.5.1 Station evaluations will be performed throughout the year, in addition to the annual oversight evaluations of operating departments. These evaluations will include checks of manuals, training, cargo, hazmat, security, loading, and records. Master checklists are on file in the IEP department. Internal Evaluation personnel will conduct these evaluations.
B.2.5.2 Maintenance operations evaluations will be conducted throughout the year by the Quality Assurance (QA) Department. These evaluations will be reviewed during and included in the annual oversight evaluation of M&E. Maintenance QA audits and records will also be reviewed during the annual oversight evaluation.
B.2.5.3 There are classifications of findings and tools for quality, conformity, effectiveness, and risk assessment. Findings will be entered under the following classifications:
1. NCP—Noncompliance (with regulations).
2. NCF—Nonconformance (with documented procedures).
3. SRC—Safety-Related Concern (currently in compliance and conformance, but the problem may have safety implications).
4. QRC—Quality-Related Concern (currently in compliance and conformance, but the problem indicates a weakness in the quality system).
5. OBS—Observation (comment).
B.2.5.4 Figure B-1, Risk Assessment Matrix, will be used to determine the risk factor associated with a given finding. An NCP finding should be brought by the DOS to the immediate attention of the Vice President of the responsible operating department in order to enable timely submission of a voluntary B-7
Appendix B disclosure to the FAA. NCP and NCF findings will require a written CAP to be issued no later than 15 days from the presentation of the finding report to the audited entity. OBS, SRC, and QRC findings that have a risk factor of 2 will require a written CAP to be issued no later than 30 days from the presentation of the finding report to the audited entity. Any finding that has a risk factor of 1 will require immediate corrective action and notification of the operating department Vice President.
FIGURE B-1. Risk Assessment Matrix
RISK ASSESSMENT MATRIX Severity Likelihood Catastrophic Critical Marginal Negligible Frequent 1 1 2 2 Probable 1 1 2 3 Occasional 1 2 2 3 Remote 2 2 3 3
Severity Scale Definitions Catastrophic Accident with serious injuries and/or fatalities. Loss (or breakdown) of an entire system or subsystem.
Critical Accident or serious incident with injuries and/or moderate damage to aircraft.
Partial breakdown of a system or subsystem.
Marginal Accident or incident with minor injury and/or minor aircraft damage. System deficiencies leading to poor air carrier performance or disruption to air carrier schedules.
Negligible Less than minor injury and/or less than minor system damage. Little or no effect on system or subsystem.
Likelihood Scale Definitions Frequent Will be continuously experienced unless action is taken to change events.
Probable Will occur often if events follow normal pattern.
Occasional Potential for infrequent occurrence.
Remote Not likely to happen (but could).
B-8
Appendix B Risk Classification 1 High Risk – Unacceptable; requires action.
2 Medium Risk – May be acceptable with review by appropriate authority; requires tracking and probable action.
3 Low Risk – Acceptable without further action.
B.2.6 Unique Terms. A glossary of acronyms used throughout this manual is included in paragraph B.6.
B.2.7 Independence. The DIE reports directly through the DOS to the President and CEO and to the Safety Committee component of the BOD. Figure B-2 depicts this reporting structure.
Figure B-2. Internal Evaluation Reporting Structure
B-9
Appendix B B.2.8 Top Management Review. The Safety Division and the IERB will report all audit findings on an annual basis to senior management.
B.2.9 Senior Management. The CEO and President constitute as senior management.
When making the report to senior management, the Safety Division and the IERB will include all audit information and findings in the report. Senior management will be actively involved in the review process to ensure audit program effectiveness and provide oversight and guidance to operating departments. The senior management’s involvement includes review of all audits and corrective actions through periodic meetings, briefings, or written reports to assess the effectiveness of each operating department’s CAP and audit program.
B.2.10 Schedule. Flight operations, M&E, customer service, and IFS will receive a focused quarterly evaluation (e.g., first quarter records, second quarter manuals, etc.). The schedule for the evaluations of these areas will be determined during the first quarter of each calendar-year, and will be entered into the system by IEP personnel. All stations within the airline’s system will be evaluated three times per year on the following basis:
1. “A” Stations (those with more than 1 million passenger enplanements per year) will be evaluated three times per year by IEP personnel.
2. “B” Stations (those with more than 250,000 but fewer than 1 million passenger enplanements per year) will be evaluated twice per year by IEP personnel, and once per year on a self-audit basis. The self-audit will be forwarded to the IEP department for entry into the database and followup activity tracking.
3. “C” Stations (those with fewer than 250,000 passenger enplanements per year) will be evaluated once per year by IEP personnel, and twice per year on a self-audit basis. The self-audits will be forwarded to the IEP department for entry into the database and followup activity tracking.
4. A list of stations denoting their status (e.g., A, B, or C) will be maintained by the IEP department and will be updated based on actual passenger enplanement records from the previous calendar-year.
5. The schedule for the evaluations and self-audits will be entered into the system and tracked by Internal Evaluation personnel. The IEP department will maintain evaluation and self-audit checklists.
6. Other evaluation and audit activity will be carried out by IEP personnel on an event-driven basis or by request from senior management.
B.2.11 CAPs. All CAPs will be tracked to completion in the system. Audits will remain in an “open” status until all corrective actions have been completed.
B.2.12 RCA. The airline has selected the XYZ system for use by the IEP team to perform RCA and to aid in the development of corrective actions for deep-rooted problems that are discovered during the course of inspections, audits, and evaluations. All IEP personnel will receive training in the use of this system.
B-10