Appendix 1
j. Schedule. Flight operations, maintenance and engineering, customer services, and inflight services will receive a focused quarterly evaluation (e.g. first quarter-records, second quarter manuals, etc.). The schedule for the evaluations of these areas will be determined during the first quarter of each calendar year, and will be entered into the system by IEP personnel. All stations within the airline’s system will be evaluated three times per year on the following basis:
(1) “A” Stations (those with more than 1 million passenger enplanements per year) will be evaluated three times per year by IEP personnel.
(2) “B” Stations (those with more than 250,000 but fewer that 1 million passenger enplanements per year) will be evaluated twice per year by IEP personnel, and once per year on a self-audit basis. The self-audit will be forwarded to the IEP department for entry into the database and followup activity tracking.
(3) “C” Stations (those with fewer than 250,000 passenger enplanements per year) will be evaluated once per year by IEP personnel, and twice per year on a self-audit basis. The selfaudits will be forwarded to the IEP department for entry into the database and followup activity tracking.
(4) A list of stations denoting their status (A, B, or C) will be maintained by the IEP department and will be updated based on actual passenger enplanement records from the previous calendar-year.
(5) The schedule for the evaluations and self-audits will be entered into the system and tracked by Internal Evaluation personnel. The Internal Evaluation department will maintain evaluation and self-audit checklists.
(6) Other evaluation and audit activity will be carried out by IEP personnel on an eventdriven basis, or by request from senior management.
k. CAPs. All CAPs will be tracked to completion in the system. Audits will remain in an “open” status until all corrective actions have been completed.
l. Root Cause Analysis. The airline has selected the XYZ system for use by the IEP team to perform root cause analysis and to aid in the development of corrective actions for deep-rooted problems that are discovered during the course of inspections, audits, and evaluations. All IEP personnel will receive training in the use of this system.
m. Training. All IEP personnel who conduct evaluations will receive training including but not limited to:
(1) Basic auditing skills course given by an accredited organization.
(2) Operational auditing skills course given by an accredited organization.
Appendix 1
(3) XYZ root cause analysis.
(4) Database training (conducted by the database manufacturer).
(5) Ground security coordinator (conducted by airline personnel).
(6) Complaint resolution official (conducted by airline personnel).
(7) Dangerous Goods Acceptance (conducted by airline personnel).
(8) On-the-job performance of audits, evaluations, and interviews (conducted by IEP personnel).
(9) All training will be documented in a file maintained within the IEP department. IEP personnel will receive the minimum training within 6 months of beginning duties in the IEP department.
n. Resources: The DIE will prepare a budget annually for approval by the DOS. This budget will include sufficient funds to support the personnel, equipment, infrastructure, and activities of the IEP department. This operating budget will be developed, submitted, and approved in accordance with the guidance contained in the Management Policies and Procedures Manual.
o. Audits. The IEP will provide a formal, continuous self-evaluation of the airline’s operations to enhance system safety and ensure continual improvement and compliance. Each operating department will be subject to a formal IEP, which will encompass all stations, facilities, domiciles, line stations, and maintenance bases under the operating departments’ control. The IEP describes a comprehensive program of audits that is conducted by operating departments and the Safety Division and the IERB to measure the effectiveness of internal programs and processes. Department-specific audit programs will be developed to review all internal processes, programs, and procedures. All departmental IEPs are outlined in later sections of this manual. This program will be accomplished through a series of self-audits or site audits that will be coordinated and managed by the Safety Division and the IERB.
p. Tools. Examples of audit tools include: audit checklist, employee performance evaluations (pilot and flight attendant check or evaluation rides), aggregate de-identified Flight Operational Quality Assurance (FOQA) data, Aviation Safety Action Program (ASAP) information, results of safety inspections, employee surveys, and any other assessment tools developed by individual departments or by the Safety Division and the IERB that measure performance and compliance. Other resources used in the development of individual audit programs include: regulations (14 CFR), Department of Transportation (DOT) Regulations, National Aviation Safety Inspection Program (NASIP) Audit Checklist, Department of Defense (DOD) Air Carrier Survey Office Maintenance Checklist, the DOD Operations Survey Checklist, and the FAA’s Air Transportation Oversight System (ATOS) assessment tools.
q. Recordkeeping and Administration. All audit tools and associated evaluation results
Appendix 1 used during the Internal Evaluation process will be retained on file in the Safety Department and be made available to the Safety Division and the IERB for review during the Safety Division and the IERB’s ongoing review process.
r. Schedule. The audit checklist will be completed in accordance with the yearly schedule for a given operating department. Status of unresolved issues or discrepancies will be forwarded to senior management as part of the annual audit report to senior management (normally at year’s end).
3. OPERATING DEPARTMENT PROGRAMS. Each operating department’s IEP is described below. The programs have been designed to monitor all critical internal departments and major functional areas. As described above, individual audit programs may be unique and will reflect the organization’s structure and mission. In order to ensure program standardization and to assure a level of continuity among the individual programs, some minimum elements have been established for mandatory inclusion in the individual departmental audits. Because of differences in mission and operational responsibility, a unique minimum element list has been provided for each major operating department and is included in the departments’ programs below.
a. Flight Operations (Includes Dispatch, Crew Scheduling, and Systems Operations Control): The DOS and the IERB have overall responsibility for the flight operations IEP. The DIE is primarily responsible to the DOS for the execution and administration of IEP and will advise the DOS regularly on the status of the program and of any findings or considerations discovered during the process. Audit checklists have been developed and continuous ongoing improvements will be incorporated in these documents, as conditions require. The flight operations IEP will include a process to analyze and improve flight operations policies and procedures at all levels of the organization.
b. Minimum Elements: The list of elements provided below represents the minimum items that must be included and evaluated in the flight operations IEP. A number in parentheses follows some items. The numbers reference the FAA’s ATOS element number, where applicable.
(1) Regulatory compliance (include an assessment of administrative/enforcement actions by Government regulatory agencies).
(2) Compliance with air carrier programs and procedures.
(3) Review of manuals for currency, accuracy, and maintenance (ATOS 2.1).
(4) Accident/incident/injury rate.
(5) Aircraft accident/incident procedures.
(6) Communication processes between employees and management.
Appendix 1
(7) Employee surveys.
(8) A vendor/supplier audit program for outsource crewmember training (ATOS 4.2.9).
(9) OpSpecs.
(10) Safety program (departmental) (ATOS 7.2.1).
(11) Aircrew standardization (ATOS 4.2 and 4.3).
(12) FOQA data (de-identified trend information).
(13) ASAP information.
(14) Captain upgrade (ATOS 4.2.3).
(15) Dispatch and flight release (ATOS 3.2.1).
(16) Personnel training and qualification (ATOS 4.0).
(17) Training program (ATOS 4.0).
(18) Crewmember (flight-crew) qualification (ATOS 4.2. and 4.3).
(19) Dispatcher training (ATOS 4.2.5).
(20) Flight Simulators and Training devices (ATOS 4.2.8).
(21) Hazmat training and procedures (ATOS 4.2 and 3.1.12).
(22) Special airport and route qualifications (ATOS 5.0).
(23) Scheduling and reporting system (ATOS 6.1.1).
(24) Crewmember (flight-crew) flight, rest, and duty time procedures (ATOS 6.1.2).
(25) Dispatcher duty/rest time (ATOS 6.1.4).
(26) Operational release (ATOS 3.2).
(27) Technical administration (ATOS 7.0).
(28) Check airman program (ATOS 4.2.7).
(29) Aircrew program designees (ATOS 4.2.10).
Appendix 1
(30) Minimum equipment list (MEL)/configuration deviation list (CDL) procedures (ATOS 3.2.3).
c. In-flight Services (IFS). The DOS and the IERB have overall responsibility for the inflight services IEP. The DIE is primarily responsible to the DOS for the execution and administration of the IEP and will advise the DOS regularly regarding the status of the program and of any findings or considerations discovered during the audit process. Audit checklists have been developed and continuous and ongoing improvements will be incorporated in these documents, as conditions require. The inflight services IEP will include a process to analyze and improve inflight services policies and procedures at all levels of the organization.
d. Minimum Elements: The list of elements provided below represent the minimum items that will be included and evaluated in the inflight services IEP. A number in parentheses follows some items. The numbers provide a cross-reference to the FAA’s corresponding ATOS element number, where applicable. The IEP will employ company developed tools and standards for these elements:
(1) Regulatory compliance (include an assessment of administrative/enforcement actions by Government regulatory agencies).
(2) Compliance with air carrier programs and procedures.
(3) Review of manuals for currency, accuracy, and maintenance (ATOS 2.1).
(4) Aircraft accident/incident procedures.
(5) Accident/incident/injury rate.
(6) Communication processes between employees and management.
(7) ASAP information.
(8) Employee surveys (when available).
(9) Intoxicated passengers screening and reporting.
(10) Carry-on baggage (ATOS 3.1.5).
(11) Exit row seating (ATOS 3.1.6)
(12) Outsource crewmember training (ATOS 4.2.9).
(13) Aircrew standardization (ATOS 4.2 and 4.3)
(14) Aircrew safety assessment (inflight observations) (ATOS 7.2.1).
Appendix 1
(15) Training of flight attendants (ATOS 4.2.4).
(16) Crewmember (aircrew) qualifications (ATOS 4.3.2).
(17) Hazmat training (ATOS 4.2).
(18) Procedures required by the Hazardous Materials/Dangerous Goods Program (ATOS
3.1.12 and 4.2.4)
(19) Flight attendant rest and duty time procedures (ATOS 6.1.3).
(20) Technical administration (ATOS 7.0).
e. IFS Additional Audit Items. The following items will be evaluated as part of the IFS Audit program:
(1) Cabin crewmember flight, rest, and duty time procedures (ATOS 6.1.3).
(2) Scheduling practices and compliance (ATOS 6.1.3).
(3) Outsourced in-flight crewmember training program (ATOS 4.2.9).
(4) List of current vendors.
(5) Flight report data management.
(6) Accident and incident rates and related statistics.
f. Maintenance and Engineering (M&E). The DOS and the IERB have overall responsibility for the M&E IEP. The DIE is primarily responsible to the DOS for the execution and administration of the IEP and will advise the DOS regularly regarding the status of the program and of any findings or considerations discovered during the process. Audit checklists have been developed and continuous ongoing improvements will be incorporated into these documents, as conditions require. The M&E IEP will include a process to analyze and improve M&E policies and procedures at all levels of the organization.
g. Minimum Elements. The list of elements provided below represents the minimum items that must be included and evaluated in the M&E IEP. A number in parentheses follows some items. The numbers reference the FAA’s ATOS element number, where applicable.
(1) Aircraft airworthiness (ATOS 1.1.1).
(2) Records and reporting system (ATOS 1.2).
(3) Maintenance organization (ATOS 1.3).
Appendix 1
(4) Continuous Analysis & Surveillance (ATOS 1.3.11).
(5) Regulatory compliance (include an assessment of administrative/enforcement actions by Government regulatory agencies).
(6) Compliance with air carrier programs and procedures.
(7) Review of manuals for currency, accuracy, and maintenance (ATOS 2.1).
(8) MEL/CDL Procedures (ATOS 1.3.5)
(9) Aircraft accident/incident procedures.
(10) Accident/incident/injury rate.
(11) Communication processes between employees and management.
(12) ASAP information.
(13) Employee surveys (when available).
(14) Personnel training and qualification (ATOS 4.0).
(15) Hazmat training for maintenance personnel.
(16) Hazmat procedures for maintenance personnel.
(17) Maintenance duty time procedures (ATOS 6.2.1).
(18) Technical administration (ATOS 7.0).
h. Customer Service. The DOS and the IERB have overall responsibility for the customer service IEP. The DIE is primarily responsible to the DOS for the execution and administration of the IEP and will advise the DOS and the IERB regularly regarding the status of the program and of any findings or considerations discovered during the audit process.
(1) Audit checklists have been developed and continuous and ongoing improvements will be incorporated in these documents, as conditions require. The customer service IEP will include a process to analyze and improve customer service policies and procedures at all levels of the organization.
(2) Documentation (including CAPs) that is provided by customer service to substantiate answers to audit questions must be maintained by customer service. Revisions to any such documentation must be forwarded to the Safety Division and the IERB for inclusion in the audit file.
Appendix 1
i. Audits. The customer service IEP is not a single, stand-alone audit. It is a collection of regularly scheduled station audits, self-audits, and on-site audits conducted throughout the year.
The responsibility for collecting, analyzing, and forwarding of audit information is the responsibility of the DIE. All audit findings will be routed through the VP of Customer Service for corrective action and followup. The DIE will compile and forward an annual audit report in accordance with this manual, normally at year’s end.
j. Minimum Elements. The list of minimum elements provided below represent the minimum items that must be included and evaluated in the customer service IEP. A number in parentheses follows some items. The numbers reference the FAA’s ATOS element number, where applicable.
(1) Hazmat (ATOS 3.1.12).
(2) Training records.
(3) Weight and balance (ATOS 3.2.2).
(4) Procedures manual.
(5) Stations files.
(6) Flight files.
(7) Security.
(8) Training program.
(9) Station personnel (ATOS 4.2.6).
(10) Mandated training.
(11) Carriage of cargo (ATOS 3.1.8).
(12) Station facilities (ATOS 5.1.5).
(13) Station manuals (ATOS 2.1).
(14) Passenger handling (ATOS 3.1.1).
(15) Carry-on baggage (ATOS 3.1.5).
(16) Exit row seating (ATOS 3.1.6).
Appendix 1
4. PROCESS.
a. The Audit. Each department may be subject to either a single annual audit or a continual progression of audits throughout the year based upon the department’s operational requirements and structure. A detailed audit schedule will be developed each calendar year by the DIE. This schedule will be maintained within the database system.
(1) Departments that are monitored by continual running audits will be subject to an annual audit review. During this review, all individual audits will be collected and reviewed by management. The collected audits will be combined and analyzed, and a CAP, where required in accordance with the criteria in this manual, will be developed for each finding and forwarded to senior management for review. The same process will be followed for departments that are under an individual annual audit program.
(2) In all cases, audit information will be analyzed by the operating department and by the Safety Division and the IERB, and, where required by this manual, a CAP will be developed.
Senior management will be briefed on the audit results and plan of corrective action. During the conduct of either style of audit, auditors should apply the appropriate classification for risk assessment (Figure 1) and for quality, conformance or effectiveness (NCP, NCF, QRC, SRC, or OBS) to each audit finding. When Risk Factor (1) or NCP findings are identified, the VP of the appropriate operating department should be notified without delay. Every effort must be made to correct or eliminate Risk Factor (1) and NCP findings immediately.
(3) Auditors must ensure that when high risk factor findings cannot be corrected on the spot that adequate precautions are taken to prevent injury or accidents. When notified of apparent violations of the regulations, management must evaluate each finding and decide if selfdisclosure in accordance with AC 00-58, Voluntary Disclosure Reporting Program, is warranted.
b. Analysis. The analysis phase begins immediately following the completion of an annual audit or at the conclusion of the annual review. Data from either an annual audit or continuing audits are combined and an analysis of findings completed. During this phase, all findings must be identified and cataloged by the IEP Department. All departments will receive reports from the Safety Division and the IERB of findings discovered during audit processes. An audit trail must be maintained throughout the audit program. Documentation will be presented to senior management and forwarded to the Safety Division and the IERB as part of the final report. The focus of data analysis should be to determine the root cause and risk.
c. Corrective Action. After the analysis of an audit has been completed and reviewed by the IERB, the operating department will be required to provide a detailed CAP for each finding.
Departments shall prepare a written plan, which outlines corrective action for each audit finding.
This plan will be part of the information presented to senior management and forwarded to the Safety Division and the IERB as part of the final report. An effective method must be developed to monitor, measure, and validate the effectiveness of corrective actions taken by the department.
Corrective action may take many forms; it may be as simple as a policy letter or require in-depth changes to training programs, manuals, or procedures. Regardless of the form or type of
Appendix 1 corrective action planned, an on-going evaluation of the action’s effectiveness must occur. Each new or subsequent audit should evaluate and consider the effectiveness of corrective action steps taken during prior audits. A continual review of corrective action steps assures a level of internal oversight as well as providing a method of validating CAPs.
d. Audit Report Resolution. After the analysis phase and CAP development phase has been completed, and not more than 90 days after an audit is completed, a formal report of audit findings, corrective action, and methods planned for tracking findings through resolution will be presented to senior management. This plan will outline specific corrective action and provide a timeline for completion. For departments that run yearlong programs, an annual CAP is due by year’s end. This report should be a consolidated summary of all findings, corrective action steps with specific time lines for resolution.
e. Senior Management Review. Each December, a senior management review will be conducted of all audit information from the preceding year. Each department will be required to present a status report of all IEP findings and corrective measures taken. Corrective actions will be evaluated during the following years’ audit to validate completion and effectiveness. These findings will be presented to senior management at the yearly meeting by the DIE.
f. Employee Feedback. The most recent results of IEP audits for a given functional area will be published quarterly in the appropriate employee group newsletter (flight operations, maintenance, inflight services, customer service). Results will include progress reports on corrective actions for previous audit findings. In addition, companywide IEP findings and status reports, as well as audit schedules, will be maintained on the IEP Web site accessible to all employees through company intranet.
5. OVERSIGHT (PROGRAM EFFECTIVENESS).
a. The Safety Division and the IERB are responsible for oversight of the IEP with regard to storage of audit information and for periodic followup with operating departments. Each department will forward complete audit information to the Safety Division and the IERB after reporting this information to senior management.
b. The Safety Division and the IERB will track audit findings that remain open or require followup action using the system. The Safety Division and the IERB will periodically review CAPs and determine if action is being completed as scheduled and is effective in correcting or eliminating the finding. A designated Safety Manager will be assigned oversight responsibility for each operating department. This manager will monitor progress within his/her assigned department with regard to corrective actions taken within the department and the effectiveness of that action. In addition, the assigned manager will perform an annual safety audit of the operating department. During that audit, a review of the department’s IEP will be conducted.
c. The operating department’s analysis phase and action plan will be considered and evaluated with regard to progress made toward resolution of audit findings. The results of safety audits will also be forwarded to senior management for review. In addition, each department will implement, where feasible, procedures for continuous measurement and analysis of safety
Appendix 1 critical processes between audits and evaluations. In order to provide an additional level of oversight to the IEP, the Internal Audit Department will add the IEP to its list of audited entities and will appraise the adequacy and effectiveness of the program’s internal controls on a biennial basis.
6. ACRONYMS USED IN THIS MANUAL.
AC Advisory Circular ASAP Aviation Safety Action Program ATA Air Transport Association ATOS Air Transportation Oversight System BOD Board of Directors CAP Corrective Action Plan CASS Continuing Analysis and Surveillance System CDL Configuration Deviation List CEO Chief Executive Officer CFR Code of Federal Regulations CSET Certification, Standard ization and Evaluation team DIE Director of Internal Evaluations DOD Department of Defense DOS Director of Safety DOT Department of Transportation FAA Federal Aviation Administration FAR Federal Aviation Regulation FOQA Flight Operational Quality Assurance GMM General Maintenance Manual HAZMAT Hazardous Materials (Dangerous Goods) HBAT Handbook Bulletin for Air Transportation HBAW Handbook Bulletin for Airworthiness IATA International Air Transport Association ICAO International Civ il Aviation Organization IEP Internal Evaluation Program IERB Internal Evaluation Review Board IFS Inflight Services MEL Minimum Equipment List NASIP National Aviation Safety Inspection Program NCF Non-conformance (with documented procedures) NCP Non-compliance (with regulations) NTSB National Transportation Safety Board OBS OSHA Observation (comment) Occupational Safety and Health Administration
Appendix 1 QA Quality Assurance QRC Quality Related Concern (currently in compliance and conformance, but the problem indicates a weakness in the Quality System) SAI Safety Attribute Inspection SIE Supervisors of Internal Evaluation SRC Safety Related Concern (currently in compliance and conformance, but the problem may have safety implications)