Par 7 Page 13
(e) Identification of the individual(s) who are assigned the responsibility for implementing each of the corrective steps.
(2) The individuals responsible for managing an IEP should facilitate the corrective action process by:
(a) Ensuring that CAPs are developed in a timely manner.
(b) Verifying that CAPs include the items outlined above.
(c) Monitoring and documenting implementation of corrective actions through resolution of the issue(s).
(d) Providing senior management with an independent assessment of CAP development and implementation
(e) Initiating scheduled and/or unannounced followup evaluations to ensure the effectiveness of corrective actions specified in CAPs.
g. Reporting. The content of the detailed evaluation report should be defined in the program. An evaluation report should be sent to the responsible managers and the air carrier’s senior management for review. Briefings should be given to senior management and to other responsible parties as appropriate. Reports of the status of corrective actions should be provided as appropriate. The air carrier should decide upon the frequency, format, and structure for informing senior management of internal evaluation schedules, results, and followup actions to validate that corrective actions have been implemented and that they are effective. It is recommended that the reporting structure also be documented by the air carrier and become a part of its program.
h. Records. The results of an internal evaluation should be documented in reports and other appropriate records, consistent with the process of internal reporting at the air carrier. Operators may find it useful to manage the results of IEP evaluations (as well as the other elements of the IEP) through a database or a quantitative application. IEP records should include:
(1) Planned evaluation reports.
(2) Special evaluation reports, including the trends or other reasons for scheduling a special evaluation.
(3) CAPs.
(4) Results of followup evaluations.
(5) Records of auditor training and qualifications.
Page 14 Par 7
i. Auditor Training and Qualifications. If feasible, the certificate holder should specify that IEP auditors have training and/or experience in recognized quality management auditing, systems analysis, root cause analysis, and risk assessment, as well as evaluation principles and techniques. Any one or combination of the following could accomplish training:
(1) In-house prepared courses.
(2) College courses.
(3) Home study course materials.
(4) Industry seminars and workshops.
(5) Selected FAA courses.
j. Determination of Resources. To determine the resources needed by an IEP, senior management should make the scope of oversight and the extent of involvement for the IEP explicit. In most cases, size of the organization and budgetary considerations will be the principal defining factors.
8. CONCLUSION. Development of IEPs, as discussed in this AC, should ensure that company policies and procedures are responsive to organizational changes and that certificate holders continually comply with appropriate safety and regulatory requirements. Furthermore, the FAA strongly encourages certificate holders to make an IEP an integral part of their management process.
9. APPENDIX. An example of an IEP manual that incorporates elements discussed within this AC is provided in the appendix. A carrier’s IEP plan should be documented and may be a standalone manual or may be a portion of some larger department’s manual (e.g., the safety department’s manual). The example included here may be more appropriate for a large carrier;
however, it can be scaled down to fit a smaller carrier’s needs.
ORIGINAL SIGNED BY:
James J. Ballough Director, Flight Standards Service
Appendix 1 APPENDIX 1. SAMPLE INTERNAL EVAL UATION PROGRAM (IEP) MANUAL (14 CFR PART 121 OPERATOR)
1. OVERVIEW.
a. The Internal Evaluation Program (IEP) will provide a comprehensive method for operational departments to continually monitor internal processes, programs, and procedures to ensure that each operating department remains in compliance with appropriate company policies and procedures and applicable Federal, State and local regulations.
b. The IEP is mandatory within this carrier for flight operations, inflight services, maintenance and engineering, and customer service. Other departments are encouraged to develop similar programs to monitor departmental performance and compliance.
2. PROGRAM.
a. Key Responsible Personnel
(1) Director of Safety (DOS). The DOS has overall responsibility to develop and implement a comprehensive safety program.
(a) The duties and responsibilities of the DOS include:
(i) Monitor and report to senior management on all air carrier activities that may have an impact on safety.
(ii) Establish a reporting system that provides for a timely and free flow of safetyrelated information.
(iii) Develop and maintain a database of incident/accident information to monitor and analyze trends.
(iv) Monitor and evaluate the various safety and malfunction reporting systems to ensure appropriate integration and evaluation of data.
(v) Investigate and report on incidents/accidents and make recommendations to preclude a recurrence.
(vi) Conduct safety audits and inspections.
(vii) Solicit and process safety improvement suggestions.
(viii) Develop and maintain a safety awareness program.
(ix) Review and evaluate the adequacy of the emergency response plan.
Appendix 1
(x) Monitor industry safety concerns that may have an impact on operations.
(xi) Maintain close liaison with the FAA, National Transportation Safety Board (NTSB) and industry safety organizations and associations.
(xii) Ensure that the necessary safety program elements have been developed, properly integrated, and coordinated throughout the air carrier. These elements include:
[a] A safety incident/accident reporting system.
[b] Accident/incident investigation.
[c] Safety audits and inspections.
[d] IEP.
[e] Operational risk assessment program.
[f] Open reporting systems.
[g] Routine monitoring and trend analysis programs.
[h] Review of external evaluation programs.
[i] Safety Committee(s).
(xiii) Discharge duties as required to meet applicable legal requirements and to maintain safe operations in accordance with Title 14 of the Code of Federal Regulations (14 CFR) part 119, section 119.65.
(b) DOS Qualifications. The DOS should meet the qualification requirements as outlined in the Joint Flight Standards Handbook Bulletin for Air Transportation (HBAT) and Airworthiness (HBAW), HBAT 99-19 and HBAW 99-16, 14 CFR Part 121 and 135 Air Carrier Safety Departments, Programs, and the Director of Safety, FAA Order 8400.10, Air Transportation Operations Inspector’s Handbook, appendix 3, current edition. The DOS should have extensive operational experience and professional qualifications in aviation. This includes the knowledge and understanding of the following:
(i) Aviation safety programs.
(ii) Aviation safety standards.
(iii) Safe aviation operating practices.
(c) DOS Expertise. DOS should have established professional qualifications. These qualifications may be any of the following:
Appendix 1
(i) An FAA commercial pilot or airline transport pilot certificate.
(ii) An FAA mechanic certificate.
(iii) An FAA aircraft dispatcher certificate.
(iv) Three years experience in a supervisory position with a 14 CFR part 121 or a scheduled 14 CFR part 135 air carrier, or 3 years U.S. military aviation operations experience in a comparable position.
(v) Three years experience in a supervisory position with a U.S. Government department, board, or agency that deals directly with aviation matters.
(d) DOS Knowledge. The DOS should have a full understanding of the following materials with respect to the airline's operation:
(i) The airline's operations specifications.
(ii) The manual required by 14 CFR part 121, section 121.133.
(iii) All appropriate maintenance and airworthiness requirements of 14 CFR chapter I (parts 1 through 199).
(e) DOS Authority. The DOS has the authority to establish and modify the policies and procedures associated with the airline’s IEP. Proposals for modifications to the IEP may be submitted by any manager or employee. Proposals will be submitted through the Director of Internal Evaluation, who will seek and obtain approval from the DOS prior to implementation.
Such modifications should be documented within the IEP database (see section 2c below concerning IEP data management).
(2) Director of Internal Evaluation (DIE). The DIE has overall responsibility for the day-to-day management of the IEP. This person will serve as the Chairman of the Internal Evaluation Review Board (IERB) with regard to IEP management and control. The Director will develop and manage the IEP to ensure compliance with the written program as outlined in this manual and will seek to continually improve and enhance the program.
(a) The duties and responsibilities of the DIE include:
(i) Develop, implement and maintain an IEP in accordance with the guidance contained in FAA Advisory Circular (AC) 120-59, Air Carrier Internal Evaluation Programs, current edition.
(ii) Through coordination with the IERB, develop an annual evaluation schedule.
(iii) Ensure that all evaluations are conducted on schedule (planned start date plus
Appendix 1 or minus days).
(iv) Review all evaluation information and prepare a summary of all evaluations for the IERB.
(v) Prepare finding reports for submission to each operating department with evaluation findings.
(vi) Review departmental corrective action plans (CAPs) for accuracy and effectiveness.
(vii) Track both findings and corrective actions to ensure followup and completion.
(viii) Prepare a monthly summary of evaluation findings and CAPs for the DOS.
(ix) Prepare quarterly and annual reports of IEP for senior management and the Board of Directors Safety Committee.
(x) Maintain the electronic database of evaluation information and audit findings.
(xi) Assume the duties of the DOS position in the event that the incumbent is unavailable to perform those duties.
(b) DIE Qualifications.
(i) Have a full understanding of the airline’s aviation safety standards and safe operating practices; 14 CFR, chapter I; the certificate holder’s operations specifications (OpSpecs); and the manual(s) required by 14 CFR part 121, section 121.133.
(ii) Have supervisory or managerial experience within flight operations, maintenance and engineering, station management, customer services, or equivalent experience in the transportation industry.
(iii) Preferred: Have successfully completed a recognized comprehensive safety certificate program.
(iv) Be familiar with the Occupational Safety and Health Administration (OSHA), the Environmental Protection Agency (EPA) and other federal safety regulations.
(v) Familiar with accepted auditing techniques.
(3) Manager of IEPs. The Manager of IEPs reports to the DIE. The Manager of IEPs shall oversee and coordinate the efforts of the Supervisors of Internal Evaluation (SIE) for the performance of evaluations and audits of systems, processes, operations, stations and facilities.
Appendix 1
(a) The responsibilities of the Manager of IEPs include:
(i) Schedule audits and evaluations in accordance with the IEP guidelines.
(ii) Conduct audits and evaluations of stations and facilities.
(iii) Conduct annual oversight evaluations of operating departments.
(iv) Manage the subordinate staff within the department.
(v) Compile and analyze audit and evaluation findings for root cause identification.
(vi) Recommend corrective actions pursuant to findings.
(vii) Track corrective actions to completion.
(viii) Maintain database of audit materials, checklists, findings, root causes, corrective actions, and analyses.
(ix) Prepare reports for the DIE, the DOS, the IERB, and senior management.
(x) Manage maintenance of the IEP Company Manual.
(b) Manager of IEPs Qualifications.
(i) Have a full understanding of the airline’s aviation safety standards and safe operating practices; 14 CFR, chapter I; the certificate holder’s OpSpecs; and the manual(s) required by 14 CFR part 121, section 121.133.
(ii) Minimum of 2 years experience in the airline industry in station or operations management.
(iii) Familiar with accepted auditing techniques.
(iv) Familiar with database management.
(4) Supervisors of Internal Evaluation (SIE). The SIEs report to the Manager of IEPs.
These positions will perform the inspections, audits, and evaluations of the airline’s stations, facilities, and operating departments in accordance with the IEP guidelines.
(a) The responsibilities of the SIEs include:
(i) Conduct scheduled and unscheduled inspections, audits, and evaluations as directed by the Manager of IEPs.
Appendix 1
(ii) Report findings related to inspections, audits, and evaluations.
(iii) Track corrective actions related to findings to completion.
(iv) Maintain files, records, and data related to the IEP.
(v) Analyze findings and deficiencies for causal factor and root cause identification.
(b) SIE Qualifications.
(i) Have an understanding of the airline’s aviation safety standards and safe operating practices; 14 CFR, chapter I; the certificate holder’s OpSpecs; and the manual(s) required by 14 CFR part 121, section 121.133.
(ii) Knowledge of airport operations, cargo, security, and Hazmat.
(iii) Minimum of 2 years experience in the airline industry.
(iv) Current or past qualification as Ground Security Coordinator.
b. Guidance. The IEP will be established and maintained in accordance with the latest revisions of 14 CFR part 119, section 119.65; Order 8400.10, volume 3, chapter 8, current edition; and AC 120-59, current edition.
c. Data Management. The airline has purchased and installed a database system to serve as a repository for all data related to the IEP. This is a networked system accessible to specified employees within the Safety Division who have been granted permission to perform certain operations within the system. Written documentation regarding the operation and functionality of the system is maintained within the Internal Evaluation Department by the DIE, and is stored electronically within the database itself. Training related to the operation and maintenance of the system is provided by the manufacturer, and a recurrent user seminar is conducted annually.
Functions and capabilities of the system include (but are not limited to):
(1) Evaluation and audit schedules.
(2) Organization structure.
(3) Tracking of staff assigned to audits and evaluations.
(4) Tracking of time spent by staff on audits and evaluations.
(5) Tracking by user ID of all actions related to the conduct of audits and evaluations.
(6) Storage and tracking of all findings related to audits and evaluations.
Appendix 1
(7) Audit and evaluation scope, objective, and frequency.
(8) Assignment of responsibility and timeline for responses to findings.
(9) Forwarding (via e-mail) of findings to the designated party responsible for corrective action.
(10) Identification of overdue audits and evaluations.
(11) Identification of overdue finding responses.
(12) Requirement that all findings are resolved before audit/evaluation status can be changed to “closed.”
(13) Automatic scheduling of followup audit/evaluation upon closure.
(14) Automatic generation of audit/evaluation/finding reports.
(15) Tools for classification of audit/evaluation results with regard to risk, quality, conformity, and effectiveness.
(16) Tools for causal factor analysis of findings.
(17) Tools for trend analysis of findings.
(18) Storage of files related to audit/evaluation accomplishment, findings, corrective actions, analysis, trending, root cause identification/correction, audit/evaluation review, and closure.
d. Document Control. In accordance with the provisions of the Management Policies and Procedures Manual, audit checklists and work papers will be retained on file for a period of 2 years. Audit reports shall be retained in active files for a period of 2 years, then archived for 8 years. In order to enable long-term analysis, derivative data, findings, corrective actions, causal factors, root cause analysis, and trending, information will be retained within the database indefinitely.
e. Audit Objectives and Scope. A statement of the objective and scope of each audit is contained in the checklist and the audit report. Areas of ongoing oversight:
(1) Station evaluations will be performed throughout the year, in addition to the annual oversight evaluations of operating departments. These evaluations will include checks of manuals, training, cargo, hazmat, security, loading, and records. Master checklists are on file in the Internal Evaluation department. Internal Evaluation personnel will conduct these evaluations.
(2) Maintenance operations evaluations will be conducted throughout the year by the
Appendix 1 Quality Assurance department. These evaluations will be reviewed during and included in the annual oversight evaluation of maintenance and engineering. Maintenance QA audits and records will also be reviewed during the annual oversight evaluation.
(3) Classification of findings and tools for quality, conformity, effectiveness, and risk assessment – Findings will be entered under the following classifications:
(a) NCP - noncompliance (with regulations).
(b) NCF - nonconformance (with documented procedures).
(c) SRC - Safety Related Concern (currently in compliance and conformance, but the problem may have safety implications).
(d) QRC - Quality Related Concern (currently in compliance and conformance, but the problem indicates a weakness in the Quality System).
(e) OBS - observation (comment).
(4) The risk assessment matrix shown in Figure 1 will be used to determine the risk factor associated with a given finding. An NCP finding should be brought by the DOS to the immediate attention of the VP of the responsible operating department in order to enable timely submission of a voluntary disclosure to the FAA. NCP and NCF findings will require a written CAP to be issued no later than 15 days from the presentation of the finding report to the audited entity. OBS, SRC, and QRC findings that have a risk factor of (2) will require a written CAP to be issued no later than 30 days from the presentation of the finding report to the audited entity.
Any finding that has a risk factor of (1) will require immediate corrective action and notification of the operating department VP.
FIGURE 1: RISK ASSESSMENT MATRIX
RISK ASSESSMENT MATRIX Severity Likelihood Catastrophic Critical Marginal Negligible
Frequent
Probable
Occasional
Remote
Appendix 1 Severity Scale Definitions Catastrophic Accident with serious injuries and/or fatalities. Loss (or breakdown) of an entire system or sub-system.
Critical Accident or Serious Incident with injuries and/or moderate damage to aircraft. Partial breakdown of a system or subsystem.
Marginal Accident or Incident with minor injury and/or minor aircraft damage. System Deficiencies leading to poor air carrier performance or disruption to the air carrier schedules.
Negligible Less than minor injury and/or less than minor system damage. Little or no effect on system or subsystem.
Likelihood Scale Definitions Frequent Will be continuously experienced unless action is taken to change events.
Probable Will occur often if events follow normal pattern.
Occasional Potential for infrequent occurrence.
Remote Not likely to happen (but could).
Risk Classification 1 High Risk - Unacceptable; requires action.
2 Medium Risk - May be acceptable with review by appropriate authority; requires tracking and probable action.
3 Low Risk - Acceptable without further action.
f. Unique Terms. A glossary of acronyms used throughout this manual is included at the end of this section.
g. Independence. The DIE reports directly through the DOS to the President and Chief Executive Officer (CEO) and to the Safety Committee component of the Board of Directors.
The following chart depicts this reporting structure:
Appendix 1
Internal Evaluation Reporting Structure President & Chief Executive Officer (CEO) Board of Directors (BOD) Safety Committee Director of Safety (DOS) Director Internal Evaluation (DIE) & Regulatory Compliance Manager Internal Evaluation Program (IEP) Supervisors Internal Evaluation (SIE) Operating Department Evaluation Programs Internal Evaluation Review Board (IERB)
h. Top Management Review. The Safety Division and the IERB will report all audit findings on an annual basis to senior management.
i. Senior Management. The CEO and President constitute senior management. When making the report to senior management, the Safety Division and the IERB shall include all audit information and findings in the report. Senior management will be actively involved in the review process to ensure audit program effectiveness and provide oversight and guidance to operating departments. Senior management’s involvement includes review of all audits and corrective actions through periodic meetings, briefings, or written reports to assess the effectiveness of each operating department’s CAP and audit program.