# AC 120-59B: Internal Evaluation Programs - Type: Advisory circular (guidance, not mandatory unless a rule or AD requires it) - Status: Current - Approved: October 30, 2018 - Last reviewed: August 13, 2024 - Family: AC 120-59 - Cancels: [AC 120-59A](https://openairdocs.org/ac/120-59A) - 14 CFR parts: 5 - 14 CFR sections: [5.71](https://www.ecfr.gov/current/title-14/section-5.71) - Pages: 35 - PDF: https://openairdocs.org/files/sha256/48/c6/48c6049de47dc2788f803fa91f94877dd9afb37a65b3d19301f806337c83359b - FAA DRS: https://drs.faa.gov/browse/excelExternalWindow/F9413FE7A22918218625835400685AEE.0001 - DRS record last changed: August 28, 2026 - Page: https://openairdocs.org/ac/120-59 ## Versions - [AC 120-59B](https://openairdocs.org/ac/120-59) (Current), approved Oct 30, 2018, 35 pages - [AC 120-59A](https://openairdocs.org/ac/120-59A) (Historical), approved Apr 17, 2006, 36 pages - [AC 120-59](https://openairdocs.org/ac/120-59?v=1B8260B4066CC578862569BC0057CED4) (Historical), approved Oct 26, 1992, 19 pages ## Text Each page's text from the PDF; open the PDF at a page with #page=N. ### Page 1 U.S. Department of Transportation Federal Aviation Administration Advisory Circular Subject: Internal Evaluation Programs Date: 10/30/18 AC No: 120-59B Initiated by: AFS-200 Change: 1 PURPOSE OF THIS ADVISORY CIRCULAR (AC). This AC provides information and guidance to persons operating under Title 14 of the Code of Federal Regulations (14 CFR) who are designing, developing, and implementing an Internal Evaluation Program (IEP). The procedures and practices outlined in this AC can be applied to all operations. 1.1 Regulatory Requirement. There is no regulatory requirement for an IEP. However, the Federal Aviation Administration (FAA) encourages the development and use of such programs to increase management and employee awareness of their responsibility to promote continual compliance with all regulatory requirements and best safety practices. Establishing the type of program described in this document is completely voluntary. If used by an air carrier, an IEP may satisfy the internal evaluation requirement of 14 CFR part 5, § 5.71. However, an IEP may be required under other programs such as the Department of Defense’s (DOD) Commercial Air Carrier Quality and Safety Requirements and the International Air Transport Association (IATA) Operational Safety Audit (IOSA). A robust IEP can facilitate data gathering for a Safety Management System (SMS) (if implemented) and quality assurance (QA) activities. Implementation of an IEP as outlined in this guidance material will assist in meeting such requirements. 2 AUDIENCE. This AC is directed to persons operating under 14 CFR who are designing, developing, and implementing an IEP. For ease of use, the term “aviation organization” is used throughout this AC and includes, but is not limited to, 14 CFR part 119 certificate holders, 14 CFR part 91 subpart K (part 91K) program managers, air agencies, and other persons operating under 14 CFR. 3 WHERE YOU CAN FIND THIS AC. You can find this AC on the FAA’s website at http://www.faa.gov/regulations_policies/advisory_circulars. 4 WHAT THIS AC CANCELS. AC 120-59A, Air Carrier Internal Evaluation Programs, dated April 17, 2006, is canceled. 5 DEFINITIONS. For more information on terms related to IEPs, see Appendix A, Definitions. 6 RELATED READING MATERIAL. For additional guidance on internal evaluation techniques and procedures, an Air Carrier Internal Evaluation Model Program Guide is available through the National Technical Information Service (NTIS) located at 5301 Shawnee Road, Alexandria, VA 22312. For more information, visit ### Page 2 http://www.ntis.gov or contact the NTIS at 703-605-6585 or email info@ntis.gov. 8.10 Information on other programs that are related to an IEP may be obtained at the following websites (see paragraph , which explains how these programs relate to an IEP): 6.1 IATA IOSA. IATA Operational Safety Audits website: http://www.iata.org/whatwedo/safety/audit/Pages/index.aspx. 6.2 DOD. U.S. Air Force Air Mobility Command Commercial Services website: http://www.amc.af.mil/Home/AMC-Commercial-Services/. 7 BACKGROUND. Aviation organizations are responsible for operational safety and are expected to have processes, policies, procedures, and documentation in place to ensure regulatory compliance. Additionally, it is to the advantage of every aviation organization to have an effective management system in place to ensure positive control of all operational activities. An IEP is a fundamental element to ensure compliance with external regulatory requirements, identify nonconformance to internal company policies and procedures, and identify opportunities to improve organizational policies, procedures, and processes. 7.1 Internal Evaluation and QA Programs. Within the international aviation community, the IEP is also synonymous with the QA function. QA programs are being expanded beyond aircraft Maintenance and Engineering (M&E) to include assessment of all operational functions in terms of regulatory compliance, and conformance to internal policies and procedures, to include evaluation of existing processes, procedures, and documentation. On a broader scale, several nations have published SMS guidelines in which the QA (or IEP) function is a fundamental element of the overall management system that includes organizational management, documentation, safety programs, QA, and emergency response planning. The thrust of such initiatives is to effectively integrate these functions into the management system. 7.2 Regulatory Noncompliance. As a matter of policy, the FAA encourages aviation organizations to identify, correct, and disclose instances of regulatory noncompliance. Therefore, the development and implementation of an IEP will benefit both aviation organizations and the flying public. For more information, refer to the current editions of AC 00-58, Voluntary Disclosure Reporting Program; and AC 121-37, Voluntary Disclosure Reporting Program – Hazardous Materials. 8 IEP ELEMENTS. An IEP is a high-level program that provides a means to maintain and refine the management system by continually monitoring the effectiveness and efficiency of processes and systems. An IEP applies various assessment and evaluation tools to derive information reporting how the management systems and key processes are meeting both internal quality and external regulatory standards. An IEP is based on the premise that the aviation organization will design and maintain an IEP that contains fundamental elements of quality and safety. An effective IEP should include the following fundamental elements: ### Page 3 8.1 Systems-Oriented Process. Internal evaluation should be a continual process that incorporates the techniques of inspections, audits, and evaluations to assess the adequacy of managerial controls and processes in critical systems and to continuously improve those systems based upon the results of regular evaluations. It should also include continuous monitoring and feedback of information on critical processes, and regular trend analysis of resulting data. 8.2 Beyond Regulatory Compliance. Internal evaluations should extend beyond regulatory compliance to determine the causes of deficiencies and detect and implement needed enhancements to company operating practices before deficiencies occur. 8.3 Independent. An IEP is an independent process that organizationally has straight-line reporting responsibility to senior management. 8.4 Defined Responsibility and Authority. The IEP should identify a person or group within the company who has the responsibility and authority to: 1. Establish and modify the IEP. 2. Perform evaluations, audits, inspections, and analysis of data as a part of an ongoing IEP. 3. Initiate, recommend, or provide corrective actions, including preventative action through designated reporting channels to address IEP audit findings. 4. Track and verify the implementation of corrective actions within specified timeframes. 5. Communicate and coordinate activities with FAA personnel on a regular basis. Note 1: With regard to the principals’ independence and authority, reporting lines are important. In some cases, internal evaluation activities may be centrally controlled under the leadership of a corporate QA manager. In other cases, QA managers may be embedded within the operational functions reporting to a respective senior manager. Note 2: The IEP description (plan/manual) should include an organizational diagram that depicts the independence of personnel who supervise and perform internal evaluation functions. This diagram should delineate the organizational chain of authority through which IEP audit results are communicated to senior management. 8.5 Senior Management Review. Senior management involvement in an IEP is crucial to program success. An effective program should include periodic senior management review of IEP audit results. 8.5.1 For the purposes of this program, senior management includes the chief executive officer (CEO), president, chief operating officer (COO), or an equivalent position that has the authority for action to resolve issues. ### Page 4 8.5.2 The management review should be accomplished on a regular basis appropriate to the scope of the IEP, but no less frequently than annually. The purpose is to review current performance and opportunities for improvement in an open discussion format to foster idea generation. An agenda should be prepared and distributed to participants at least 1 week prior to the meeting. At a minimum, the agenda should include: 1. Followup actions from previous management reviews, to include progress reports on the status of previously established improvement objectives. 2. A report on significant or systemic deficiencies, with associated status reports detailing corrective actions and planned followup activities. 3. Strategic or operational changes that could affect the IEP. 4. Establishment of new or revised objectives. 5. Recommendations for improvement and required resources. 8.5.3 Management reviews should be documented to include not only the content, but the resulting management-directed action items, if any. A fundamental concept of the IEP is that senior management is accountable for acting on the information it receives from the program. The IEP should include assessment of the effectiveness of the management review process and identify opportunities for improvement. 8.6 Feedback Loop. Quality control takes place by use of a feedback loop. Feedback is a dynamic process whereby output of a system is passed (fed back) to the input to continually achieve the desired results. Often, this is done intentionally to control the behavior of the system. The use of feedback to continually monitor the divergence between objectives and outcomes while implementing changes to reduce the variance is also known as a closed loop. An IEP should be designed to achieve quality control through closed-loop feedback. 8.6.1 An effective IEP provides quality-related information to the affected employee group and associated management team members. In addition, information derived from an IEP should be fed back into the operating departments from which the data was obtained. Effective corrective actions that were driven by a Root Cause Analysis (RCA) of findings may have applicability in other areas of the aviation organization. Therefore, opportunities for improvement across the various departments within the aviation organization may exist. 8.6.2 Quality-related information is similar to safety-related information typically provided to employee groups via a variety of channels. Feedback may include the sharing of best practices, typical system/process weaknesses, common management errors, etc. Such information may be disseminated via less formal channels, such as employee “Read and Sign” files, bulletins, newsletters, or websites, or it may be incorporated in structural changes, such as program/policy changes, enhancements to training, manual revisions, procedural changes, or personnel reorganization. ### Page 5 8.6.3 The method of feedback should be in accordance with the size and sophistication of the aviation organization. The process for this feedback loop should be defined in the IEP documentation. 8.7 Continual Process. To effectively anticipate problem areas and correct them before actual findings occur, an IEP should be a continual, ongoing process. An internal evaluation should be more than spot-check inspections of operating practices. Standalone, spot-check inspections do little more than identify symptoms of potential problems. 8.7.1 A continual process is needed to verify whether findings are isolated instances or actual symptoms of systemic policy, process, or procedural problems. An IEP should involve more than planned evaluations, tracking corrective actions, followup evaluations, and special evaluations based on identification of trends. It should also include ongoing data collection and analysis to identify opportunities for preventative action to preclude adverse events before they occur. A well-structured program ensures that all areas of operations are assessed at appropriate intervals. It also institutionalizes the process so that a change in personnel does not adversely affect the program. 8.7.2 A continuous process is equally important to identify problems that may otherwise be missed by periodic audits. Where appropriate, a continuing process of data collection and analysis can enhance the efficiency of the IEP by reducing the need for periodic audits, enabling a data-driven determination of when audits are needed, or providing the information needed to validate the effectiveness of corrective action previously initiated. 8.8 FAA Interface. 8.8.1 Program Assistance. The FAA does not approve or accept an IEP. However, the FAA is available to provide advice, assistance, or direction to aviation organizations interested in developing an IEP. Preparing program documentation (as discussed in paragraph 9) will provide the FAA with an opportunity to review the proposed duties, responsibilities, procedures, and organization of the IEP. 8.8.2 Sharing Reports. The FAA encourages aviation organizations to openly share the results of their IEP with the assigned Flight Standards office. Proprietary information contained in IEP reports/records should be maintained and secured on the aviation organization’s premises. If given to the FAA, proprietary information will be protected by the FAA in accordance with applicable laws and regulations. Sharing IEP information can enhance the working relationship between the FAA and the aviation organization. Information not required by regulation that remains on the aviation organization’s property would not normally be subject to public disclosure. 8.8.3 Disclosure of Findings to the FAA. For aviation organizations electing to voluntarily disclose apparent violations discovered by an IEP, further information is provided in AC 00-58 and AC 121-37. Under this policy, when an aviation organization finds a potential violation of the regulations, reports it to the FAA, and meets other specific conditions, the FAA will close this event with either corrective action or administrative action (i.e., a Letter of Correction (LOC)). Disclosures such as these require corrective 5 ### Page 6 action plans (CAP) designed to eliminate the underlying cause of the problem. Even though the internal evaluation process supports voluntary disclosures, it is not necessary to have an IEP to participate in the voluntary disclosure program, nor is it mandatory to disclose findings discovered during an internal evaluation. However, if the FAA discovers regulatory violations that were not disclosed, the aviation organization may receive enforcement action, or alternatively, Compliance Action if willing and able to cooperate with the FAA. 8.9 External Review. It is recommended that an external and independent review be done on the IEP to assess the effectiveness of the program. 8.10 IEP Relationship to Other Programs. 8.10.1 IATA IOSA Program. Since 2001, IATA has led the effort among the world’s leading airlines, regulatory authorities, and other industry participants to develop a standardized audit program based on internationally recognized standards called the IATA Operational Safety Audit (IOSA). The IOSA is an internationally recognized and accepted system designed to assess the operational management and control systems of an airline. The system employs internationally recognized quality audit principles conducted in a standardized manner. Audit standards are identified in the IOSA Standards Manual (ISM) and specific audit processes are described in the IOSA Program Manual (IPM). The IOSA and IEP have no direct interface. However, the IOSA standards are internationally recognized and may be used in the development and implementation of an effective IEP. 8.10.2 DOD Program. DOD Instruction 4500.53, Department of Defense Commercial Air Transportation Quality and Safety Review Program, charges the Commander of the United States Transportation Command (CDRUSTRANSCOM) with managing and executing the program. It also charges the CDRUSTRANSCOM with ensuring the establishment of a Commercial Airlift Review Board (CARB) and providing policy guidance and direction for its operation. Title 32 of the Code of Federal Regulations (32 CFR) Part 861, Department of Defense Commercial Air Transportation Quality and Safety Review Program, establishes DOD quality and safety criteria for air carriers providing or seeking to provide air transportation and, at the discretion of the CARB or higher authority, operational support services to the DOD. Part 861 also includes the operating procedures of the CARB. The CARB has the authority to suspend air carriers from DOD use or take other actions when issues of air carrier quality and air safety arise. Under the DOD program, civil air carriers and operators providing air transportation and operational support services must have an IEP acceptable to the DOD. 8.10.3 Continuing Analysis and Surveillance System (CASS). CASS is a quality management system that monitors and analyzes the performance and effectiveness of airworthiness inspection and maintenance programs. A CASS is required by 14 CFR part 121, § 121.373, and part 135, § 135.431 (for certificate holders subject to § 135.411(a)(2)), and part 91, § 91.1431. ### Page 7 8.10.3.1 As required by regulation, a CASS monitors a certificate holder/program manager’s inspection and maintenance programs for compliance with applicable requirements, including FAA regulations and manufacturer instructions. Each certificate holder/program manager subject to these regulations is required to establish and maintain a system for the continuing analysis and surveillance of the performance and effectiveness of its inspection program and the program covering other maintenance, preventive maintenance, and alterations and for the correction of any deficiency in those programs, regardless of whether those programs are carried out by the certificate holder/program manager or by another person. 8.10.3.2 FAA guidance on CASS is available in the current edition of AC 120-79, Developing and Implementing an Air Carrier Continuing Analysis and Surveillance System. An IEP is an independent program intended to provide information to senior management as to how well critical programs (e.g., CASS) are working. An IEP is not a substitute for CASS. An IEP poses questions necessary to determine if the certificate holder/program manager’s systems (e.g., CASS) are effective and efficient and if there are opportunities for improvement in those systems. 8.10.4 Safety Program. A safety program and an IEP are complementary functions within the management system. Both provide top management feedback regarding the health and effectiveness of the organization as a whole. 8.10.4.1 Safety programs include the capture and analysis of employee safety/hazard reports, the investigation of operational incidents and accidents, the oversight of risk assessment activities, and flight operations data assessment programs. As commonly practiced, safety programs are often reactive in nature in that they involve analysis of events that have already occurred (e.g., investigative efforts are oriented to identify root cause and establish corrective actions to prevent reoccurrence or limit frequency to acceptable risk levels). Focus of employee-reported safety/hazard reports should reveal information that, when utilized correctly, will concentrate on objective facts to identify system deficiencies, help prevent future recurrences, and improve system reliability. It is not as important to identify “who did it” as it is to learn why it happened. Within this process, it is important to distinguish between error and intentional/willful noncompliant actions. 8.10.4.2 In contrast, quality programs (or an IEP) are designed to systematically and proactively search for weaknesses and gaps in the management system, operational processes/procedures, or documentation. An effective IEP seeks to assure that key processes and controls are in place across the full spectrum of operational safety. The IEP is focused on verifying organizational compliance with all external regulatory requirements and internal organizational policies and procedures. These comprehensive system audits identify opportunities for improvement, which ultimately enhance safety through establishment of predictable and reliable business processes. ### Page 8 9 IEP DESCRIPTION. Aviation organizations that are interested in developing an IEP are encouraged to define and document the following essential elements in the program: 9.1 Program Documentation. IEP procedures and responsibilities should be documented. The following are suggestions for preparing and structuring program documentation. As an example, a sample program manual using the program elements discussed in this AC is provided in Appendix B, Sample Internal Evaluation Program Manual (14 CFR Part 121 Air Carrier). 9.1.1 Program documentation should describe the duties, responsibilities, procedures, and organization of the IEP. 9.1.2 Aviation organizations should review the size and complexity of their operation to determine the appropriate level of documentation. 9.1.3 Copies of the program documentation should be distributed to the appropriate company personnel so they are aware of and are familiar with the IEP procedures. 9.1.4 Revisions should be made as necessary to ensure that the program documentation continues to reflect the current internal evaluation procedures and organization. 9.2 IEP Scope. Consideration needs to be given to defining the scope of the program. For example, the carrier may include organizational management, flight operations and dispatch, station operations, In-Flight Services (IFS), aircraft maintenance, security, hazardous materials (hazmat), and ground handling. It is recommended that any business area that affects the integrity of the aviation organization should be considered. 9.3 Evaluation Standards. Aviation organizations should identify and communicate a specific standard(s) against which each functional area will be evaluated. Each documented standard should be developed with the assistance of the applicable management personnel. Once established, these standards should be clearly communicated and understood by the management team. A specific internal standard may be developed as a composite of multiple sources including, but not limited to: 1. External regulations (e.g., 14 CFR, Occupational Safety and Health Association (OSHA) regulations in Title 29 of the Code of Federal Regulations (29 CFR), and hazmat regulations in Title 49 of the Code of Federal Regulations (49 CFR)). 2. Internal company operating policies and procedures. 3. IOSA Standards and Recommended Practices (ISARP). 4. DOD Quality and Safety Requirements. 5. International Civil Aviation Organization (ICAO) requirements. 6. International Organization for Standardization (ISO) 9001: management elements (as amended). ### Page 9 9.4 Types of Evaluations. 9.4.1 Planned Evaluations. An effective program will establish a Schedule of Events (SOE) that will be performed during a set calendar period under the IEP. It is helpful to divide the complete schedule into segments that are practical from a workload standpoint. However, it is important to schedule evaluations to allow enough flexibility for resources to be committed to special evaluations or followup evaluations, if needed. 9.4.2 Special Evaluations. Conduct special evaluations based on concerns or priorities set by senior management. The need for special evaluations can be driven by such factors as a review of industry trends, FAA concerns, or adverse trends identified by the IEP or other internal programs. 9.4.3 Followup Evaluations. Schedule and conduct followup evaluations to ensure that corrective action commitments were met and to verify that corrective actions were effective. 9.5 Evaluation Schedule. It is essential for an IEP to include a defined schedule of planned activities. This schedule serves to verify that the IEP is comprehensive, well controlled, and timely. A published schedule also provides a vehicle for keeping management informed. An internal evaluation schedule should include a planned cycle for periodically reviewing areas to be covered by the IEP. The evaluation cycle can be flexible, but should not exceed 3 years. The scheduling process should also be dynamic and allow for special evaluations when trends are identified. In addition, followup evaluations should be scheduled as necessary to verify that corrective action commitments were met and that they were effective in eliminating any reported findings. 9.6 Corrective Action and Followup. An IEP should include procedures that ensure CAPs are developed in response to findings and that they verify timely and effective implementation of corrective action. Internal evaluation personnel may participate in the development of a CAP and should review the plan prior to implementation. However, organizational responsibility and accountability for the development and implementation of CAPs should reside with the operational departments cited in the finding. 9.6.1 A CAP should include: 1. A detailed description of the finding and how it was discovered. This should include the discussion of the scope and the extent of the problem so that solutions can be analyzed properly. 2. An analysis of the evidence to determine the root cause(s) of the finding. RCA treats errors as defects in the system rather than in a person. RCA looks beyond the symptom to find the organizational defect that permitted an error to occur. Its goal is to correct the fundamental problem and to prevent recurrence. The more thorough the analysis, the greater the likelihood the aviation organization will uncover why the system deficiency occurred and how the organization can respond effectively. ### Page 10 3. An identification of planned corrective actions to be taken in response to the finding to include specification of how, when, and where these actions will be taken. 4. An implementation schedule, including a timeframe for putting corrective actions in place. 5. An identification of the individual(s) who are assigned the responsibility for implementing each of the corrective steps. 9.6.2 The individuals responsible for managing an IEP should facilitate the corrective action process by: 1. Ensuring that CAPs are developed in a timely manner. 2. Verifying that CAPs include the items outlined above. 3. Monitoring and documenting implementation of corrective actions through resolution of the issue(s). 4. Providing senior management with an independent assessment of CAP development and implementation. 5. Initiating scheduled and/or unannounced followup evaluations to ensure the effectiveness of corrective actions specified in CAPs. 9.7 Reporting. The content of the detailed evaluation report should be defined in the program. An evaluation report should be sent to the responsible managers and the senior management for review. Briefings should be given to senior management and other responsible parties, as appropriate. Reports of the status of corrective actions should be provided, as appropriate. The aviation organization should decide upon the frequency, format, and structure for informing senior management of internal evaluation schedules, results, and followup actions to validate that corrective actions have been implemented and that they are effective. It is recommended that the reporting structure also be documented by the aviation organization and become a part of its program. 9.8 Records. The results of an internal evaluation should be documented in reports and other appropriate records consistent with the process of internal reporting at the aviation organization. Aviation organizations may find it useful to manage the results of IEP evaluations (as well as the other elements of the IEP) through a database or a quantitative application. IEP records should include: 1. Planned evaluation reports. 2. Special evaluation reports (including the trends or other reasons for scheduling a special evaluation). 3. CAPs. 4. Results of followup evaluations. 5. Records of auditor training and qualifications. ### Page 11 9.9 Auditor Training and Qualifications. If feasible, the aviation organization should specify that IEP auditors have training and/or experience in recognized quality management auditing, systems analysis, RCA, and risk assessment, as well as evaluation principles and techniques. Any one or combination of the following could accomplish training: 1. In-house prepared courses. 2. College courses. 3. Home study course materials. 4. Industry seminars and workshops. 5. Selected FAA courses. 9.10 Determination of Resources. To determine the resources needed by an IEP, senior management should make the scope of oversight and the extent of involvement for the IEP explicit. In most cases, the size of the organization and budgetary considerations will be the principal defining factors. 10 EXAMPLE IEP MANUAL. An example of an IEP manual that incorporates elements discussed within this AC is provided in Appendix B. An aviation organization’s IEP plan should be documented and may be a standalone manual or a portion of some larger department’s manual (e.g., the safety department’s manual). The example included in Appendix B may be more appropriate for a large part 121 air carrier. However, it can be scaled down and modified to fit other aviation organization’s needs. 11 AC FEEDBACK FORM. For your convenience, the AC Feedback Form is the last page of this AC. Note any deficiencies found, clarifications needed, or suggested improvements regarding the contents of this AC on the Feedback Form. ORIGINAL SIGNED by /s/ Rick Domingo Executive Director, Flight Standards Service 11 ### Page 12 Appendix A APPENDIX A. DEFINITIONS A.1 Audit. A methodical, planned review which builds on the principles of inspection. An audit is used to determine how business is being conducted and compares results with how business should have been conducted in accordance with established procedures. The most frequently used audit categories are (quality) system audits, service/product audits, and process audits. A.2 Audit Scope. The operational disciplines and/or operational areas that are assessed during the conduct of an audit. A.3 Auditor. An individual who has satisfied defined experience prerequisites and is successfully qualified under a defined training program to conduct audits. A.4 Authority. As a safety attribute, a clearly identifiable, qualified, and knowledgeable person who has the authority to set up and change a process. A.5 Concern. A conclusion by an auditor, supported by objective evidence, that is seen as a potential problem, trend, or inefficiency that may become a finding. A.6 Continual. A close, prolonged succession or recurrence, infinite in time, without interruption. A.7 Control. Key procedure, responsibility, or decision-making position within an organization, department, division, or functional area. Checks and restraints are designed into a process to ensure a desired result. Comprehensive evaluations (system audits) will focus on verifying and testing the controls within the organization. A.8 Corrective Action. The action(s) taken to correct a deficiency with the intent to preclude recurrence of the finding or noncompliance of an approved standard. A.9 Corrective Action Plan (CAP). The total plan of an aviation organization to close all findings through implementation of comprehensive corrective action. This plan should include the changes in policy and/or procedures that will ensure continued compliance. A.10 Evaluation. A functionally independent review of company policies, procedures, and systems. If accomplished by the company itself, the evaluation should be done by an element of the company other than the one performing the function being evaluated. The evaluation process builds on the concepts of audit and inspection. An evaluation is an anticipatory process, and is designed to identify and correct potential findings before they occur. An evaluation is synonymous with the term “systems audit.” A.11 Evaluation Standard. Specific criteria on which basis a functional area will be evaluated in terms of compliance or conformance. A.12 Evidence. A documented statement of fact, prepared by an aviation organization, which may be quantitative or qualitative, and is based on observations, inspections, measurements, or tests that can be verified. For the purpose of internal evaluation, A-1 ### Page 13 Appendix A evidence should generally be written documentation that supports an IEP’s analysis and review. This data is necessary to substantiate findings or concerns and to enable management or evaluators to determine the root causes of any reported findings. A.13 Finding. A determination as a result of an audit that compliance or conformance with an evaluation standard is not being achieved. A.14 Followup. A process involving monitoring of CAPs to verify timely and effective implementation designed to eliminate the underlying (root) cause of the deficiency. A.15 Inspection. The act of observing a particular event or action to ensure that correct procedures and requirements are followed during the accomplishment of that event or action. The primary purpose of an inspection is to verify that established standards are followed during an observed event or action. Note: The term “inspection” is defined in this AC within the context of quality auditing principles. It does not address or define FAA inspections. A.16 International Air Transport Association (IATA) Operational Safety Audit (IOSA). An internationally recognized evaluation system designed to assess the operational management and control systems of a part 121 air carrier, which is based on ICAO standards. A.17 Policy. A high-level overall plan embracing the general goals and acceptable practices of a group. Policies state how goals will be achieved. A.18 Procedures. Written or unwritten methods (regulatory or nonregulatory) that an aviation organization uses to accomplish a particular process. A.19 Process. A set of interrelated resources and activities that transform inputs to outputs. Resources may include personnel, finance, facilities, equipment, techniques, and methods. A.20 Process Audit. A documented activity that assesses the effectiveness and efficiency of a series of related or sequential work activities. A.21 Quality Assurance (QA). The independent activity of providing the evidence needed to establish confidence, among all concerned, that the quality function is being performed effectively. This activity “assures quality” through independent evaluation of established processes, procedures, and documentation. A.22 Quality Control. The determination of the quality of a product by inspection and testing to determine compliance with standards. This activity “controls quality” through establishment of effective controls, documentation, and procedures within specific functional areas. A-2 ### Page 14 Appendix A A.23 Quality Management System. A defined organizational structure, written management responsibilities, and associated system of processes, procedures, and detail documents to ensure compliance with internal standards or regulatory requirements. A.24 Quality System Audit. A documented activity performed to verify by examination and evaluation of objective evidence that applicable elements of the quality management system are documented and effectively implemented in accordance with specific requirements. A.25 Responsibility. As a safety attribute, a clearly identifiable, qualified, and knowledgeable person who is accountable for the quality of a process. A.26 Root Cause Analysis (RCA). Determination of what caused a finding. The identification of the root cause is the key to the implementation of an effective corrective action. Items to consider when determining the root cause should include deficiencies in training, materials, procedures, empowerment, or management oversight. A.27 Senior Management. The highest level of management within an organization that has the authority and responsibility for setting policy, demonstrating commitment, meeting requirements, approving resources, setting objectives, implementing processes, and achieving desired outcomes. A.28 Service/Product Audit. An objective and structured assessment of conformance to required service-level or product-level performance characteristics. It may be qualitative or quantitative, as appropriate. A-3 ### Page 15 Appendix B APPENDIX B. SAMPLE INTERNAL EVALUATION PROGRAM MANUAL (14 CFR PART 121 AIR CARRIER) B.1 Overview. B.1.1 The IEP will provide operational departments a comprehensive method to continually monitor internal processes, programs, and procedures to ensure that each operating department remains in compliance with appropriate company policies and procedures and applicable Federal, state, and local regulations. B.1.2 The IEP is mandatory within this carrier for flight operations, In-Flight Services (IFS), Maintenance and Engineering (M&E), and customer service. Other departments are encouraged to develop similar programs to monitor departmental performance and compliance. B.2 Program. B.2.1 Key Responsible Personnel. B.2.1.1 Director of Safety (DOS). The DOS has overall responsibility to develop and implement a comprehensive safety program. B.2.1.1.1 The duties and responsibilities of the DOS include: 1. Monitoring and reporting to senior management on all air carrier activities that may have an impact on safety. 2. Establishing a reporting system that provides for a timely and free flow of safety-related information. 3. Developing and maintaining a database of incident/accident information to monitor and analyze trends. 4. Monitoring and evaluating the various safety and malfunction-reporting systems to ensure appropriate integration and evaluation of data. 5. Investigating and reporting incidents/accidents and making recommendations to preclude a recurrence. 6. Conducting safety audits and inspections. 7. Soliciting and processing safety improvement suggestions. 8. Developing and maintaining a safety awareness program. 9. Reviewing and evaluating the adequacy of the emergency response plan. 10. Monitoring industry safety concerns that may have an impact on operations. B-1 ### Page 16 Appendix B 11. Maintaining close liaison with the FAA, National Transportation Safety Board (NTSB), and industry safety organizations and associations. 12. Ensuring that the necessary safety program elements have been developed, properly integrated, and coordinated throughout the air carrier. These elements include: • A safety incident/accident reporting system. • Accident/incident investigation. • Safety audits and inspections. • IEP. • Operational risk assessment program. • Open reporting systems. • Routine monitoring and trend analysis programs. • Review of external evaluation programs. • Safety committee(s). 13. Discharging duties as required to meet applicable legal requirements and to maintain safe operations in accordance with part 119, § 119.65. B.2.1.1.2 DOS Qualifications. The DOS should meet the qualification requirements as outlined in FAA Order 8900.1, Volume 2, Chapter 2, Section 3, Safety Assurance System: Evaluate Part 121/135 Management Personnel. The DOS should have extensive operational experience and professional qualifications in aviation. This includes the knowledge and understanding of the following: 1. Aviation safety programs. 2. Aviation safety standards. 3. Safe aviation operating practices. B.2.1.1.3 DOS Expertise. The DOS should have established professional qualifications. These qualifications may be any of the following: 1. An FAA Commercial Pilot or Airline Transport Pilot (ATP) Certificate. 2. An FAA Mechanic Certificate. 3. An FAA Aircraft Dispatcher Certificate. 4. Three years of supervisory experience in any combination of the following: • Supervisory position with a 14 CFR part 121 or a scheduled part 135 air carrier; • Supervisory position comparable to the above in the U.S. military; B-2 ### Page 17 Appendix B • Supervisory position with a U.S. Government department, board, or agency that deals directly with aviation matters; or • Supervisory position with a part 91 or 125 operations/aviation department. B.2.1.1.4 DOS Knowledge. The DOS should have a full understanding of the following materials with respect to the airline’s operation: 1. Aviation safety standards and safe operating practices. 2. Title 14 CFR parts 1–199. 3. The airline’s operations specifications (OpSpecs). 4. The manual required by part 121, § 121.133. 5. All appropriate maintenance and airworthiness requirements of parts 1–199. B.2.1.1.5 DOS Authority. The DOS has the authority to establish and modify the policies and procedures associated with the airline’s IEP. Proposals for modifications to the IEP may be submitted by any manager or employee. Proposals will be submitted through the Director of Internal Evaluation (DIE), who will seek and obtain approval from the DOS prior to implementation. Such modifications should be documented within the IEP database (see paragraph B.2.3 concerning IEP data management). B.2.1.2 Director of Internal Evaluation (DIE). The DIE has overall responsibility for the day-to-day management of the IEP. This person will serve as the chairman of the Internal Evaluation Review Board (IERB) with regard to IEP management and control. The DIE will develop and manage the IEP to ensure compliance with the written program as outlined in this manual and will seek to continually improve and enhance the program. B.2.1.2.1 The duties and responsibilities of the DIE include: 1. Developing, implementing, and maintaining an IEP in accordance with the guidance contained in the current edition of FAA Advisory Circular (AC) 120-59, Internal Evaluation Programs. 2. Developing an annual evaluation schedule through coordination with the IERB. 3. Ensuring all evaluations are conducted on schedule (planned start date plus or minus days). 4. Reviewing all evaluation information and preparing a summary of all evaluations for the IERB. B-3 ### Page 18 Appendix B 5. Preparing finding reports for submission to each operating department with evaluation findings. 6. Reviewing departmental corrective action plans (CAP) for accuracy and effectiveness. 7. Tracking both findings and corrective actions to ensure followup and completion. 8. Preparing a monthly summary of evaluation findings and CAPs for the DOS. 9. Preparing quarterly and annual reports of the IEP for senior management and the Board of Directors (BOD) Safety Committee. 10. Maintaining the electronic database of evaluation information and audit findings. 11. Assuming the duties of the DOS position in the event that the incumbent is unavailable to perform those duties. B.2.1.2.2 DIE Qualifications. 1. Have a full understanding of the airline’s aviation safety standards and safe operating practices; parts 1–199; the certificate holder’s OpSpecs; and the manual(s) required by § 121.133. 2. Have supervisory or managerial experience within flight operations, M&E, station management, customer service, or equivalent experience in the transportation industry. 3. Have successfully completed a recognized comprehensive safety certificate program (this qualification is preferred). 4. Be familiar with the Occupational Safety and Health Administration (OSHA), the Environmental Protection Agency (EPA), and other Federal safety regulations. 5. Be familiar with accepted auditing techniques. B.2.1.3 Manager of IEPs. The Manager of IEPs reports to the DIE. The Manager of IEPs will oversee and coordinate the efforts of the Supervisors of Internal Evaluation (SIE) for the performance of evaluations and audits of systems, processes, operations, stations, and facilities. B.2.1.3.1 The responsibilities of the Manager of IEPs include: 1. Scheduling audits and evaluations in accordance with the IEP guidelines. 2. Conducting audits and evaluations of stations and facilities. 3. Conducting annual oversight evaluations of operating departments. B-4 ### Page 19 Appendix B 4. Managing the subordinate staff within the department. 5. Compiling and analyzing audit and evaluation findings for root cause identification. 6. Recommending corrective actions pursuant to findings. 7. Tracking corrective actions to completion. 8. Maintaining a database of audit materials, checklists, findings, root causes, corrective actions, and analysis. 9. Preparing reports for the DIE, the DOS, the IERB, and senior management. 10. Managing maintenance of the IEP Company Manual. B.2.1.3.2 Manager of IEPs Qualifications. 1. Have a full understanding of the airline’s aviation safety standards and safe operating practices; parts 1–199; the certificate holder’s OpSpecs; and the manual(s) required by § 121.133. 2. Have a minimum of 2 years’ experience in the airline industry in station or operations management. 3. Be familiar with accepted auditing techniques. 4. Be familiar with database management. B.2.1.4 Supervisors of Internal Evaluation (SIE). The SIEs report to the Manager of IEPs. These positions will perform the inspections, audits, and evaluations of the airline’s stations, facilities, and operating departments in accordance with the IEP guidelines. B.2.1.4.1 The responsibilities of the SIEs include: 1. Conducting scheduled and unscheduled inspections, audits, and evaluations as directed by the Manager of IEPs. 2. Reporting findings related to inspections, audits, and evaluations. 3. Tracking corrective actions (related to findings) to completion. 4. Maintaining files, records, and data related to the IEP. 5. Analyzing findings and deficiencies for causal factor and root cause identification. B.2.1.4.2 SIE Qualifications. 1. Have an understanding of the airline’s aviation safety standards and safe operating practices; parts 1–199; the certificate holder’s OpSpecs; and the manual(s) required by § 121.133. B-5 ### Page 20 Appendix B 2. Have knowledge of airport operations, cargo, security, and hazmat. 3. Have a minimum of 2 years’ experience in the airline industry. 4. Have a current or past qualification as a Ground Security Coordinator (GSC). B.2.2 Guidance. The IEP will be established and maintained in accordance with § 119.65 and the current edition of AC 120-59. B.2.3 Data Management. The airline has purchased and installed a database system to serve as a repository for all data related to the IEP. This is a networked system accessible to specified employees within the Safety Division who have been granted permission to perform certain operations within the system. Written documentation regarding the operation and functionality of the system is maintained within the IEP department by the DIE, and is stored electronically within the database itself. Training related to the operation and maintenance of the system is provided by the manufacturer, and a recurrent user seminar is conducted annually. Functions and capabilities of the system include, but are not limited to: 1. Evaluation and audit schedules. 2. Organization structure. 3. Tracking of staff assigned to audits and evaluations. 4. Tracking of time spent by staff on audits and evaluations. 5. Tracking by user identification of all actions related to the conduct of audits and evaluations. 6. Storage and tracking of all findings related to audits and evaluations. 7. Audit and evaluation scope, objective, and frequency. 8. Assignment of responsibility and timeline for responses to findings. 9. Forwarding (via email) of findings to the designated party responsible for corrective action. 10. Identification of overdue audits and evaluations. 11. Identification of overdue finding responses. 12. Requirement that all findings are resolved before audit/evaluation status can be changed to “closed.” 13. Automatic scheduling of followup audit/evaluation upon closure. 14. Automatic generation of audit/evaluation/finding reports. 15. Tools for classification of audit/evaluation results with regard to risk, quality, conformity, and effectiveness. 16. Tools for causal factor analysis of findings. B-6 ### Page 21 Appendix B 17. Tools for trend analysis of findings. 18. Storage of files related to audit/evaluation accomplishment, findings, corrective actions, analysis, trending, root cause identification/correction, audit/evaluation review, and closure. B.2.4 Document Control. In accordance with the provisions of the Management Policies and Procedures Manual, audit checklists and work papers will be retained on file for a period of 2 years. Audit reports will be retained in active files for a period of 2 years, then archived for 8 years. In order to enable long-term analysis, derivative data, findings, corrective actions, causal factors, Root Cause Analysis (RCA), and trending, information will be retained within the database indefinitely. B.2.5 Audit Objectives and Scope. A statement of the objective and scope of each audit is contained in the checklist and the audit report. The following are areas of ongoing oversight: B.2.5.1 Station evaluations will be performed throughout the year, in addition to the annual oversight evaluations of operating departments. These evaluations will include checks of manuals, training, cargo, hazmat, security, loading, and records. Master checklists are on file in the IEP department. Internal Evaluation personnel will conduct these evaluations. B.2.5.2 Maintenance operations evaluations will be conducted throughout the year by the Quality Assurance (QA) Department. These evaluations will be reviewed during and included in the annual oversight evaluation of M&E. Maintenance QA audits and records will also be reviewed during the annual oversight evaluation. B.2.5.3 There are classifications of findings and tools for quality, conformity, effectiveness, and risk assessment. Findings will be entered under the following classifications: 1. NCP—Noncompliance (with regulations). 2. NCF—Nonconformance (with documented procedures). 3. SRC—Safety-Related Concern (currently in compliance and conformance, but the problem may have safety implications). 4. QRC—Quality-Related Concern (currently in compliance and conformance, but the problem indicates a weakness in the quality system). 5. OBS—Observation (comment). B.2.5.4 Figure B-1, Risk Assessment Matrix, will be used to determine the risk factor associated with a given finding. An NCP finding should be brought by the DOS to the immediate attention of the Vice President of the responsible operating department in order to enable timely submission of a voluntary B-7 ### Page 22 Appendix B disclosure to the FAA. NCP and NCF findings will require a written CAP to be issued no later than 15 days from the presentation of the finding report to the audited entity. OBS, SRC, and QRC findings that have a risk factor of 2 will require a written CAP to be issued no later than 30 days from the presentation of the finding report to the audited entity. Any finding that has a risk factor of 1 will require immediate corrective action and notification of the operating department Vice President. FIGURE B-1. Risk Assessment Matrix RISK ASSESSMENT MATRIX Severity Likelihood Catastrophic Critical Marginal Negligible Frequent 1 1 2 2 Probable 1 1 2 3 Occasional 1 2 2 3 Remote 2 2 3 3 Severity Scale Definitions Catastrophic Accident with serious injuries and/or fatalities. Loss (or breakdown) of an entire system or subsystem. Critical Accident or serious incident with injuries and/or moderate damage to aircraft. Partial breakdown of a system or subsystem. Marginal Accident or incident with minor injury and/or minor aircraft damage. System deficiencies leading to poor air carrier performance or disruption to air carrier schedules. Negligible Less than minor injury and/or less than minor system damage. Little or no effect on system or subsystem. Likelihood Scale Definitions Frequent Will be continuously experienced unless action is taken to change events. Probable Will occur often if events follow normal pattern. Occasional Potential for infrequent occurrence. Remote Not likely to happen (but could). B-8 ### Page 23 Appendix B Risk Classification 1 High Risk – Unacceptable; requires action. 2 Medium Risk – May be acceptable with review by appropriate authority; requires tracking and probable action. 3 Low Risk – Acceptable without further action. B.2.6 Unique Terms. A glossary of acronyms used throughout this manual is included in paragraph B.6. B.2.7 Independence. The DIE reports directly through the DOS to the President and CEO and to the Safety Committee component of the BOD. Figure B-2 depicts this reporting structure. Figure B-2. Internal Evaluation Reporting Structure B-9 ### Page 24 Appendix B B.2.8 Top Management Review. The Safety Division and the IERB will report all audit findings on an annual basis to senior management. B.2.9 Senior Management. The CEO and President constitute as senior management. When making the report to senior management, the Safety Division and the IERB will include all audit information and findings in the report. Senior management will be actively involved in the review process to ensure audit program effectiveness and provide oversight and guidance to operating departments. The senior management’s involvement includes review of all audits and corrective actions through periodic meetings, briefings, or written reports to assess the effectiveness of each operating department’s CAP and audit program. B.2.10 Schedule. Flight operations, M&E, customer service, and IFS will receive a focused quarterly evaluation (e.g., first quarter records, second quarter manuals, etc.). The schedule for the evaluations of these areas will be determined during the first quarter of each calendar-year, and will be entered into the system by IEP personnel. All stations within the airline’s system will be evaluated three times per year on the following basis: 1. “A” Stations (those with more than 1 million passenger enplanements per year) will be evaluated three times per year by IEP personnel. 2. “B” Stations (those with more than 250,000 but fewer than 1 million passenger enplanements per year) will be evaluated twice per year by IEP personnel, and once per year on a self-audit basis. The self-audit will be forwarded to the IEP department for entry into the database and followup activity tracking. 3. “C” Stations (those with fewer than 250,000 passenger enplanements per year) will be evaluated once per year by IEP personnel, and twice per year on a self-audit basis. The self-audits will be forwarded to the IEP department for entry into the database and followup activity tracking. 4. A list of stations denoting their status (e.g., A, B, or C) will be maintained by the IEP department and will be updated based on actual passenger enplanement records from the previous calendar-year. 5. The schedule for the evaluations and self-audits will be entered into the system and tracked by Internal Evaluation personnel. The IEP department will maintain evaluation and self-audit checklists. 6. Other evaluation and audit activity will be carried out by IEP personnel on an event-driven basis or by request from senior management. B.2.11 CAPs. All CAPs will be tracked to completion in the system. Audits will remain in an “open” status until all corrective actions have been completed. B.2.12 RCA. The airline has selected the XYZ system for use by the IEP team to perform RCA and to aid in the development of corrective actions for deep-rooted problems that are discovered during the course of inspections, audits, and evaluations. All IEP personnel will receive training in the use of this system. B-10 ### Page 25 Appendix B B.2.13 Training. All IEP personnel who conduct evaluations will receive training including, but not limited to: 1. A basic auditing skills course given by an accredited organization. 2. An operational auditing skills course given by an accredited organization. 3. Training on the XYZ system to perform RCA. 4. Database training (conducted by the database manufacturer). 5. GSC training (conducted by airline personnel). 6. Complaint resolution official training (conducted by airline personnel). 7. Hazmat acceptance training (conducted by airline personnel). 8. On-the-job performance training of audits, evaluations, and interviews (conducted by IEP personnel). Note: All training will be documented in a file maintained within the IEP department. IEP personnel will receive the minimum training within 6 months of beginning duties in the IEP department. B.2.14 Resources. The DIE will prepare a budget annually for approval by the DOS. This budget will include sufficient funds to support the personnel, equipment, infrastructure, and activities of the IEP department. This operating budget will be developed, submitted, and approved in accordance with the guidance contained in the Management Policies and Procedures Manual. B.2.15 Audits. The IEP will provide a formal, continuous self-evaluation of the airline’s operations to enhance system safety and ensure continual improvement and compliance. Each operating department will be subject to a formal IEP, which will encompass all stations, facilities, domiciles, line stations, and maintenance bases under the operating departments’ control. The IEP describes a comprehensive program of audits that is conducted by operating departments and the Safety Division and the IERB to measure the effectiveness of internal programs and processes. Department-specific audit programs will be developed to review all internal processes, programs, and procedures. All departmental IEPs are outlined in later sections of this manual. This program will be accomplished through a series of self-audits or site audits that will be coordinated and managed by the Safety Division and the IERB. B.2.16 Tools. Examples of audit tools include an audit checklist, employee performance evaluations (pilot and flight attendant (F/A) check or evaluation rides), aggregate de-identified flight operations quality assurance (FOQA) data, Aviation Safety Action Program (ASAP) information, results of safety inspections, employee surveys, and any other assessment tools developed by individual departments or by the Safety Division and the IERB that measure performance and compliance. Other resources used in the development of individual audit programs include 14 CFR regulations, 49 CFR Department of Transportation (DOT) regulations, the National Aviation Safety Inspection B-11 ### Page 26 Appendix B Program (NASIP) Audit Checklist, the DOD Air Carrier Survey Office Maintenance Checklist, and the DOD Operations Survey Checklist. B.2.17 Recordkeeping and Administration. All audit tools and associated evaluation results used during the Internal Evaluation process will be retained on file in the Safety Department and be made available to the Safety Division and the IERB for review during the Safety Division and the IERB’s ongoing review process. B.2.18 Checklist Completion. The audit checklist will be completed in accordance with the annual schedule for a given operating department. Status of unresolved issues or discrepancies will be forwarded as part of the annual audit report to senior management (normally at year’s end). B.3 Operating Department Programs. Each operating department’s IEP is described below. The programs have been designed to monitor all critical internal departments and major functional areas. As described above, individual audit programs may be unique and will reflect the organization’s structure and mission. In order to ensure program standardization and to assure a level of continuity among the individual programs, some minimum elements have been established for mandatory inclusion in the individual departmental audits. Due to differences in mission and operational responsibility, a unique minimum element list has been provided for each major operating department and is included in the departments’ programs below. B.3.1 Flight Operations (Includes Dispatch, Crew Scheduling, and Systems Operations Control). The DOS and the IERB have overall responsibility for the flight operations IEP. The DIE is primarily responsible to the DOS for the execution and administration of the IEP and will advise the DOS regularly on the status of the program and of any findings or considerations discovered during the process. Audit checklists have been developed and continuous ongoing improvements will be incorporated into these documents, as conditions require. The flight operations IEP will include a process to analyze and improve flight operations policies and procedures at all levels of the organization. B.3.1.1 Minimum Elements. The list of elements provided below represents the minimum items that must be included and evaluated in the flight operations IEP: 1. Regulatory compliance (include an assessment of administrative/enforcement actions by Government regulatory agencies). 2. Compliance with air carrier programs and procedures. 3. Review of manuals for currency, accuracy, and maintenance. 4. Accident/incident/injury rate. 5. Aircraft accident/incident procedures. 6. Communication processes between employees and management. 7. Employee surveys. B-12 ### Page 27 Appendix B 8. Vendor/supplier audit program for outsource crewmember training. 9. OpSpecs. 10. Safety program (departmental). 11. Flightcrew standardization. 12. FOQA data (de-identified trend information). 13. ASAP information. 14. Captain upgrade. 15. Dispatch and flight release. 16. Personnel training and qualification. 17. Training program. 18. Crewmember (flightcrew) qualification. 19. Dispatcher training. 20. Flight simulation training devices (FSTD). 21. Hazmat training and procedures. 22. Special airport and route qualifications. 23. Scheduling and reporting system. 24. Crewmember (flightcrew) flight, rest, and duty time procedures. 25. Dispatcher duty/rest time. 26. Operational release. 27. Technical administration. 28. Check pilot program. 29. Aircrew program designees. 30. Minimum equipment list (MEL)/Configuration Deviation List (CDL) procedures. B.3.2 IFS. The DOS and the IERB have overall responsibility for the IFS IEP. The DIE is primarily responsible to the DOS for the execution and administration of the IEP and will advise the DOS regularly regarding the status of the program and of any findings or considerations discovered during the audit process. Audit checklists have been developed and continuous and ongoing improvements will be incorporated into these documents, as conditions require. The IFS IEP will include a process to analyze and improve IFS policies and procedures at all levels of the organization. B.3.2.1 Minimum Elements. The list of elements provided below represents the minimum items that will be included and evaluated in the IFS IEP. The IEP will employ company-developed tools and standards for these elements: B-13 ### Page 28 Appendix B 1. Regulatory compliance (include an assessment of administrative/enforcement actions by Government regulatory agencies). 2. Compliance with air carrier programs and procedures. 3. Review of manuals for currency, accuracy, and maintenance. 4. Aircraft accident/incident procedures. 5. Accident/incident/injury rate. 6. Communication processes between employees and management. 7. ASAP information. 8. Employee surveys (when available). 9. Intoxicated passengers screening and reporting. 10. Carry-on baggage. 11. Exit row seating. 12. Outsource crewmember training. 13. Crewmember standardization 14. Crewmember safety assessment (in-flight observations). 15. Training of F/As. 16. Crewmember qualifications. 17. Hazmat training. 18. Procedures required by the Hazardous Materials Program. 19. F/A rest and duty time procedures. 20. Technical administration. B.3.2.2 IFS Additional Audit Items. The following items will be evaluated as part of the IFS Audit Program: 1. Cabin crewmember flight, rest, and duty time procedures. 2. Scheduling practices and compliance. 3. Outsourced in-flight crewmember training program. 4. List of current vendors. 5. Flight report data management. 6. Accident and incident rates and related statistics. B.3.3 M&E. The DOS and the IERB have overall responsibility for the M&E IEP. The DIE is primarily responsible to the DOS for the execution and administration of the IEP and will advise the DOS regularly regarding the status of the program and of any findings or B-14 ### Page 29 Appendix B considerations discovered during the process. Audit checklists have been developed and continuous ongoing improvements will be incorporated into these documents, as conditions require. The M&E IEP will include a process to analyze and improve M&E policies and procedures at all levels of the organization. B.3.3.1 Minimum Elements. The list of elements provided below represents the minimum items that must be included and evaluated in the M&E IEP: 1. Aircraft airworthiness. 2. Records and reporting system. 3. Maintenance organization. 4. Continuing Analysis and Surveillance System (CASS). 5. Regulatory compliance (include an assessment of administrative/enforcement actions by Government regulatory agencies). 6. Compliance with air carrier programs and procedures. 7. Review of manuals for currency, accuracy, and maintenance. 8. MEL/CDL procedures. 9. Aircraft accident/incident procedures. 10. Accident/incident/injury rate. 11. Communication processes between employees and management. 12. ASAP information. 13. Employee surveys (when available). 14. Personnel training and qualification. 15. Hazmat training for maintenance personnel. 16. Hazmat procedures for maintenance personnel. 17. Maintenance duty time procedures. 18. Technical administration. B.3.4 Customer Service. The DOS and the IERB have overall responsibility for the Customer Service IEP. The DIE is primarily responsible to the DOS for the execution and administration of the Customer Service IEP and will advise the DOS and the IERB regularly regarding the status of the program and of any findings or considerations discovered during the audit process. B.3.4.1 Audit checklists have been developed and continuous and ongoing improvements will be incorporated into these documents, as conditions require. The Customer Service IEP will include a process to analyze and improve customer service policies and procedures at all levels of the organization. B-15 ### Page 30 Appendix B B.3.4.2 Customer service must maintain documentation (including CAPs) that it provides to substantiate answers to audit questions. Revisions to any such documentation must be forwarded to the Safety Division and the IERB for inclusion in the audit file. B.3.5 Audits. The Customer Service IEP is not a single, standalone audit. It is a collection of regularly scheduled station audits, self-audits, and onsite audits conducted throughout the year. The responsibility for collecting, analyzing, and forwarding of audit information is the responsibility of the DIE. All audit findings will be routed through the Vice President of Customer Service for corrective action and followup. The DIE will compile and forward an annual audit report in accordance with this manual, normally at year’s end. B.3.5.1 Minimum Elements. The list of minimum elements provided below represents the minimum items that must be included and evaluated in the Customer Service IEP: 1. Hazmat. 2. Training records. 3. Weight and Balance (W&B). 4. Procedures manual. 5. Station files. 6. Flight files. 7. Security. 8. Training program. 9. Station personnel. 10. Mandated training. 11. Carriage of cargo. 12. Station facilities. 13. Station manuals. 14. Passenger handling. 15. Carry-on baggage. 16. Exit row seating. B.4 Process. B.4.1 The Audit. Each department may be subject to either a single annual audit or a continual progression of audits throughout the year based upon the department’s operational requirements and structure. A detailed audit schedule will be developed each calendar-year by the DIE. This schedule will be maintained within the database system. B-16 ### Page 31 Appendix B B.4.1.1 Departments that are monitored by continual running audits will be subject to an annual audit review. During this review, all individual audits will be collected and reviewed by management. The collected audits will be combined and analyzed, and a CAP (where required in accordance with the criteria in this manual) will be developed for each finding and forwarded to senior management for review. The same process will be followed for departments that are under an individual annual audit program. B.4.1.2 In all cases, audit information will be analyzed by the operating department and by the Safety Division and the IERB, and (where required by this manual) a CAP will be developed. Senior management will be briefed on the audit results and the plan of corrective action. During the conduct of either style of audit, auditors should apply the appropriate classification for risk assessment (Figure B-1) and for quality, conformance or effectiveness (NCP, NCF, QRC, SRC, or OBS) to each audit finding. When Risk Factor 1 or NCP findings are identified, the Vice President of the appropriate operating department should be notified without delay. Every effort must be made to correct or eliminate Risk Factor 1 and NCP findings immediately. B.4.1.3 Auditors must ensure that when high-risk-factor findings cannot be corrected on the spot, adequate precautions are taken to prevent injury or accidents. When notified of apparent violations of the regulations, management must evaluate each finding and decide if self-disclosure in accordance with the current edition of AC 00-58, Voluntary Disclosure Reporting Program, is warranted. B.4.2 Analysis. The analysis phase begins immediately following the completion of an annual audit or at the conclusion of the annual review. Data from either an annual audit or continuing audits are combined and an analysis of findings completed. During this phase, all findings must be identified and cataloged by the IEP department. All departments will receive reports from the Safety Division and the IERB of findings discovered during audit processes. An audit trail must be maintained throughout the audit program. Documentation will be presented to senior management and forwarded to the Safety Division and the IERB as part of the final report. The focus of data analysis should be to determine the root cause and risk. B.4.3 Corrective Action. After the analysis of an audit has been completed and reviewed by the IERB, the operating department will be required to provide a detailed CAP for each finding. Departments will prepare a written plan, which outlines corrective action for each audit finding. This plan will be part of the information presented to senior management and forwarded to the Safety Division and the IERB as part of the final report. An effective method must be developed to monitor, measure, and validate the effectiveness of corrective actions taken by the department. Corrective action may take many forms. It may be as simple as a policy letter or require indepth changes to training programs, manuals, or procedures. Regardless of the form or type of corrective action planned, an ongoing evaluation of the action’s effectiveness must occur. Each new or subsequent audit should evaluate and consider the effectiveness of the corrective action B-17 ### Page 32 Appendix B steps taken during prior audits. A continual review of the corrective action steps assures a level of internal oversight as well as providing a method of validating CAPs. B.4.4 Audit Report Resolution. After the analysis phase and the CAP development phase have been completed, and not more than 90 days after an audit is completed, a formal report of audit findings, corrective action, and methods planned for tracking findings through resolution will be presented to senior management. This plan will outline specific corrective action and provide a timeline for completion. For departments that run year-long programs, an annual CAP is due by year’s end. This report should be a consolidated summary of all findings and corrective action steps with specific timelines for resolution. B.4.5 Senior Management Review. Each December, a senior management review will be conducted of all audit information from the preceding year. Each department will be required to present a status report of all IEP findings and corrective measures taken. Corrective actions will be evaluated during the following year’s audit to validate completion and effectiveness. These findings will be presented to senior management at the annual meeting by the DIE. B.4.6 Employee Feedback. The most recent results of IEP audits for a given functional area will be published quarterly in the appropriate employee group newsletter (e.g., flight operations, maintenance, IFS, or customer service). Results will include progress reports on corrective actions for previous audit findings. In addition, company-wide IEP findings and status reports (as well as audit schedules) will be maintained on the IEP website that is accessible to all employees through the company intranet. B.5 Oversight (Program Effectiveness). The Safety Division and the IERB are responsible for oversight of the IEP with regard to storage of audit information and for periodic followup with operating departments. Each department will forward complete audit information to the Safety Division and the IERB after reporting this information to senior management. B.5.1 Open Audit Findings. The Safety Division and the IERB will track audit findings that remain open or require followup action using the system. The Safety Division and the IERB will periodically review CAPs and determine if they are being completed as scheduled and are effective in correcting or eliminating the audit findings. A designated Safety Manager will be assigned oversight responsibility for each operating department. This manager will monitor progress within his or her assigned department with regard to corrective actions taken within the department and the effectiveness of that action. In addition, the assigned manager will perform an annual safety audit of the operating department. During that audit, a review of the department’s IEP will be conducted. B.5.2 Analysis Phase and Action Plan. The operating department’s analysis phase and action plan will be considered and evaluated with regard to progress made toward resolution of audit findings. The results of safety audits will also be forwarded to senior management for review. In addition, each department will implement (where feasible) procedures for continuous measurement and analysis of safety-critical processes between audits and B-18 ### Page 33 Appendix B evaluations. In order to provide an additional level of oversight to the IEP, the Internal Audit Department will add the IEP to its list of audited entities and will appraise the adequacy and effectiveness of the program’s internal controls on a biennial basis. B.6 Acronyms Used in This Manual. 14 CFR Title 14 of the Code of Federal Regulations 49 CFR Title 49 of the Code of Federal Regulations AC Advisory Circular ASAP Aviation Safety Action Program ATP Airline Transport Pilot BOD Board of Directors CAP Corrective Action Plan CASS Continuing Analysis and Surveillance System CDL Configuration Deviation List CEO Chief Executive Officer DIE Director of Internal Evaluation DOD Department of Defense DOS Director of Safety DOT Department of Transportation EPA Environmental Protection Agency F/A Flight Attendant FAA Federal Aviation Administration FOQA Flight Operations Quality Assurance FSTD Flight Simulation Training Device GSC Ground Security Coordinator hazmat Hazardous Materials IEP Internal Evaluation Program B-19 ### Page 34 Appendix B IERB Internal Evaluation Review Board IFS In-Flight Services M&E Maintenance and Engineering MEL Minimum Equipment List NASIP National Aviation Safety Inspection Program NCF Nonconformance (with documented procedures) NCP Noncompliance (with regulations) NTSB National Transportation Safety Board OBS Observation (comment) OpSpecs Operations Specifications OSHA Occupational Safety and Health Administration QA Quality Assurance QRC Quality-Related Concern (currently in compliance and conformance, but the problem indicates a weakness in the Quality System) RCA Root Cause Analysis SIE Supervisors of Internal Evaluation SRC Safety-Related Concern (currently in compliance and conformance, but the problem may have safety implications) W&B Weight and Balance B-20 ### Page 35 Advisory Circular Feedback Form If you find an error in this AC, have recommendations for improving it, or have suggestions for new items/subjects to be added, you may let us know by contacting the Air Transportation Division (AFS-200) at 9-AWA-AVS-AFS-200-Air-Transportation-Division@faa.gov or the Flight Standards Directives Management Officer at 9-AWA-AFS-140-Directives@faa.gov. Subject: AC 120-59B, Internal Evaluation Programs Date: _____________________ Please check all appropriate line items: An error (procedural or typographical) has been noted in paragraph ____________ on page _______. Recommend paragraph _____________ on page __________ be changed as follows: ______________________________________________________________________ ______________________________________________________________________ In a future change to this AC, please cover the following subject: (Briefly describe what you want added.) ______________________________________________________________________ ______________________________________________________________________ Other comments: ______________________________________________________________________ ______________________________________________________________________ I would like to discuss the above. Please contact me. Submitted by: Date: ______________________ --- OpenAirDocs.org is an independent research project of [Contrail Labs](https://contraillabs.com/). It is not affiliated with the FAA or the NTSB. Source documents are U.S. government works in the public domain.