U.S. Department of Transportation Federal Aviation Administration Advisory Circular Subject: Repair Station Internal Evaluation Programs Date: 2/16/17 AC No: 145-5 Initiated by: AFS-300 Change: 1
1. PURPOSE. This advisory circular (AC) provides information and guidance material that may be used by repair station certificate holders operating under Title 14 of the Code of Federal Regulations (14 CFR) part 145 to design and implement an Internal Evaluation Program (IEP).
The procedures and practices outlined in this AC can be applied to all of a repair station’s operations.
2. PRINCIPAL CHANGES. This change adds information about FAA compliance inspection results to paragraph 3; replaces the discussion of the Joint Aviation Authorities (JAA) with information about the European Aviation Safety Agency (EASA) in paragraphs 3 and 6;
adds new information regarding FAA resources and assistance for certificate holders to paragraph 10; replaces the related reading materials in Appendix 3 with new online resources;
and updates terminology and references throughout the document.
PAGE CONTROL CHART Remove Pages Dated Insert Pages Dated Pages 1 thru 5 9/27/95 Pages 1 thru 5 2/16/17 Pages 7 thru 20 9/27/95 Pages 6 thru 18 2/16/17 Appendix 1, Page 2 9/27/95 Appendix 1, Page 2 2/16/17 Appendix 2, Pages 2 thru 5 9/27/95 Appendix 2, Pages 2 thru 5 (and 6) 2/16/17 Appendix 3, Pages 1 thru 3 9/27/95 Appendix 3, Page 1 (and 2) 2/16/17 ORIGINAL SIGNED by /s/ John Barbagallo Deputy Director, Flight Standards Service
U.S. Department of Transportation Federal Aviation Administration Advisory Circular Subject: Repair Station Internal Evaluation Programs Date: 9/27/95 AC No: 145-5 Initiated by: AFS-300 Change:
1. PURPOSE. This advisory circular (AC) provides information and guidance material that may be used by repair station certificate holders operating under Title 14 of the Code of Federal Regulations (14 CFR) part 145 to design and implement an Internal Evaluation Program (IEP).
The procedures and practices outlined in this AC can be applied to all of a repair station’s operations. There is no regulatory requirement for an IEP; however, the Federal Aviation Administration (FAA) encourages such a program to increase the awareness of management and all employees of their responsibility to promote continuous compliance with all regulatory requirements and good operating practices. Establishing the type of program described in this document is completely voluntary.
2. RELATED READING MATERIAL. A list of additional information that may be useful in understanding and implementing an IEP can be found in Appendix 3, Information to Locate Related Reading Materials.
3. BACKGROUND. Although safety in air transportation is a responsibility shared by certificate holders and the FAA, the ways in which this responsibility is discharged differ. While the FAA monitors certificate holders’ compliance through periodic inspections, the certificate holders clearly have a superior vantage point in monitoring their own performance and have greater resources to do so than the FAA. Although the FAA provides standards and guidance and promotes compliance through its surveillance, enforcement, and other programs, it is the certificate holders upon whom rests the primary and ultimate responsibility to verify that their operations are continuously in compliance with all regulatory requirements. Because certificate holders are in the best position to identify deficiencies and promptly correct them, it is to their advantage to have a procedure in place whereby they monitor compliance and continuously improve their compliance posture. Internal reviews will improve the certificate holder’s ability to identify and correct compliance problems in a more timely manner. There are many ways to quickly identify and correct problems. One of the methods that has evolved over the past few years, primarily used by the air carrier industry, is called internal evaluation.
a. Development of the IEP. The concept of an internal evaluation, as defined by the FAA, was developed in the late 1980s as part of several FAA initiatives to improve the partnership between the FAA and the air carrier industry.
(1) At that time, the FAA considered making an IEP a regulatory requirement.
This contemplated regulation would have instituted a process whereby an air carrier’s management would be more directly responsible for not only complying with 14 CFR, but also
Par 3 Page 2 having a system to continuously verify that operations were in compliance with 14 CFR, instead of relying on the FAA to do such verification through its surveillance. The FAA decided it was premature to develop a regulatory requirement. However, the FAA considered the concept to be sound and believed that voluntary IEPs should be strongly encouraged.
(2) Encouragement came in the form of meetings with industry representatives to explain the concept and discuss how the FAA could support such a voluntary program.
In February 1992, the FAA published the Air Carrier Internal Evaluation Model Program Guide to provide additional guidance to the air carrier industry and to FAA inspectors on the implementation of IEPs for different types of air carrier operations.
b. Related Programs. The IEP is closely related to two other programs of interest to many certificated repair stations.
(1) The first program is the FAA voluntary disclosure policy. Under this policy, implemented for repair stations on May 20, 1994, if a repair station finds a potential violation of 14 CFR, reports it to the FAA, and meets other specific conditions, the FAA will not pursue an enforcement action.
(a) An IEP is an effective method for discovering potential violations and developing corrective actions that are aimed at permanently eliminating the underlying cause of a problem.
Corrective actions of this type are one of the requirements of the voluntary disclosure program.
(b) Even though the internal evaluation process supports voluntary disclosures, it is not necessary to have an IEP to participate in the voluntary disclosure program, nor is it necessary to voluntarily disclose any findings from an internal evaluation. Although the two programs work well together, they can be mutually exclusive.
(2) The second related program is the quality system required by the European Aviation Safety Agency (EASA) for organizations holding EASA Part-145 approval.
c. FAA Compliance Inspections Results.
(1) The FAA field offices regularly conduct inspections of repair stations as a part of their surveillance work programs. Results of these inspections and routine surveillance indicate that there are recurring areas of noncompliance that would not have occurred as frequently, or at all, if IEPs had been implemented. These areas may include:
(a) Failure to maintain and follow a quality control (QC) system as required by part 145, § 145.211.
(b) Failure to maintain and use the equipment, materials, and data required by § 145.109.
(c) Inadequate or noncurrent personnel records required by § 145.161.
Par 3 Page 3
(d) Performing work for which the repair station is not authorized (§ 145.201).
(e) Failure to follow the requirements contained in § 145.215 when utilizing a capability list.
(2) While an IEP will not eliminate all instances of noncompliance with the regulations, it will provide the certificate holder with a better overview of the organization, systems, and procedures that may, in fact, reveal systemic weaknesses or a lack of sufficient management controls. These weaknesses represent the potential for noncompliance; consequently, identifying them in a systematic manner will often permit corrective action before a noncompliance situation occurs.
4. DEFINITIONS. The following definitions apply to the discussion in this AC and may not be the same as similar terms used in other documents or applications.
a. Audit. A methodical, planned examination of processes, records, and transactions to verify compliance with a specific audit criterion. Inspections are normally a part of an audit.
Results are primarily presented in terms of findings and concerns. Auditing measures against a defined standard, but does not analyze the standard.
b. Concern. A conclusion concerning a system or process that identifies a condition that may become a finding or a system weakness, which could be the underlying cause of a future noncompliance situation. A concern is not, at the time of the examination, an example of noncompliance with a standard. If so, it would be a finding.
c. Control. A key procedure, responsibility, or decisionmaking position within an organization or functional area. Effective controls reduce the probability of action contrary to established requirements and reduce the adverse consequences of such actions should they occur.
The controls used in all phases of the repair station’s operations should receive the majority of the attention from the internal evaluation.
d. Evaluation. An examination and evaluation of all aspects of a process, both positive and negative, that includes a review of standards, if appropriate. An evaluation is a process designed to take a larger view of an operation or organization than an audit. An audit is a subset of an evaluation. Results of evaluations are presented in terms of findings, concerns, observations, and recommendations.
e. Finding. A conclusion, supported by evidence, that there has been or is a process or product that is not in compliance with an established standard.
f. Inspection. A critical examination of an event or object for conformance with a standard.
Results of inspections are reported only in terms of findings.
g. Internal Evaluation. A comprehensive, continual monitoring process that is initiated and managed by the repair station, even though the personnel actually conducting various audits as part of the evaluation program may be internal or external to the repair station. The objective of this process is to promote attitudes and procedures that build quality into a product rather than depending on corrections of deficiencies to meet quality goals.
Par 4 Page 4
h. Observation. A noteworthy feature of a system or procedure. The feature noted is usually a positive or commendable aspect that should be brought to the attention of management to ensure that the feature is preserved and perhaps adopted in other places, if appropriate.
i. Recommendation. A specific suggested action that addresses a deficiency (either a finding or a concern). A recommendation from an internal evaluation team should be analyzed to determine if it should be adopted. Correction of deficiencies should be the responsibility of the affected organization.
j. Top Management. A certificate holder’s chief executive officer (CEO), chief operating officer (COO), president, or a person in an equivalent position who has the authority to resolve issues and take action and can be held accountable for quality issues. The FAA believes that top management should be well aware of the plans, results (findings, concerns, and observations), and followup actions undertaken in an IEP.
5. PROGRAM FEATURES. Internal evaluations should be part of an overall IEP that has written descriptions of the key elements of the program. The program should have a structured and planned series of evaluations that are designed to improve the quality of all steps and functions in the process that lead to a final repair station product.
a. Understanding the Basic Features. An IEP is:
(1) A Systems-Oriented Process. A continual process that incorporates the techniques of inspections, audits, and evaluations to assess the adequacy of managerial controls and processes in critical systems and to continuously improve those systems based upon the results of regular evaluations.
(2) More Than an Audit. A review that extends beyond regulatory compliance to determine the causes of deficiencies and detect needed enhancements to company operating practices before deficiencies occur. An IEP would normally verify compliance with company procedures, policies, and sound business practices, as well as legal requirements.
(3) A Corrective Process. A process that analyzes deficiencies, develops corrective action plans (CAP) to correct deficiencies, and performs followup evaluations to verify that those corrections have taken place and have been effective.
(4) A Program with High-Level Attention. An independent process that organizationally has straight-line reporting responsibility to top management.
(5) An Additional Quality Assurance (QA) Process. The IEP should not be misunderstood as a program that replaces existing regulatory requirements, such as the QC system described in § 145.211. It is comprehensive and includes identifying corrective actions, verifying that those actions have taken place, and ensuring that problems do not reoccur.
Further, one of the most critical aspects of an IEP is the regular day-to-day involvement of top management, which typically distinguishes it from the normal repair station inspection functions.
Par 5 Page 5
b. Broad Applicability. The internal evaluation should cover all systems, processes, and products that are basic parts of the repair station’s activities. There is no set list of items to be covered since each operation is unique, but a representative list of areas to evaluate would include:
(1) Facilities and equipment.
(2) Repair station authority and limitations versus actual practice, including controls over any deviation authority.
(3) Personnel qualifications, training, and staffing levels.
(4) Manuals and airworthiness data.
(5) Continuity of work and supervision during personnel changes.
(6) Supplier selection, approval, and surveillance.
(7) Parts and materials handling.
(8) Inspection and QC processes.
(9) Tool adequacy and calibration.
(10) Maintenance release process.
(11) Defect reporting.
(12) Records and recordkeeping procedures.
c. Potential Benefits. An effective IEP may bring benefits to repair stations that go beyond regulatory compliance. The discipline, structure, and added oversight afforded an organization through internal evaluation can provide better management information and controls, leading to increased efficiency.
6. IEP DESCRIPTION. The following are essential elements of an IEP. Each of these should be described in a program document.
a. Independent and Well-Defined Responsibility. As a part of identifying internal evaluation responsibility and independence, certificate holders should identify resources and personnel to conduct the IEP, and they should describe their organizational independence within the company in light of their internal evaluation functions. Individuals conducting internal evaluations should not be responsible for accomplishing or managing work in the areas being evaluated or the tasks being reviewed. This concept may have to be modified for some very small operations.
(1) For some certificate holders, the size of the operation may justify the costs associated with having full-time, dedicated resources, and personnel in a separate internal evaluation department. However, when full-time, dedicated resources and personnel are not practical,
Par 6 Page 6 procedures should be established so that any persons having direct responsibility for the areas to be evaluated are not involved in the selection of the evaluation team.
(2) At very small operations, a reasonable IEP might consist of developing checklists and a schedule (monthly, quarterly, semiannual, or annual) for accomplishing checklist items.
Even in such cases, the review should include a written statement acknowledging the completion of the checklist items and the signature of a top management official. Under these conditions, occasional independent oversight of checklist item development and accomplishment should be considered.
(3) Certificate holders that use outside resources in support of, or to accomplish, an IEP should show that use of those outside resources is coordinated through a chain of command that reflects independence and contact with top management.
(4) A certificate holder’s IEP document should identify the person and/or group within the organization who has the responsibility and authority to:
(a) Perform or cause to be performed evaluations, audits, and inspections as a part of an ongoing IEP.
(b) Identify and record any findings or concerns and the evidence necessary to substantiate those findings or concerns.
(c) Recommend or assist with the development of solutions to findings or concerns.
(d) Verify the implementation of solutions consistent with the action plan and validate that solutions work.
(e) Communicate and coordinate activities with FAA personnel on a regular basis.
(5) A senior level management representative should accept the responsibility to ensure that an IEP is properly established, implemented, and maintained. Further, this individual should have the authority to see that appropriate corrective actions are implemented in a timely manner.
This management position should be above the level that directly supervises work accomplishment or procedural development and should have direct contact with the CEO or equivalent. For smaller certificate holders, it is appropriate that this person be the CEO, president, or equivalent.
b. Top Management Involvement and Accountability. To be effective, an IEP must have the attention of top management regularly, not just when the program is initiated. Top management should review internal evaluation results to verify that satisfactory corrective actions have been implemented.
c. A Continual, Structured Process. In order to effectively anticipate potential problem areas and correct them before actual findings occur, an IEP should be a continual, ongoing function. An internal evaluation is intended to be more than spot-check inspections of
Par 6 Page 7 operating practices. Of and by themselves, spot-check inspections will do little more than identify symptoms of potential problems.
(1) A continual process is needed to verify whether findings are isolated instances or actual symptoms of policy, procedural, or managerial problems. A continual program should include scheduled evaluations, followup evaluations as necessary, and special evaluations when trends are identified.
(2) Having a well-structured program ensures that all areas of operations are covered at appropriate intervals. It also institutionalizes the process so that a change in personnel does not adversely affect the program.
d. A Plan for Scheduling Evaluations. It is essential for a repair station’s IEP to include a defined schedule of activities. This planned schedule will serve to verify that the IEP is comprehensive, well controlled, and timely. A published schedule also provides a vehicle for keeping management informed. A proper internal evaluation schedule should include a planned cycle for periodically reviewing areas specified to be covered by the certificate holder’s IEP.
However, the scheduling process should also be dynamic and allow for special evaluations when trends are identified. In addition, followup evaluations should be scheduled as necessary to verify that corrective action commitments were met and that they were effective in eliminating any reported findings or concerns. Planned, special, and followup evaluations, all of which comprise an effective internal evaluation schedule, are further described below.
(1) Planned Cycle. An effective program will establish a schedule of events that will be performed during a set calendar period under the IEP. All key areas will be reviewed at least once each year. It is helpful to divide the complete schedule into segments that are practical from the workload standpoint. However, it is important to schedule evaluations to allow enough flexibility for resources to be committed to special evaluations or followup evaluations, if needed.
(2) Special Evaluations. Conduct special evaluations based on concerns or priorities set by top management. The need for special evaluations can be driven by such factors as a review of industry trends, FAA concerns, or identified internal adverse trends.
(3) Followup Evaluations. Schedule and conduct followup evaluations to ensure that corrective action commitments were met, to verify that corrective actions (described in subparagraph 6e) were effective, and to review steps taken as a result of FAA surveillance findings.
e. CAPs. An IEP should include procedures that ensure that CAPs are developed in response to findings or concerns and for monitoring CAPs to verify their timely and effective implementation. Internal evaluation personnel should participate in the development of CAPs.
However, organizational responsibility and accountability for the development and implementation of CAPs should reside with the technical departments cited in the finding or concern.
Par 6 Page 8
(1) A proper CAP should include the following elements:
(a) A detailed description of the finding or concern and how it was discovered.
This should include discussion of the scope and extent of the problem so that candidate solutions can be analyzed properly.
(b) Analysis of evidence to determine the root cause(s) of the finding or concern.
(c) Identification of planned corrective steps to be taken in response to the finding or concern.
(d) An implementation schedule, including a timeframe for putting corrective steps in place.
(e) The individual responsible for implementing each of the corrective steps.
(2) The individuals responsible for managing an IEP should facilitate the corrective action process by:
(a) Ensuring that CAPs are developed in a timely manner.
(b) Verifying that CAPs include the elements outlined above.
(c) Monitoring implementation of corrective actions.
(d) Providing top management with an independent assessment of CAP development and implementation.
(e) Initiating scheduled and/or unannounced followup evaluations to ensure the effectiveness of corrective steps specified in CAPs.
f. Records. The results of an internal evaluation and the review of internal evaluation information by top management should be documented in reports and other appropriate records, consistent with the process of internal reporting at the repair station. The repair station should decide upon the frequency, format, and structure for informing top management of internal evaluation plans, results, and followup actions. It is recommended that the reporting structure also be documented by the repair station and become a part of its program plan.
(1) IEP files should include:
• Scheduled evaluation reports;
• Special evaluation reports, including the trends or other reasons for scheduling a special evaluation;
• CAPs; and
• Results of followup evaluations.
Par 6 Page 9
(2) Recognizing that much of the information contained in IEP records could be proprietary, a repair station should maintain and secure these records on their premises. If given to the FAA, proprietary information will be protected by the FAA in accordance with applicable laws and regulations.
(3) Sharing IEP information with the FAA at the repair station offices can enhance the working relationship between the FAA and the repair station. Information not required by regulation that remains on the repair station’s property would not normally be subject to public disclosure.
g. Specialized Training and Experience for Evaluators. The evaluators that are used by the repair station should have training in the specific procedures used by the repair station.
(1) Evaluators should be trained and experienced specifically with the evaluation of quality systems. General inspection experience only is usually insufficient.
(2) The internal evaluation concept emphasizes the analysis of procedures, systems, and management controls. It is not another layer of transaction examination and hardware inspections. Rather than just verifying the work of the inspectors, for example, it analyzes the inspection system. Consequently, it requires different skills.
7. AUDITING PROCEDURES—EXAMPLE. The basic tool for accomplishing an evaluation is the audit. An audit provides assurances that an organization’s methods conform to its requirements, and it assesses the effectiveness of management controls. Audits help answer the question, “Are the actual practices consistent with the defined standards?” The example in Figure 1, Internal Evaluation Report—Example, demonstrates internal evaluation procedures and the use of auditing concepts in the context of a typical repair station function. This example addresses the procedures and systems for ensuring that all parts used in modifications or repairs are approved for such use. Additional information on auditing techniques is provided in Appendix 2, Auditing Techniques. The example demonstrates the need for an evaluator to understand the complete system and the applicable requirements. It is important to remember that the use of the word “system” in this discussion refers to all elements that impact a function:
the relevant policies, procedures, management, operating personnel, training, paperwork, facilities, and equipment.
a. Background. For the purposes of this example, assume that a repair station has an IEP in place and there are trained evaluators available, either within the company or from outside sources. As part of an ongoing series of audits, a small team of evaluators has initiated a review of the systems that the repair station uses to control its parts supply and ensure that only correct (approved) parts are used in any of its work. The objective of this internal evaluation task is to verify that the repair station has the proper procedures, personnel are properly trained, and the management controls are functioning so that unapproved parts will not be used. It is not simply a spot inspection of parts and/or the related paperwork to see if any unapproved parts can be found.
Par 7 Page 10
b. Process. For most internal evaluation tasks, including this example, the basic steps are as follows.
(1) Step 1: Understanding the System. The processes that could affect parts usage in this repair station include the establishment of policy, the documentation of procedures, the ordering system, the receiving inspection process, parts storage and tracking, parts selection by the mechanic, and feedback on the performance of parts. Each of these processes should be reviewed by internal evaluators. To illustrate the concept, the parts ordering and receiving inspection processes will be discussed. The types of questions that should be asked to help understand this system are illustrated below. Note that this is not a comprehensive list, but only examples of the types of questions that should be on the checklist. For the most part, the questions are oriented toward systems, procedures, and responsibilities rather than toward the inspection of parts.
(a) What written policy exists? Who is responsible for developing that policy?
How is complete dissemination of the policy ensured?
(b) Where are the purchasing procedures found? Who is responsible for them and for seeing that they are followed? Who is responsible for parts selection when ordering?
Are automatic reorder quantities periodically audited?
(c) How does the repair station identify approved sources of parts? Is there some form of supplier surveillance? How are rotables tracked when they are at another repair station?
(d) What are the receiving inspection procedures with regard to parts? Who is responsible for these procedures? How are the procedures updated when necessary?
(e) How do receiving inspections personnel know about the policy and procedures?
If there is a change, what is in place to be sure the change is known by all inspectors?
(f) How are parts stored and traced in the inventory? If there is a shelf life, how is it tracked? What happens when a part in stock exceeds its shelf life?
(g) What is the system/procedure for a mechanic to draw a part from stock? Does this system protect against the selection of an incorrect or unapproved part? Does it work, based on personnel perceptions?
(2) Step 2: Identifying Controls. Once the evaluators have developed a good understanding of how the system operates, the next step is to identify the built-in controls that ensure the system functions in accordance with the certificate holder’s Repair Station Manual (RSM) and/or Quality Control Manual (QCM) and 14 CFR.
(a) Does a schedule exist to conduct periodic reviews of the repair station’s policy to ensure that it reflects the actual functions of the repair station? When an item is added or deleted from the repair station’s capabilities, what procedures are in place to ensure company documentation is updated?