Par 7 Page 12
(d) Do training records and interviews indicate that initial training and receiving inspection training have been conducted for inspection personnel? Do records indicate that training was conducted within a reasonable time after each revision of the RSM and/or QCM?
(e) How often are differences discovered between the inventory system and the actual number of parts on hand? Also, how often are stocks depleted without making a new order? How often must the repair station order items on an Aircraft on the Ground (AOG) status?
(f) When conducting the shelf life inventory, how often are expired items found? Do inventory records indicate that these items have been removed from stock and properly disposed of? If items were found that were substantially beyond limits, is there any documentation of what action was taken?
(g) How many warranty claims has the repair station been required to adjust in the past 6 months? How many FAA enforcement actions or letters of investigation has the repair station experienced in the past 6 months?
(h) Select a sample of parts using statistical sampling procedures (see Appendix 1, Program Plan Sample Outline). Are the parts properly tagged or otherwise identified? What paperwork is available to demonstrate that the part is approved?
(4) Step 4: Reporting the Results. Results of an audit generally do not cover positive items, nor do they recommend actions. Under an IEP, these, along with findings and concerns, are important elements of the report. Consequently, an evaluation report should include at least the following:
(a) Scope of the Evaluation. This should include the areas evaluated, personnel interviewed (in general terms to provide management an indication as to the scope and depth of the review without violating any confidentiality), records examined, sampling plans, etc.
(b) Results. Descriptions of each finding or observation presented in such a manner as to indicate the relative importance of each. This will allow responsible personnel to set priorities for developing responses.
(c) Recommended Corrective Actions. In some cases, there may be more than one alternative recommended; in others, there may not be sufficient information for the evaluators to develop a recommendation.
(d) Positive Results. Some might be shared between different units within the repair station.
(e) Internal Evaluation Report Form. An AC 145-5 evaluation report standard form should be established. A possible format is shown in Figure 1.
(5) Step 5: Developing CAPs. CAPs should be developed principally by the organization responsible for implementing the corrective action; however, if the evaluation team has properly conducted its evaluation, it will have a detailed understanding of the systems and procedures underlying the problems and should be able to assist with the analysis of alternatives.
Par 7 Page 13 The internal evaluation team should ensure that a CAP is developed in a timely manner and includes all the key elements, particularly when corrective action is to be implemented, and who is responsible for implementation.
(6) Step 6: Followup Evaluations. To be effective, the IEP should have followup evaluations any time a significant corrective action is planned. The purpose is twofold: to confirm that the action has taken place as planned and to verify that the fix has been effective. If a properly implemented corrective action does not work, new alternatives should be developed as soon as possible. Keeping management aware of the results of followup evaluations is an essential part of the program.
FIGURE 1. INTERNAL EVALUATION REPORT—EXAMPLE Date: September 30, 1994 Evaluation Area: Control of Parts.
Evaluation Objective: Evaluate the procedures and systems for ensuring that all parts used in modification or repairs are approved for such use.
Evaluator: J. Doe Method: Management controls identified per attached checklist; 10 percent sample of parts in the stock room and on the shop floor.
Findings/Concerns:
Findings: No significant findings. Isolated discrepancy resolved.
Concerns: The separation of serviceable and non-serviceable parts is not always clear. Although the tagging procedures were correct for the sample of parts reviewed, additional physical separation would increase the controls.
Proposed Actions: Reorganize the parts room to establish a separate area for the parts that are not serviceable. Consider separating those parts into those that are beyond economic repair and those awaiting repair or overhaul.
Followup Recommended: Y X N ____, within 6 months after the evaluation.
Report Routing: Director Material, Chief Inspector, Internal Evaluation File.
Par 8 Page 14
8. PROGRAM PLAN. The FAA recommends documenting IEP procedures and responsibilities in a program plan. This paragraph provides suggestions for preparing and structuring a program plan.
a. Planning Process.
(1) Certificate holders should determine what level of formal program plan development is consistent with the size and complexity of their operation.
(2) A program plan should describe the duties, AC 145-5 responsibilities, procedures, and organization of a certificate holder’s IEP. Terms and elements defined in program plans should be consistent with those outlined in paragraph 4.
(3) Copies of the program plan should be distributed to appropriate company personnel so that they are aware of IEP procedures. In addition, revisions should be made as necessary to ensure that the program plan continues to reflect the certificate holder’s current internal evaluation procedures and organization.
(4) Even if a certificate holder elects to prepare a written program plan or revise an existing plan, the FAA will not approve or otherwise formally accept program plan contents.
Documenting the procedures and responsibilities associated with any program is considered good practice. For certificate holders who choose to prepare a program plan, the FAA will be available to provide assistance, if requested.
b. Plan Structure. A sample outline of a program plan, using the program elements discussed in this AC, is provided in Appendix 1. This outline should be viewed as a sample of items that warrant consideration when a certificate holder is designing an IEP. The number of items addressed and how they are documented will ultimately depend on the complexity of the operation.
9. ORGANIZATIONAL MODELS. Each repair station is unique with regard to size, ratings, facilities, personnel resources, and method of operation. Therefore, no single type of structure will be appropriate for all types of internal evaluation organizations. Three basic types that may be used are a dedicated internal evaluation department (see Figure 2, Internal Evaluation Program Model: Dedicated Internal Evaluation Department), a dedicated individual manager and temporary evaluators (see Figure 3, Internal Evaluation Program Model: Dedicated Individual Manager and Temporary Evaluators), and a combination of internal and external resources (see Figure 4, Internal Evaluation Program Model: Combination of Internal and External Resources). Other organizational structures may also be appropriate. The most critical elements are an adequate level of independence and a reporting process that ensures an accountable manager is aware of the evaluation results. The internal evaluation process is generally more effective when these organizations have visible support from the highest levels of the company.
a. Dedicated Internal Evaluation Department. This model presupposes a large organization with multiple departments and, perhaps, satellite locations. This department is responsible for performing periodic, comprehensive evaluations of the entire organization. The department would report directly to the CEO or COO of the repair station as a staff function, yet
Par 9 Page 15 work closely with the operating departments to remain aware of current operating practices and to provide constant feedback to those departments.
FIGURE 2. INTERNAL EVALUATION PROGRAM MODEL:
DEDICATED INTERNAL EVALUATION DEPARTMENT
b. D edicated Individual Manager and Temporary Evaluators. This organizational model has only one individual fully dedicated to the internal evaluation process. Employees from the various operating departments, usually one from each department, are used as temporary evaluators. This method conserves personnel resources, yet ensures a continuing IEP using qualified personnel on a part-time basis. The selected individuals report to the head of internal evaluation only for the time they participate in the internal evaluation process. This model is suited to the medium-sized company where a full complement of dedicated internal evaluation personnel may not be justified.
Par 9 Page 16 FIGURE 3. INTERNAL EVALUATION PROGRAM MODEL:
DEDICATED INDIVIDUAL MANAGER AND TEMPORARY EVALUATORS
c. C ombination of Internal and External Resources. This model combines the use of internal and external resources to create a hybrid IEP. At least one company person is designated as the head of the internal evaluation function either on a full-time or part-time basis. This individual is supported by a combination of company, personnel, and external experts in the field to perform periodic comprehensive evaluations on the organization. The external experts may come from an industry association, a parent company, or a contracted organization. However, the primary internal resources remain responsible for the results of the evaluations, sharing of information with the FAA, incorporating needed changes to their ongoing duties, and the organization’s basic integrity.
Par 9 Page 17 FIGURE 4. INTERNAL EVALUATION PROGRAM MODEL: COMBINATION OF
INTERNAL AND EXTERNAL RESOURCES
10.
FAA RESOURCES AND ASSISTANCE. Although the development and implementation of repair station IEPs is completely voluntary, the FAA can provide assistance in developing, implementing, and the continuing operation of the program.
a. Principal Inspector (PI)-Provided Resources for Certificate Holders. Certificate holders should contact their PI. The PI can provide information on available resources, including data from various FAA databases. Internal FAA and/or privacy policies may limit the availability of some resources, particularly those of an automated nature. However, certain generic data may be helpful in identifying trends and suggesting areas that may warrant additional focus.
b. FAA-Provided Resources for PIs. Coordination through the PI will ensure appropriate consideration of each request for information or assistance made by the certificate holder. The PI may also request assistance from the responsible Flight Standards office or Office of Safety Standards (OSS), as required. Relevant training may be available through the FAA Academy (AMA), located at the Mike Monroney Aeronautical Center (MMAC) in Oklahoma City, OK.
11. CONCLUSION. Development of IEPs, as discussed in this AC, should help the company to ensure that company policies and procedures are responsive to growth and change and continually comply with appropriate safety requirements. Furthermore, the FAA strongly encourages certificate holders to make IEPs an integral part of their everyday management process and take full advantage of the voluntary disclosure policy. Aviation safety is best served by programs that allow certificate holders to identify and correct instances of noncompliance and invest more resources in efforts to preclude their recurrence, rather than pay civil penalties after noncompliance.
Par 12 Page 18
12. WHERE YOU CAN FIND THIS AC. You can find this AC on the FAA’s website at https://www.faa.gov/regulations_policies/advisory_circulars.
13. AC FEEDBACK FORM. For your convenience, the AC Feedback Form is the last page of this AC. Note any deficiencies found, clarifications needed, or suggested improvements regarding the contents of this AC on the Feedback Form.
Appendix 1 APPENDIX 1. PROGRAM PLAN SAMPLE OUTLINE The following is an outline of a typical Internal Evaluation Program (IEP) plan, with brief descriptions of each main subject area.
1. OBJECTIVE AND POLICY. The objective should be a statement that clearly defines the purpose and structure of the repair station’s IEP. Policy statements following the objective should indicate that the corporate view of the program is that it is independent, that it actively involves top management, and that it is an ongoing process designed to identify potential problem areas.
2. DEFINITION OF TERMS. Terms that will be used consistently in the IEP should be defined. For example, a repair station should have a procedure for categorizing results
(e.g., a finding or concern). If desired, there can also be a scheme for ordering findings so that management can quickly identify the ones that have the most serious consequences. These categories, as well as other terms applicable to the internal evaluation function, should be clearly defined and documented so that company personnel can understand and properly interpret them.
3. DUTIES AND RESPONSIBILITIES. The duties and responsibilities of internal evaluation personnel should be documented.
a. The repair station should specify which personnel are responsible for performing the following tasks:
(1) Supervising the internal evaluation function.
(2) Developing evaluation schedules.
(3) Performing evaluations, audits, and inspections as a part of internal evaluation.
(4) Identifying and recording any findings or concerns.
(5) Collecting the evidence necessary to substantiate findings or concerns.
(6) Initiating, recommending, or providing solutions to problems.
(7) Monitoring the development and implementation of corrective action plans (CAP).
(8) Maintaining and updating internal evaluation files.
(9) Verifying the implementation of solutions.
(10) Coordinating IEP activities with FAA personnel.
b. This section of the program plan should show that personnel responsible for the tasks listed above are independent of company procedural development and work accomplishment that would be subject to evaluations. The supervisor of the internal evaluation function should either be from top management or have straight-line reporting authority to top management.
Appendix 1
c. When full-time dedicated resources are not practical, developed procedures should show that persons having direct responsibility for the areas to be evaluated are not involved in the selection or supervision of the internal evaluation team, nor are they involved in any evaluation functions. In addition, it is highly desirable to have temporary evaluation personnel exempt from other duties and completely dedicated to the IEP.
4. ORGANIZATION CHART. An organization chart that clearly shows the position of the internal evaluation function in the repair station’s organization should be prepared. This position should reflect both the program’s independence within the corporate structure and straight-line reporting to top management.
5. REPORTING PROCEDURES. Reporting procedures should show that top management will be informed through straight-line reporting channels about the schedule, plans, results, and followup actions of the IEP. The procedures outlined in this section of the program plan should specify the frequency, format, and structure for reporting information to top management. A procedure should also be developed that explains how the review by top management will be documented.
6. SPECIFIED AREAS COVERED. A certificate holder should specify the areas that are within the scope of review under the IEP. The FAA believes that the most effective IEP will encompass a complete review of all of the repair station’s technical operations. However, a repair station may elect to limit the evaluations to only a part of the technical operations, and it may elect to include nontechnical functions that are not subject to Title 14 of the Code of Federal Regulations (14 CFR) requirements. Also, the repair station may elect to phase in an IEP with limited coverage for an initial period and expanded coverage in later phases.
7. SCHEDULE PROCESS. The scheduling process should be comprised of the following elements:
a. Scheduled evaluations over a predetermined calendar period.
b. Special evaluations when trends are identified or priorities are set by top management.
c. Followup evaluations to verify the effectiveness of CAPs or in response to findings.
d. The program plan should include procedures for planning, developing, and coordinating the internal evaluation schedule. The responsibility for planning and developing schedule activities should also be defined.
8. RECORDS. The IEP should have a defined recordkeeping process. Procedures should specify how records are filed and maintained. Standard forms or formats for filing reports should also be specified. The following is a suggested list of records that should be maintained:
a. Scheduled evaluation reports.
b. Special evaluation reports.
c. Followup evaluation reports.
Appendix 1
d. Responses to findings or concerns contained in reports.
e. CAPs submitted in response to findings.
9. TRAINING. If feasible, the repair station should specify that evaluators receive some type of training in quality auditing and evaluation principles, and systems analysis techniques. This training could be from any one or combination of the following:
a. In-house training course or on-the-job training (OJT).
b. College courses.
c. Home study course materials.
d. Industry seminars and workshops.
Appendix 2 APPENDIX 2. AUDITING TECHNIQUES
1. PURPOSE. This appendix reviews some of the standard auditing techniques that would apply to evaluations conducted under a repair station Internal Evaluation Program (IEP).
2. INTRODUCTION. Every organization establishes plans, objectives, and a method for translating objectives into actions. This is generally accomplished through the development of policies, systems, and procedures.
a. Policies are decisions; systems and procedures are the methods for carrying out decisions.
Hence, establishing and maintaining effective policies, systems, and procedures are ways of controlling the activities of an organization.
b. Whatever an organization’s functional activities, some method of review or appraisal is necessary to compare accomplishments with intended policies, systems, and procedures.
Examples of poor products may be the result of a breakdown in procedures, lack of supervision, or insufficient management standards.
3. AUDIT PROGRAM. A management audit program is defined by its areas of examination.
In any management audit, the critical areas for review are:
a. Organization. How the area under appraisal is organized and how the actual organization compares to the stated company organization.
b. Policies. What policies exist and how effective are those policies? Such a review should also include a determination of how policies support compliance with applicable laws and regulations.
c. Systems and Procedures. What systems and procedures exist and what possible defects or irregularities exist?
d. Controls. What system controls exist and are they adequate and effective?
4. SYSTEMATIC AUDIT PROCEDURES. A systematic audit method is based upon accepted auditing practices, so that those relying on the auditing program can be confident that results are accurate.
a. The main steps in auditing are the planning, implementation, and reporting phases. In the planning stage, the scope and objectives of an audit are determined by the needs of the client (in conjunction with auditors) along with the timing for the initiation and completion of the audit.
Auditors are selected based on individual qualifications and the scope of the task. Work assignments for the auditors are determined. Entities within an organization are notified of a pending audit. The audit team makes a preliminary assessment of the established system
(e.g., controls, procedures, systems, and documentary evidence) within the area to be audited.
Checklists for the audit area are developed and a schedule for conducting the audit is determined.
b. Evidence is gathered via review of documents, observations, records checks, and interviews with key individuals in the areas under review. Detailed checklists are used to ensure
Appendix 2 that all areas of interest are covered. Auditors are careful to keep a record of auditor activities, including the times and dates for completing an audit activity. In that way, audit results can generally be reproduced, and the status of work completed and still to be accomplished is constantly known.
c. The evidence developed during an audit may be portrayed as findings or concerns. Upon completion of an audit, results are presented to relevant management before a written report is developed.
d. In the reporting phase of the audit, a written report on audit results is prepared, including information that can help the user of the report to take appropriate corrective actions. A written report generally will include a description of the purpose and scope of the audit, details of the audit activity, audit results, and supporting evidence.
5. APPLYING AUDITING TECHNIQUES TO INTERNAL EVALUATION. The following discussion provides an overview of the four fundamental steps suggested when conducting an audit under an IEP.
a. Identify Key System Characteristics: Understanding the Requirements. As a first step, an evaluator preparing an audit program needs to be cognizant of external requirements and other factors that may impact management’s decisions. Obvious examples of such external influences within the aviation environment include:
(1) Regulations.
(2) FAA guidance and policy.
(3) Equipment manufacturer guidance and specifications.
(4) Industry trends or incidents.
(5) An effective audit program includes provisions to verify that appropriate external influences are recognized in the management structure. An evaluator should use current FAA programs and materials (available through the responsible Flight Standards office to ensure that the audit program addresses all appropriate external requirements and other factors.
(6) An internal evaluation generally will go beyond merely verifying compliance with the regulations and other applicable legal standards. A thorough program will be designed to determine and evaluate how an organization’s policies, practices, systems, and procedures account for and incorporate all external and internal requirements. Accordingly, it is helpful for the evaluator to be able to translate pertinent requirements into key system characteristics.
(7) For example, 14 CFR § 145.161 states that a certificate holder must keep certain records pertaining to management, supervisory, and inspection personnel. A compliance audit would simply ensure that copies of personnel qualification records are maintained. The results either confirm compliance or noncompliance at the point and time of the audit, but provide no assurance that compliance will exist each day unless, of course, such an audit is done every day.