Par 7 Page 10
b. Process. For most internal evaluation tasks, including this example, the basic steps are as follows.
(1) Step 1: Understanding the System. The processes that could affect parts usage in this repair station include the establishment of policy, the documentation of procedures, the ordering system, the receiving inspection process, parts storage and tracking, parts selection by the mechanic, and feedback on the performance of parts. Each of these processes should be reviewed by internal evaluators. To illustrate the concept, the parts ordering and receiving inspection processes will be discussed. The types of questions that should be asked to help understand this system are illustrated below. Note that this is not a comprehensive list, but only examples of the types of questions that should be on the checklist. For the most part, the questions are oriented toward systems, procedures, and responsibilities rather than toward the inspection of parts.
(a) What written policy exists? Who is responsible for developing that policy?
How is complete dissemination of the policy ensured?
(b) Where are the purchasing procedures found? Who is responsible for them and for seeing that they are followed? Who is responsible for parts selection when ordering?
Are automatic reorder quantities periodically audited?
(c) How does the repair station identify approved sources of parts? Is there some form of supplier surveillance? How are rotables tracked when they are at another repair station?
(d) What are the receiving inspection procedures with regard to parts? Who is responsible for these procedures? How are the procedures updated when necessary?
(e) How do receiving inspections personnel know about the policy and procedures?
If there is a change, what is in place to be sure the change is known by all inspectors?
(f) How are parts stored and traced in the inventory? If there is a shelf life, how is it tracked? What happens when a part in stock exceeds its shelf life?
(g) What is the system/procedure for a mechanic to draw a part from stock? Does this system protect against the selection of an incorrect or unapproved part? Does it work, based on personnel perceptions?
(2) Step 2: Identifying Controls. Once the evaluators have developed a good understanding of how the system operates, the next step is to identify the built-in controls that ensure the system functions in accordance with the certificate holder’s Repair Station Manual (RSM) and/or Quality Control Manual (QCM) and 14 CFR.
(a) Does a schedule exist to conduct periodic reviews of the repair station’s policy to ensure that it reflects the actual functions of the repair station? When an item is added or deleted from the repair station’s capabilities, what procedures are in place to ensure company documentation is updated?
Par 7 Page 11
(b) Do purchase orders above a certain dollar amount require a higher level of management’s approval?
(c) How do purchasers know whether a source of parts is approved? Does the repair station have an approved vendor list, and is it periodically updated? Who is responsible for the list?
(d) Do the shipping and receiving facilities provide segregated areas for incoming and outgoing serviceable and unserviceable parts?
(e) Do inspection personnel receive training on receiving inspection procedures?
Is the training documented? Do new inspection personnel receive indoctrination training before conducting receiving inspections?
(f) When inspection procedures are revised, do inspectors complete a record that indicates they have been made aware of the revision? Is this record reviewed to ensure all inspectors have received the appropriate information?
(g) Are periodic parts room inventories conducted to ensure that the inventory tracking system reflects the actual quantities on hand?
(h) Are shelf life sensitive items tracked separately in the computer or other tracking system? Do parts room personnel conduct periodic reviews of shelf life items and discard any expired items? Is it clear who is responsible? Is a report that documents the completion of the shelf life review provided to management? What happens if a shelf life item is found expired?
(i) Is access to the parts room restricted to authorized personnel or may anyone enter? How is access controlled? Who is responsible for ensuring that access is controlled?
(3) Step 3: Evaluation Controls—Do They Work? An evaluation of whether a repair station’s controls are properly functioning may require a review of records, inventory lists, parts documentation, warranty claim records, and an inspection of a sample of parts. Also, an important part of any verification process is the personnel interviews. If personnel do not know policies or feel that they are ignored, management controls are not functioning properly.
The following types of questions may be used to conduct this portion of the evaluation.
(a) Does a record exist that indicates that a periodic policy review has been conducted? Is it complete and up to date? Does a review of the repair station’s procedures reconcile against the current QCM?
(b) Do high-dollar purchase orders indicate that management approved the purchase with a signature, initials, etc.?
(c) Do records list how many parts have been returned to vendors because they do not pass the receiving inspection? Does the current approved vendor list include a revision date?
Par 7 Page 12
(d) Do training records and interviews indicate that initial training and receiving inspection training have been conducted for inspection personnel? Do records indicate that training was conducted within a reasonable time after each revision of the RSM and/or QCM?
(e) How often are differences discovered between the inventory system and the actual number of parts on hand? Also, how often are stocks depleted without making a new order?
How often must the repair station order items on an Aircraft on the Ground (AOG) status?
(f) When conducting the shelf life inventory, how often are expired items found?
Do inventory records indicate that these items have been removed from stock and properly disposed of? If items were found that were substantially beyond limits, is there any documentation of what action was taken?
(g) How many warranty claims has the repair station been required to adjust in the past 6 months? How many FAA enforcement actions or letters of investigation has the repair station experienced in the past 6 months?
(h) Select a sample of parts using statistical sampling procedures (see Appendix 1, Program Plan Sample Outline). Are the parts properly tagged or otherwise identified?
What paperwork is available to demonstrate that the part is approved?
(4) Step 4: Reporting the Results. Results of an audit generally do not cover positive items, nor do they recommend actions. Under an IEP, these, along with findings and concerns, are important elements of the report. Consequently, an evaluation report should include at least the following:
(a) Scope of the Evaluation. This should include the areas evaluated, personnel interviewed (in general terms to provide management an indication as to the scope and depth of the review without violating any confidentiality), records examined, sampling plans, etc.
(b) Results. Descriptions of each finding or observation presented in such a manner as to indicate the relative importance of each. This will allow responsible personnel to set priorities for developing responses.
(c) Recommended Corrective Actions. In some cases, there may be more than one alternative recommended; in others, there may not be sufficient information for the evaluators to develop a recommendation.
(d) Positive Results. Some might be shared between different units within the repair station.
(e) Internal Evaluation Report Form. An AC 145-5 evaluation report standard form should be established. A possible format is shown in Figure 1.
(5) Step 5: Developing CAPs. CAPs should be developed principally by the organization responsible for implementing the corrective action; however, if the evaluation team has properly conducted its evaluation, it will have a detailed understanding of the systems and procedures underlying the problems and should be able to assist with the analysis of alternatives.
Par 7 Page 13 The internal evaluation team should ensure that a CAP is developed in a timely manner and includes all the key elements, particularly when corrective action is to be implemented, and who is responsible for implementation.
(6) Step 6: Followup Evaluations. To be effective, the IEP should have followup evaluations any time a significant corrective action is planned. The purpose is twofold:
to confirm that the action has taken place as planned and to verify that the fix has been effective.
If a properly implemented corrective action does not work, new alternatives should be developed as soon as possible. Keeping management aware of the results of followup evaluations is an essential part of the program.
FIGURE 1. INTERNAL EVALUATION REPORT—EXAMPLE Date: September 30, 1994 Evaluation Area: Control of Parts.
Evaluation Objective: Evaluate the procedures and systems for ensuring that all parts used in modification or repairs are approved for such use.
Evaluator: J. Doe Method: Management controls identified per attached checklist; 10 percent sample of parts in the stock room and on the shop floor.
Findings/Concerns:
Findings: No significant findings. Isolated discrepancy resolved.
Concerns: The separation of serviceable and non-serviceable parts is not always clear. Although the tagging procedures were correct for the sample of parts reviewed, additional physical separation would increase the controls.
Proposed Actions: Reorganize the parts room to establish a separate area for the parts that are not serviceable. Consider separating those parts into those that are beyond economic repair and those awaiting repair or overhaul.
Followup Recommended: Y _X_ N ___, within 6 months after the evaluation.
Report Routing: Director Material, Chief Inspector, Internal Evaluation File.
Par 8 Page 14
8. PROGRAM PLAN. The FAA recommends documenting IEP procedures and responsibilities in a program plan. This paragraph provides suggestions for preparing and structuring a program plan.
a. Planning Process.
(1) Certificate holders should determine what level of formal program plan development is consistent with the size and complexity of their operation.
(2) A program plan should describe the duties, AC 145-5 responsibilities, procedures, and organization of a certificate holder’s IEP. Terms and elements defined in program plans should be consistent with those outlined in paragraph 4.
(3) Copies of the program plan should be distributed to appropriate company personnel so that they are aware of IEP procedures. In addition, revisions should be made as necessary to ensure that the program plan continues to reflect the certificate holder’s current internal evaluation procedures and organization.
(4) Even if a certificate holder elects to prepare a written program plan or revise an existing plan, the FAA will not approve or otherwise formally accept program plan contents.
Documenting the procedures and responsibilities associated with any program is considered good practice. For certificate holders who choose to prepare a program plan, the FAA will be available to provide assistance, if requested.
b. Plan Structure. A sample outline of a program plan, using the program elements discussed in this AC, is provided in Appendix 1. This outline should be viewed as a sample of items that warrant consideration when a certificate holder is designing an IEP. The number of items addressed and how they are documented will ultimately depend on the complexity of the operation.
9. ORGANIZATIONAL MODELS. Each repair station is unique with regard to size, ratings, facilities, personnel resources, and method of operation. Therefore, no single type of structure will be appropriate for all types of internal evaluation organizations. Three basic types that may be used are a dedicated internal evaluation department (see Figure 2, Internal Evaluation Program Model: Dedicated Internal Evaluation Department), a dedicated individual manager and temporary evaluators (see Figure 3, Internal Evaluation Program Model: Dedicated Individual Manager and Temporary Evaluators), and a combination of internal and external resources (see Figure 4, Internal Evaluation Program Model: Combination of Internal and External Resources). Other organizational structures may also be appropriate. The most critical elements are an adequate level of independence and a reporting process that ensures an accountable manager is aware of the evaluation results. It should be emphasized that any of those organizations must have the support of the highest levels of the company to ensure the effectiveness of the internal evaluation process.
a. Dedicated Internal Evaluation Department. This model presupposes a large organization with multiple departments and, perhaps, satellite locations. This department is responsible for performing periodic, comprehensive evaluations of the entire organization.
The department would report directly to the CEO or COO of the repair station as a staff function,
Par 9 Page 15 yet work closely with the operating departments to remain aware of current operating practices and to provide constant feedback to those departments.
FIGURE 2. INTERNAL EVALUATION PROGRAM MODEL:
DEDICATED INTERNAL EVALUATION DEPARTMENT
b. Dedicated Individual Manager and Temporary Evaluators. This organizational model has only one individual fully dedicated to the internal evaluation process. Employees from the various operating departments, usually one from each department, are used as temporary evaluators. This method conserves personnel resources, yet ensures a continuing IEP using qualified personnel on a part-time basis. The selected individuals report to the head of internal evaluation only for the time they participate in the internal evaluation process. This model is suited to the medium-sized company where a full complement of dedicated internal evaluation personnel may not be justified.
Par 9 Page 16 FIGURE 3. INTERNAL EVALUATION PROGRAM MODEL:
DEDICATED INDIVIDUAL MANAGER AND TEMPORARY EVALUATORS
c. Combination of Internal and External Resources. This model combines the use of internal and external resources to create a hybrid IEP. At least one company person is designated as the head of the internal evaluation function either on a full-time or part-time basis. This individual is supported by a combination of company, personnel, and external experts in the field to perform periodic comprehensive evaluations on the organization. The external experts may come from an industry association, a parent company, or a contracted organization. However, the primary internal resources remain responsible for the results of the evaluations, sharing of information with the FAA, incorporating needed changes to their ongoing duties, and the organization’s basic integrity.
Par 9 Page 17 FIGURE 4. INTERNAL EVALUATION PROGRAM MODEL: COMBINATION OF
INTERNAL AND EXTERNAL RESOURCES
10. FAA RESOURCES AND ASSISTANCE. Although the development and implementation of repair station IEPs is completely voluntary, the FAA can provide assistance in developing, implementing, and the continuing operation of the program.
a. Principal Inspector (PI)-Provided Resources for Certificate Holders. Certificate holders should contact their PI. The PI can provide information on available resources, including data from various FAA databases. Internal FAA and/or privacy policies may limit the availability of some resources, particularly those of an automated nature. However, certain generic data may be helpful in identifying trends and suggesting areas that may warrant additional focus.
b. FAA-Provided Resources for PIs. Coordination through the PI will ensure appropriate consideration of each request for information or assistance made by the certificate holder. The PI may also request assistance from regional or headquarters (HQ) offices, as required. Relevant training may be available through the FAA Academy (AMA), located at the Mike Monroney Aeronautical Center (MMAC) in Oklahoma City, OK.
11. CONCLUSION. Development of IEPs, as discussed in this AC, should help the company to ensure that company policies and procedures are responsive to growth and change and continually comply with appropriate safety requirements. Furthermore, the FAA strongly encourages certificate holders to make IEPs an integral part of their everyday management process and take full advantage of the voluntary disclosure policy. Aviation safety is best served by programs that allow certificate holders to identify and correct instances of noncompliance and invest more resources in efforts to preclude their recurrence, rather than pay civil penalties after noncompliance.
Par 12 Page 18
12. WHERE YOU CAN FIND THIS AC. You can find this AC on the FAA’s website at http://www.faa.gov/regulations_policies/advisory_circulars.
13. AC FEEDBACK FORM. For your convenience, the AC Feedback Form is the last page of this AC. Note any deficiencies found, clarifications needed, or suggested improvements regarding the contents of this AC on the Feedback Form.
Appendix 1 APPENDIX 1. PROGRAM PLAN SAMPLE OUTLINE The following is an outline of a typical Internal Evaluation Program (IEP) plan, with brief descriptions of each main subject area.
1. OBJECTIVE AND POLICY. The objective should be a statement that clearly defines the purpose and structure of the repair station’s IEP. Policy statements following the objective should indicate that the corporate view of the program is that it is independent, that it actively involves top management, and that it is an ongoing process designed to identify potential problem areas.
2. DEFINITION OF TERMS. Terms that will be used consistently in the IEP should be defined. For example, a repair station should have a procedure for categorizing results
(e.g., a finding or concern). If desired, there can also be a scheme for ordering findings so that management can quickly identify the ones that have the most serious consequences. These categories, as well as other terms applicable to the internal evaluation function, should be clearly defined and documented so that company personnel can understand and properly interpret them.
3. DUTIES AND RESPONSIBILITIES. The duties and responsibilities of internal evaluation personnel should be documented.
a. The repair station should specify which personnel are responsible for performing the following tasks:
(1) Supervising the internal evaluation function.
(2) Developing evaluation schedules.
(3) Performing evaluations, audits, and inspections as a part of internal evaluation.
(4) Identifying and recording any findings or concerns.
(5) Collecting the evidence necessary to substantiate findings or concerns.
(6) Initiating, recommending, or providing solutions to problems.
(7) Monitoring the development and implementation of corrective action plans (CAP).
(8) Maintaining and updating internal evaluation files.
(9) Verifying the implementation of solutions.
(10) Coordinating IEP activities with FAA personnel.
b. This section of the program plan should show that personnel responsible for the tasks listed above are independent of company procedural development and work accomplishment that would be subject to evaluations. The supervisor of the internal evaluation function should either be from top management or have straight-line reporting authority to top management.
Appendix 1
c. When full-time dedicated resources are not practical, developed procedures should show that persons having direct responsibility for the areas to be evaluated are not involved in the selection or supervision of the internal evaluation team, nor are they involved in any evaluation functions. In addition, it is highly desirable to have temporary evaluation personnel exempt from other duties and completely dedicated to the IEP.
4. ORGANIZATION CHART. An organization chart that clearly shows the position of the internal evaluation function in the repair station’s organization should be prepared. This position should reflect both the program’s independence within the corporate structure and straight-line reporting to top management.
5. REPORTING PROCEDURES. Reporting procedures should show that top management will be informed through straight-line reporting channels about the schedule, plans, results, and followup actions of the IEP. The procedures outlined in this section of the program plan should specify the frequency, format, and structure for reporting information to top management.
A procedure should also be developed that explains how the review by top management will be documented.
6. SPECIFIED AREAS COVERED. A certificate holder should specify the areas that are within the scope of review under the IEP. The FAA believes that the most effective IEP will encompass a complete review of all of the repair station’s technical operations. However, a repair station may elect to limit the evaluations to only a part of the technical operations, and it may elect to include nontechnical functions that are not subject to Title 14 of the Code of Federal Regulations (14 CFR) requirements. Also, the repair station may elect to phase in an IEP with limited coverage for an initial period and expanded coverage in later phases.
7. SCHEDULE PROCESS. The scheduling process should be comprised of the following elements:
a. Scheduled evaluations over a predetermined calendar period.
b. Special evaluations when trends are identified or priorities are set by top management.
c. Followup evaluations to verify the effectiveness of CAPs or in response to findings.
d. The program plan should include procedures for planning, developing, and coordinating the internal evaluation schedule. The responsibility for planning and developing schedule activities should also be defined.
8. RECORDS. The IEP should have a defined recordkeeping process. Procedures should specify how records are filed and maintained. Standard forms or formats for filing reports should also be specified. The following is a suggested list of records that should be maintained:
a. Scheduled evaluation reports.
b. Special evaluation reports.
c. Followup evaluation reports.
Appendix 1 Page 3 (and 4)
d. Responses to findings or concerns contained in reports.
e. CAPs submitted in response to findings.
9. TRAINING. If feasible, the repair station should specify that evaluators receive some type of training in quality auditing and evaluation principles, and systems analysis techniques. This training could be from any one or combination of the following:
a. In-house training course or on-the-job training (OJT).
b. College courses.
c. Home study course materials.
d. Industry seminars and workshops.