Appendix 2
(8) Like a compliance audit, an internal evaluation will also verify that qualification records are maintained in accordance with § 145.161. However, because internal evaluation is a management audit, the first step becomes identifying how this external requirement is managed within the personnel qualification process. In other words, an evaluator should recognize that the recordkeeping system is a key characteristic of the personnel qualification process.
(9) The personnel qualification process should include a series of controls that monitor and verify the retention of personnel qualification records. The evaluator needs to verify that management has designed the personnel qualification process to include such controls. An audit of personnel records should not begin until the existence of these controls has been verified.
b. Identifying System Controls: Use of the Checklist.
(1) The second step of the internal evaluation process involves translating the understanding of key system characteristics into a method for identifying system controls, which are inherent management features that ensure that internal and external requirements are fulfilled.
(2) A key technique for the evaluator to accomplish this step is through the development of a checklist. On the simplest level, a checklist denotes points to be checked. More importantly, checklists help evaluators and auditors determine the correct order in which to proceed with an evaluation. A checklist supplements an evaluator’s memory and provides the basis for reconstructing an audit trail.
c. Criteria Checklist.
(1) One important form of checklist is the “criteria” checklist. A criteria checklist verifies compliance with a standard, regulation, or called-for procedure. In essence, a criteria checklist question is the transposition of a standard, regulation, or procedural requirement into a question.
(2) For example, § 145.211 requires repair stations to prepare and keep current a Quality Control Manual (QCM) containing inspection procedures. A criteria checklist question of such a requirement asks, “Does the certificate holder maintain a current Quality Control Manual?”
Standards, requirements, etc., often have subsets of standards. A criteria checklist, therefore, needs to address individual subsets of a particular standard. Each manual required by § 145.211 includes a number of required subsets, such as explaining the internal inspection system, stating the preliminary inspection procedures, and describing the continuity of inspection responsibility.
Such subsets must be determined and made part of the criteria checklist when assessing a standard.
d. Departmental Checklist.
(1) Another type of checklist, sometimes referred to as a “departmental” checklist, structures questions according to discrete departments and/or tasks. Such checklists define the organization, resources, equipment (if any), and/or tasks, and ask questions such as:
• What is the personnel level required for this department or tasks?
• Are sufficient personnel available to perform the task or departmental function?
Appendix 2
(2) For example, a “departmental” checklist for the evaluation of a repair station’s incoming materials inspection process will ask questions such as:
• Who receives incoming parts and materials?
• Is the received part invoice compared to the purchase order?
• What procedure ensures that high priority parts are handled quickly and carefully?
e. Differences Between the Criteria and Departmental Checklist.
(1) As demonstrated from the above discussion, a departmental checklist goes beyond verifying that an organization complies with required criteria. A departmental checklist identifies the existence of controls within a given area to guide the accomplishment of external requirements and to meet internal plans and objectives.
(2) For example, recordkeeping should be a key characteristic of the personnel qualification process. A criteria checklist would ask, “Are personnel qualification records kept current?” A departmental checklist, however, would break down this requirement into a series of questions designed to identify the existence of controls that will ensure that adequate recordkeeping methods are a part of the personnel qualification process. For instance, questions might include:
• Who is responsible for maintaining personnel qualification records?
• What procedure verifies that personnel qualifications records are maintained for each supervisor? Are these procedures written?
(3) Obtaining answers to these types of questions is a primary part of the internal evaluation process. They provide indications of how well the organization is managing compliance. The answers help the evaluator identify areas of potential weakness, and thus determine best how to allocate audit resources to test the process.
f. Evaluation of Controls—Do They Work? Only after identifying the existence of key controls should the evaluator actually inspect, observe, or sample the defined audit population.
In this manner, the evaluator not only verifies current compliance, but also evaluates the effectiveness of identified controls to ensure continuous compliance.
(1) An essential technique for the evaluator to apply during this step is sampling.
A sampling rate of 100 percent would, of course, contribute to a high degree of reliability and accuracy. Practical considerations such as time and economics, however, often obviate the desire to sample every item in a population. But faced with 1,000 records of an action, how many records should an auditor examine before he or she can be reasonably certain that a conclusion about the records is reliable? Tables to guide sample size selection are readily available.
(2) Table 1, Table of Sample Sizes – Number of Inspections Recommended to Achieve a 95 Percent Confidence Level, is an example of those used in sampling. This table is part of the guidance FAA inspectors receive for conducting inspections. This same table is found in FAA Order 8900.1, Volume 6, Chapter 2, Section 1, Safety Assurance System: General Policies and Procedures for Part 91K Surveillance.
Appendix 2 TABLE 1. TABLE OF SAMPLE SIZES – NUMBER OF INSPECTIONS RECOMMENDED TO ACHIEVE A 95 PERCENT CONFIDENCE LEVEL Population of Homogeneous Group Recommended Number of Inspections Up to 100 50% (up to 50) 200 40% (80) 400 35% (140) 500 33% (165) 1,000 28% (280) 2,000 16% (322) 3,000 11% (330) 4,000 8.8% (352) 5,000 7.7% (355) 10,000 3.7% (370)
(3) Evaluators should recognize that sampling is only one tool used in an audit.
The sampling table above is used as part of a random sampling process. Random sampling of an entire population may not be the most effective means to check for a parameter. Samples supplement, but do not replace, common sense, good judgment, and experience. When combined with the results of the checklist, deficiencies found in the sample can begin to be evaluated as either isolated occurrences or signs of control/management weaknesses.
g. Reporting Results. As with any audit, the final stage of an audit performed under the internal evaluation approach should be the concise reporting of results.
Appendix 3 APPENDIX 3. RELATED READING MATERIALS LOCATION
1. RELATED READING MATERIALS (current editions):
a. European Union Aviation Safety Agency (EASA) Acceptable Means of Compliance (AMC) and Guidance Material (GM) to Part-145, Maintenance organisation approvals, available at https://easa.europa.eu/document-library/acceptable-means-of-compliance-and-guidancematerials/group/part-145---maintenance-organisation-approvals.
b. International Organization for Standardization (ISO) 9000:2005, Quality management systems—Fundamentals and vocabulary, available at https://ansi.org/.
c. AC 145-9, Guide for Developing and Evaluating Repair Station and Quality Control Manuals, available at https://www.faa.gov/regulations_policies/advisory_circulars/.
Advisory Circular Feedback Form If you find an error in this AC, have recommendations for improving it, or have suggestions for new items/subjects to be added, you may let us know by contacting the Flight Standards Directives Management Officer at 9-AWA-AFB-120-Directives@faa.gov.
Subject: AC 145-5 CHG 1, Repair Station Internal Evaluation Programs Date: _____________________ Please check all appropriate line items:
An e rro r (pro cedural or typographical) has been noted in paragraph ____________ on page _______.
Recommend paragraph _____________ on page __________ be changed as follows:
______________________________________________________________________ ______________________________________________________________________ In a future change to this AC, please cover the following subject:
(Briefly describe what you want added.)
______________________________________________________________________ ______________________________________________________________________ Other comments:
______________________________________________________________________ ______________________________________________________________________ I would like to discuss the above. Please contact me.
Submitted by: Date: ______________________