Appendix 2
(8) Like a compliance audit, an internal evaluation will also verify that qualification records are maintained in accordance with 14 CFR § 145.161. However, because internal evaluation is a management audit, the first step becomes identifying how this external requirement is managed within the personnel qualification process. In other words, an evaluator should recognize that the recordkeeping system is a key characteristic of the personnel qualification process.
(9) The personnel qualification process should include a series of controls that monitor and verify the retention of personnel qualification records. The evaluator needs to verify that management has designed the personnel qualification process to include such controls. An audit of personnel records should not begin until the existence of these controls has been verified.
b. Identifying System Controls: Use of the Checklist.
(1) The second step of the internal evaluation process involves translating the understanding of key system characteristics into a method for identifying system controls, which are inherent management features that ensure that internal and external requirements are fulfilled.
(2) A key technique for the evaluator to accomplish this step is through the development of a checklist. On the simplest level, a checklist denotes points to be checked. More importantly, checklists help evaluators and auditors determine the correct order in which to proceed with an evaluation. A checklist supplements an evaluator’s memory and provides the basis for reconstructing an audit trail.
c. Criteria Checklist.
(1) One important form of checklist is the “criteria” checklist. A criteria checklist verifies compliance with a standard, regulation, or called-for procedure. In essence, a criteria checklist question is the transposition of a standard, regulation, or procedural requirement into a question.
(2) For example, 14 CFR § 145.211 requires repair stations to prepare and keep current a Quality Control Manual (QCM) containing inspection procedures. A criteria checklist question of such a requirement asks, “Does the certificate holder maintain a current Quality Control Manual?” Standards, requirements, etc., often have subsets of standards. A criteria checklist, therefore, needs to address individual subsets of a particular standard. Each manual required by 14 CFR § 145.211 includes a number of required subsets, such as explaining the internal inspection system, stating the preliminary inspection procedures, and describing the continuity of inspection responsibility. Such subsets must be determined and made part of the criteria checklist when assessing a standard.
d. Departmental Checklist.
(1) Another type of checklist, sometimes referred to as a “departmental” checklist, structures questions according to discrete departments and/or tasks. Such checklists define the organization, resources, equipment (if any), and/or tasks, and ask questions such as:
Appendix 2
• What is the personnel level required for this department or tasks?
• Are sufficient personnel available to perform the task or departmental function?
(2) For example, a “departmental” checklist for the evaluation of a repair station’s incoming materials inspection process will ask questions such as:
• Who receives incoming parts and materials?
• Is the received part invoice compared to the purchase order?
• What procedure ensures that high priority parts are handled quickly and carefully?
e. Differences Between the Criteria and Departmental Checklist.
(1) As demonstrated from the above discussion, a departmental checklist goes beyond verifying that an organization complies with required criteria. A departmental checklist identifies the existence of controls within a given area to guide the accomplishment of external requirements and to meet internal plans and objectives.
(2) For example, recordkeeping should be a key characteristic of the personnel qualification process. A criteria checklist would ask, “Are personnel qualification records kept current?” A departmental checklist, however, would break down this requirement into a series of questions designed to identify the existence of controls that will ensure that adequate recordkeeping methods are a part of the personnel qualification process. For instance, questions might include:
• Who is responsible for maintaining personnel qualification records?
• What procedure verifies that personnel qualifications records are maintained for each supervisor? Are these procedures written?
(3) Obtaining answers to these types of questions is a primary part of the internal evaluation process. They provide indications of how well the organization is managing compliance. The answers help the evaluator identify areas of potential weakness, and thus determine best how to allocate audit resources to test the process.
f. Evaluation of Controls—Do They Work? Only after identifying the existence of key controls should the evaluator actually inspect, observe, or sample the defined audit population. In this manner, the evaluator not only verifies current compliance, but also evaluates the effectiveness of identified controls to ensure continuous compliance.
(1) An essential technique for the evaluator to apply during this step is sampling. A sampling rate of 100 percent would, of course, contribute to a high degree of reliability and accuracy. Practical considerations such as time and economics, however, often obviate the desire to sample every item in a population. But faced with 1,000 records of an action, how many records should an auditor examine before he or she can be reasonably certain that a conclusion about the records is reliable? Tables to guide sample size selection are readily available.
Appendix 2
(2) Table 1, Table of Sample Sizes – Number of Inspections Recommended to Achieve a 95 Percent Confidence Level, is an example of those used in sampling. This table is part of the guidance FAA inspectors receive for conducting inspections.
TABLE 1. TABLE OF SAMPLE SIZES – NUMBER OF INSPECTIONS RECOMMENDED TO ACHIEVE A 95 PERCENT CONFIDENCE LEVEL Population of Homogeneous Group Recommended Number of Inspections Up to 100 50% (up to 50) 200 40% (80) 400 35% (140) 500 33% (165) 1,000 28% (280) 2,000 16% (322) 3,000 11% (330) 4,000 8.8% (352) 5,000 7.7% (355) 10,000 3.7% (370)
(3) Evaluators should recognize that sampling is only one tool used in an audit. The sampling table above is used as part of a random sampling process. Random sampling of an entire population may not be the most effective means to check for a parameter. Samples supplement, but do not replace, common sense, good judgment, and experience. When combined with the results of the checklist, deficiencies found in the sample can begin to be evaluated as either isolated occurrences or signs of control/management weaknesses.
g. Reporting Results. As with any audit, the final stage of an audit performed under the internal evaluation approach should be the concise reporting of results.
Appendix 3 APPENDIX 3. RELATED READING MATERIALS LOCATION
1. RELATED READING MATERIALS (current editions):
a. European Union Aviation Safety Agency (EASA) Acceptable Means of Compliance (AMC) and Guidance Material (GM) to Part-145, Maintenance Organisation Approvals, available at https://www.easa.europa.eu/en/document-library/acceptable-means-of-complianceand-guidance-material/group/part-145---maintenance-organisation-approvals.
b. International Organization for Standardization (ISO) 9000, Quality Management Systems—Fundamentals and Vocabulary, available at https://www.ansi.org/.
c. AC 145-9, Guide for Developing and Evaluating Repair Station and Quality Control Manuals, available at https://www.faa.gov/regulations_policies/advisory_circulars/.
OMB Control Number: 2120-0746 Expiration Date:12/31/2027 Advisory Circular Feedback Form Paperwork Reduction Act Burden Statement: A federal agency may not conduct or sponsor, and a person is not required to respond to, nor shall a person be subject to a penalty for failure to comply with a collection of information subject to the requirements of the Paperwork Reduction Act unless that collection of information displays a currently valid OMB Control Number. The OMB Control Number for this information collection is 2120-0746. Public reporting for this collection of information is estimated to be approximately 20 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering, and maintaining the data needed, completing, and reviewing the collection of information.
All responses to this collection of information are voluntary. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to:
Information Collection Clearance Officer, Federal Aviation Administration, 10101 Hillwood Parkway, Fort Worth, TX 76177-1524.
If you find an error in this Advisory Circular, have recommendations for improving it, or have suggestions for new items/subjects to be added, you may let us know by emailing this form to the Flight Standards Directives Management Officer at 9-AWA-AFB-120-Directives@faa.gov.
Subject:
AC 145-5 CHG 1, Repair Station Internal Evaluation Programs Date: _____________________ Please mark all appropriate line items:
☐ An error (procedural or typographical) has been noted in paragraph ____________ on page _______.
☐ Recommend paragraph _____________ on page __________ be changed as follows:
☐ In a future change to this AC, please cover the following subject:
(Briefly describe what you want added.)
☐ Other comments:
☐ I would like to discuss the above. Please contact me using the information below.
Submitted by: Date: ______________________ FAA 1320-73 (01/25) SUPERSEDES PREVIOUS EDITIONS