Appendix 2 APPENDIX 2. AUDITING TECHNIQUES
1. PURPOSE. This appendix reviews some of the standard auditing techniques that would apply to evaluations conducted under a repair station Internal Evaluation Program (IEP).
2. INTRODUCTION. Every organization establishes plans, objectives, and a method for translating objectives into actions. This is generally accomplished through the development of policies, systems, and procedures.
a. Policies are decisions; systems and procedures are the methods for carrying out decisions.
Hence, establishing and maintaining effective policies, systems, and procedures are ways of controlling the activities of an organization.
b. Whatever an organization’s functional activities, some method of review or appraisal is necessary to compare accomplishments with intended policies, systems, and procedures.
Examples of poor products may be the result of a breakdown in procedures, lack of supervision, or insufficient management standards.
3. AUDIT PROGRAM. A management audit program is defined by its areas of examination.
In any management audit, the critical areas for review are:
a. Organization. How the area under appraisal is organized and how the actual organization compares to the stated company organization.
b. Policies. What policies exist and how effective are those policies? Such a review should also include a determination of how policies support compliance with applicable laws and regulations.
c. Systems and Procedures. What systems and procedures exist and what possible defects or irregularities exist?
d. Controls. What system controls exist and are they adequate and effective?
4. SYSTEMATIC AUDIT PROCEDURES. A systematic audit method is based upon accepted auditing practices, so that those relying on the auditing program can be confident that results are accurate.
a. The main steps in auditing are the planning, implementation, and reporting phases. In the planning stage, the scope and objectives of an audit are determined by the needs of the client (in conjunction with auditors) along with the timing for the initiation and completion of the audit.
Auditors are selected based on individual qualifications and the scope of the task. Work assignments for the auditors are determined. Entities within an organization are notified of a pending audit. The audit team makes a preliminary assessment of the established system
(e.g., controls, procedures, systems, and documentary evidence) within the area to be audited.
Checklists for the audit area are developed and a schedule for conducting the audit is determined.
b. Evidence is gathered via review of documents, observations, records checks, and interviews with key individuals in the areas under review. Detailed checklists are used to ensure
Appendix 2 that all areas of interest are covered. Auditors are careful to keep a record of auditor activities, including the times and dates for completing an audit activity. In that way, audit results can generally be reproduced, and the status of work completed and still to be accomplished is constantly known.
c. The evidence developed during an audit may be portrayed as findings or concerns.
Upon completion of an audit, results are presented to relevant management before a written report is developed.
d. In the reporting phase of the audit, a written report on audit results is prepared, including information that can help the user of the report to take appropriate corrective actions. A written report generally will include a description of the purpose and scope of the audit, details of the audit activity, audit results, and supporting evidence.
5. APPLYING AUDITING TECHNIQUES TO INTERNAL EVALUATION. The following discussion provides an overview of the four fundamental steps suggested when conducting an audit under an IEP.
a. Identify Key System Characteristics: Understanding the Requirements. As a first step, an evaluator preparing an audit program needs to be cognizant of external requirements and other factors that may impact management’s decisions. Obvious examples of such external influences within the aviation environment include:
(1) Regulations.
(2) FAA guidance and policy.
(3) Equipment manufacturer guidance and specifications.
(4) Industry trends or incidents.
(5) An effective audit program includes provisions to verify that appropriate external influences are recognized in the management structure. An evaluator should use current FAA programs and materials (available through the local Flight Standards District Office (FSDO)) to ensure that the audit program addresses all appropriate external requirements and other factors.
(6) An internal evaluation should normally go beyond merely verifying compliance with the regulations and other applicable legal standards. A thorough program will be designed to determine and evaluate how an organization’s policies, practices, systems, and procedures account for and incorporate all external and internal requirements. Therefore, the evaluator must know how to translate pertinent requirements into key system characteristics.
(7) For example, 14 CFR part 145, § 145.161 states that a certificate holder must keep certain records pertaining to management, supervisory, and inspection personnel. A compliance audit would simply ensure that copies of personnel qualification records are maintained.
The results either confirm compliance or noncompliance at the point and time of the audit, but provide no assurance that compliance will exist each day unless, of course, such an audit is done every day.
Appendix 2
(8) Like a compliance audit, an internal evaluation will also verify that qualification records are maintained in accordance with § 145.161. However, because internal evaluation is a management audit, the first step becomes identifying how this external requirement is managed within the personnel qualification process. In other words, an evaluator should recognize that the recordkeeping system is a key characteristic of the personnel qualification process.
(9) The personnel qualification process should include a series of controls that monitor and verify the retention of personnel qualification records. The evaluator needs to verify that management has designed the personnel qualification process to include such controls. An audit of personnel records should not begin until the existence of these controls has been verified.
b. Identifying System Controls: Use of the Checklist.
(1) The second step of the internal evaluation process involves translating the understanding of key system characteristics into a method for identifying system controls, which are inherent management features that ensure that internal and external requirements are fulfilled.
(2) A key technique for the evaluator to accomplish this step is through the development of a checklist. On the simplest level, a checklist denotes points to be checked. More importantly, checklists help evaluators and auditors determine the correct order in which to proceed with an evaluation. A checklist supplements an evaluator’s memory and provides the basis for reconstructing an audit trail.
c. Criteria Checklist.
(1) One important form of checklist is the “criteria” checklist. A criteria checklist verifies compliance with a standard, regulation, or called-for procedure. In essence, a criteria checklist question is the transposition of a standard, regulation, or procedural requirement into a question.
(2) For example, § 145.211 requires repair stations to prepare and keep current a Quality Control Manual (QCM) containing inspection procedures. A criteria checklist question of such a requirement asks, “Does the certificate holder maintain a current Quality Control Manual?”
Standards, requirements, etc., often have subsets of standards. A criteria checklist, therefore, needs to address individual subsets of a particular standard. Each manual required by § 145.211 includes a number of required subsets, such as explaining the internal inspection system, stating the preliminary inspection procedures, and describing the continuity of inspection responsibility.
Such subsets must be determined and made part of the criteria checklist when assessing a standard.
d. Departmental Checklist.
(1) Another type of checklist, sometimes referred to as a “departmental” checklist, structures questions according to discrete departments and/or tasks. Such checklists define the organization, resources, equipment (if any), and/or tasks, and ask questions such as:
• What is the personnel level required for this department or tasks?
• Are sufficient personnel available to perform the task or departmental function?
Appendix 2
(2) For example, a “departmental” checklist for the evaluation of a repair station’s incoming materials inspection process will ask questions such as:
• Who receives incoming parts and materials?
• Is the received part invoice compared to the purchase order?
• What procedure ensures that high priority parts are handled quickly and carefully?
e. Differences Between the Criteria and Departmental Checklist.
(1) As demonstrated from the above discussion, a departmental checklist goes beyond verifying that an organization complies with required criteria. A departmental checklist identifies the existence of controls within a given area to guide the accomplishment of external requirements and to meet internal plans and objectives.
(2) For example, recordkeeping should be a key characteristic of the personnel qualification process. A criteria checklist would ask, “Are personnel qualification records kept current?” A departmental checklist, however, would break down this requirement into a series of questions designed to identify the existence of controls that will ensure that adequate recordkeeping methods are a part of the personnel qualification process. For instance, questions might include:
• Who is responsible for maintaining personnel qualification records?
• What procedure verifies that personnel qualifications records are maintained for each supervisor? Are these procedures written?
(3) Obtaining answers to these types of questions is a primary part of the internal evaluation process. They provide indications of how well the organization is managing compliance. The answers help the evaluator identify areas of potential weakness, and thus determine best how to allocate audit resources to test the process.
f. Evaluation of Controls—Do They Work? Only after identifying the existence of key controls should the evaluator actually inspect, observe, or sample the defined audit population.
In this manner, the evaluator not only verifies current compliance, but also evaluates the effectiveness of identified controls to ensure continuous compliance.
(1) An essential technique for the evaluator to apply during this step is sampling.
A sampling rate of 100 percent would, of course, contribute to a high degree of reliability and accuracy. Practical considerations such as time and economics, however, often obviate the desire to sample every item in a population. But faced with 1,000 records of an action, how many records should an auditor examine before he or she can be reasonably certain that a conclusion about the records is reliable? Tables to guide sample size selection are readily available.
(2) Table 1, Table of Sample Sizes – Number of Inspections Recommended to Achieve a 95 Percent Confidence Level, is an example of those used in sampling. This table is part of the guidance FAA inspectors receive for conducting inspections. This same table is found in FAA Order 8900.1, Volume 6, Chapter 2, Section 1, Safety Assurance System: General Policies and Procedures for Part 91K Surveillance.
Appendix 2 Page 5 (and 6) TABLE 1. TABLE OF SAMPLE SIZES – NUMBER OF INSPECTIONS RECOMMENDED TO ACHIEVE A 95 PERCENT CONFIDENCE LEVEL Population of Homogeneous Group Recommended Number of Inspections Up to 100 50% (up to 50) 200 40% (80) 400 35% (140) 500 33% (165) 1,000 28% (280) 2,000 16% (322) 3,000 11% (330) 4,000 8.8% (352) 5,000 7.7% (355) 10,000 3.7% (370)
(3) Evaluators should recognize that sampling is only one tool used in an audit.
The sampling table above is used as part of a random sampling process. Random sampling of an entire population may not be the most effective means to check for a parameter. Samples supplement, but do not replace, common sense, good judgment, and experience. When combined with the results of the checklist, deficiencies found in the sample can begin to be evaluated as either isolated occurrences or signs of control/management weaknesses.
g. Reporting Results. As with any audit, the final stage of an audit performed under the internal evaluation approach should be the concise reporting of results.
Appendix 3 Page 1 (and 2) APPENDIX 3. RELATED READING MATERIALS LOCATION
1. RELATED READING MATERIALS (current editions):
a. European Aviation Safety Agency (EASA) Acceptable Means of Compliance (AMC) and Guidance Material (GM) to Part-145, Maintenance organisation approvals, available at http://easa.europa.eu/document-library/acceptable-means-of-compliance-and-guidancematerials/group/part-145---maintenance-organisation-approvals.
b. International Organization for Standardization (ISO) 9000:2005, Quality management systems—Fundamentals and vocabulary, available at http://ansi.org/.
c. AC 145-9, Guide for Developing and Evaluating Repair Station and Quality Control Manuals, available at http://www.faa.gov/regulations_policies/advisory_circulars/.
Advisory Circular Feedback Form If you find an error in this AC, have recommendations for improving it, or have suggestions for new items/subjects to be added, you may let us know by contacting the Flight Standards Directives Management Officer at 9-AWA-AFS-140-Directives@faa.gov.
Subject: AC 145-5 CHG 1, Repair Station Internal Evaluation Programs Date: _____________________ Please check all appropriate line items:
☐ An error (procedural or typographical) has been noted in paragraph ____________ on page _______.
☐ Recommend paragraph _____________ on page __________ be changed as follows:
______________________________________________________________________ ______________________________________________________________________ ☐ In a future change to this AC, please cover the following subject:
(Briefly describe what you want added.)
______________________________________________________________________ ______________________________________________________________________ ☐ Other comments:
______________________________________________________________________ ______________________________________________________________________ ☐ I would like to discuss the above. Please contact me.
Submitted by: Date: ______________________