3-47
3.5.4.3.1 Evaluations performed at planned intervals help the aviation organization’s management determine if its safety management methods and practices are meeting the safety objectives and expectations documented in the safety policy. Evaluation planning should consider the safety criticality of the processes that are being evaluated and the results from previous evaluations.
The scope, content, and frequency of evaluations should be based on the organization’s need for information to assess the health of operational processes and the SMS. Aviation organizations also need to define criteria for selecting evaluators.
3.5.4.3.2 The results of evaluations can be recorded in a paper or electronic medium, such as in a common logbook-style binder, an electronic file folder, or a secure email account.
3.5.4.4 Implementation Strategies. The following examples demonstrate ways aviation organizations might choose to implement this requirement based on where the aviation organization may fall on the spectrum of size or complexity. The following are examples only and are not the only ways the requirements could be met.
3.5.4.4.1 In a small organization, the evaluation process could be carried out periodically by the accountable executive/owner. In larger organizations, evaluations could be performed by a key management person or designated employees as a collateral duty. The process could be accomplished by the Director of Safety (DOS) or the safety department on a monthly, quarterly, or other periodic basis, as determined by the information needs of the accountable executive or other senior management decisionmakers.
3.5.4.4.2 Aviation organizations on the higher end of the spectrum of size or complexity may accomplish evaluations by utilizing a safety department or an Internal Evaluation Program (IEP) office on a quarterly, annual, or other periodic basis, as determined by the information needs of the accountable executive or other senior management decisionmakers. Many part 121 and some part 135 organizations have IEPs, and their outputs can be integrated into the SMS. Analysis of evaluations is typically performed by a safety department. The resulting data would be acted on by the appropriate operational department with the safety department managing the data and assisting the responsible process owners in resolving identified issues. Most large organizations have standing management committees that consider results of evaluations and any corrective action needed.
3.5.4.5 Considerations for Small and Single-Individual Organizations.
A single-individual organization should review their records on a defined interval to look for trends in operational performance that could identify ineffective processes and procedures.
3-48
3.5.5 Investigation of Incidents and Accidents.
3.5.5.1 References. Section 5.71(a)(5).
3.5.5.2 Part 5 Requirement.
§ 5.71(a)(5), Safety performance monitoring and measurement.
(a) Any person required to have an SMS under this part must develop and maintain processes and systems to acquire data with respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following:
(5) Investigations of incidents and accidents.
3.5.5.3 Discussion. Investigations should be treated as an opportunity for organizational learning to prevent a repeat of errors and/or change organizational processes so that mistakes do not recur.
3.5.5.3.1 Aviation organizations need to define the types of incidents and accidents that require investigations. Part 5 only requires incidents and accidents that directly affect the aviation product or service be investigated.
3.5.5.3.2 Processes used will vary based on the size and complexity of the organization and aviation service the organization provides or supports. Investigations should focus on what went wrong rather than who caused the error and emphasize improvement of safety performance. To the extent permitted by law, the organization should include data, if available, from outside sources, such as the FAA or National Transportation Safety Board (NTSB) investigations.
3.5.5.3.3 Implementation Strategies. The following examples demonstrate ways aviation organizations might choose to implement this requirement based on where the aviation organization may fall on the spectrum of size or complexity. The following are examples only and are not the only ways the requirements could be met.
3.5.5.3.4 In smaller or less complex aviation organizations, investigations can be conducted by the accountable executive or assigned employees. Investigations can be conducted by a safety department with additional assigned line personnel providing technical expertise. Aviation organizations on the higher end of the spectrum of size or complexity may have safety teams with specialized disciplines conduct the investigations. The results of investigations can be recorded in paper or electronic medium (e.g., in a common logbook-style binder, an electronic file folder, or other electronic system, such as WBAT or another suitable system).
3.5.5.4 Considerations for Small and Single-Individual Organizations.
Investigation is a necessary activity within any SMS, regardless of the
3-49 organization’s size. If, as a single-individual organization, you do not have the knowledge or experience to develop a process to conduct investigations, consider using industry resources outside of your organization or contracting services from a third-party provider. As a one-person organization, you know exactly how it operates. With a sound documented process to conduct an investigation, you should be able to come up with relevant conclusions as long as you remain objective.
3.5.6 I nvestigation of Potential Noncompliance.
3.5.6.1 References. Section 5.71(a)(5).
3.5.6.2 Part 5 Requirement.
§ 5.71(a)(6), Safety performance monitoring and measurement.
(a) Any person required to have an SMS under this part must develop and maintain processes and systems to acquire data with respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following:
(6) Investigations of reports regarding potential non-compliance with regulatory standards or other safety risk controls established by the person through the safety risk management process established in subpart C of this part.
3.5.6.3 Discussion.
3.5.6.3.1 This subject is very similar to § 5.71(a)(5) in that the focus of the investigation should reveal information that, when utilized correctly, will identify system deficiencies that led to a noncompliance with regulatory standards or other safety risk controls. It is not as important to identify “who did it” as it is to determine why it happened. Within this process, it is important to distinguish between errors and intentional/willful noncompliant actions. Investigations of reports regarding potential noncompliance with regulatory standards or of inadequate safety risk controls established by the aviation organization must be mitigated through the organization’s corrective action process as required by § 5.75. Instances of noncompliance with an FAA regulation may be reported through the Voluntary Disclosure Reporting Program (VDRP), where applicable. For instances involving individual employee noncompliance with FAA regulations, these employees may use an Aviation Safety Action Program (ASAP), if one is available.
3.5.6.3.2 A n SMS does not relieve aviation organizations from other regulatory requirements. This also includes documented processes and procedures that may be contained in maintenance manuals or Aircraft Flight Manuals (AFM) or that are organization specific.
3-50
3.5.6.4 Implementation Strategies. Methods of conducting investigations of potential noncompliance can be accomplished in a manner similar to that for investigations of accidents and incidents.
3.5.7 Confidential Employee Reporting System.
3.5.7.1 References. Sections 5.21(a)(4) and 5.71(a)(7).
3.5.7.2 Part 5 Requirement.
§ 5.71(a)(7), Safety performance monitoring and measurement.
(a) Any person required to have an SMS under this part must develop and maintain processes and systems to acquire data with respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following:
(7) A confidential employee reporting system in which employees can report hazards, issues, concerns, occurrences, incidents, as well as propose solutions and safety improvements, without concern of reprisal for reporting.
3.5.7.3 Discussion. The data-gathering process should include information provided by the workforce. Frontline employees are in the best position to observe aspects of the operation or environment that are not expected and may not be included in audits or evaluation protocols. In this respect, employee reporting systems can fill important gaps in the organization’s data collection process.
3.5.7.3.1 Open communication is a key component of a successful SMS. A robust employee reporting system is a critical part of this communication concept.
Aviation organizations must have documented processes and procedures for employee reporting. The methods utilized could be an ASAP for employees covered by a memorandum of understanding or computer-based reporting for employees not covered by an ASAP. Smaller organizations could be part of a community organization working with a managed employee reporting program. The important thing is all employees in an aviation-related position must have a means of reporting hazards and issues they come across in their normal duties.
3.5.7.3.2 Part 5 requires aviation organizations to establish confidential employee reporting programs. Aviation organizations are not required to establish anonymous reporting systems in order to comply with this requirement. In fact, anonymous reporting has some disadvantages. While not prohibited by the regulation, anonymous reporting does not allow the aviation organization the ability to get additional information concerning the hazard or incident since the ability to reach out to the reporter is lost.
3.5.7.3.3 Employees should not fear retribution for reporting issues to the organization if those issues do not involve gross negligence or willful misconduct.
Employees should also be encouraged and have a means of providing
3-51 proposed solutions to the issues raised. In order to be effective, an environment in which employees feel comfortable reporting hazards, issues, and concerns, as well as occurrences, incidents, etc., should be established.
This is known as a nonretributive culture or a just culture. The accountable executive and management team need to encourage employees to report safety issues and not fear reprisal from management as long as the safety issues or identified hazards are not based on an employee’s gross negligence or deliberate misconduct. Policies that assure employees of fair treatment and a nonreprisal reporting policy that establishes clear standards of behavior are an essential part of the reporting process.
3.5.7.3.4 A key aspect of the confidential reporting system is that the submitter’s identity is protected. Therefore, aviation organizations must define methods for employee reporting and de-identification of sources without losing essential information. As confidential reporting procedures are developed and implemented, employees will begin to trust the organization to work toward hazard identification and elimination of systemic problems. When employees recognize the organization’s commitment to address employee reports, they will be more willing to report safety concerns.
3.5.7.3.5 With a confidential reporting program, a trusted source who manages the confidential reporting system hides the reporter’s identity from the process owners responsible for addressing the identified hazard. If additional information is needed, the trusted source could contact the reporter and request the additional information needed for the investigation. With an anonymous reporting program, any opportunity to obtain additional information is lost as the original reporter would remain unknown.
3.5.7.3.6 ASAPs can be used as part of the employee reporting system for the employee groups covered by a memorandum of understanding that is established with the implementation of an ASAP. However, the confidential employee reporting system required by part 5 must include all employees in the organization whose work directly affects aviation safety. Other methods of employee reporting may include hotlines, suggestion boxes, or information and forms for NASA ASRS. If WBAT is used, this system provides a portal for ASRS reporting. ASRS provides certificated employees with limited immunity in the form of waivers of sanctions for reported events with certain restrictions. One problem with using ASRS as an organization’s employee reporting program is that there is no transmittal to the organization when a report has been filed and no opportunity to conduct an investigation.
3.5.7.4 Implementation Strategies. The following examples demonstrate ways aviation organizations might choose to implement this requirement based on where the aviation organization may fall on the spectrum of size or complexity. The following are examples only and are not the only ways the requirements could be met.
3-52
3.5.7.4.1 An employee reporting system need not be highly sophisticated to be effective. The employees might report a hazard either orally or in a note or email to their supervisor. Several industry groups provide an employee reporting program for small aviation organizations and provide management and de-identification of information. They also provide access to other de-identified reports so identified hazards can be integrated into the participating aviation organization’s SMS. Aviation organizations on the higher end of the size or complexity spectrum may have an existing online employee reporting system or ASAPs for some employee groups. Data collection for the reporting system can take many forms, from a simple suggestion box to organizational websites or intranets or a dedicated email address. Data management can be accomplished with a common desktop spreadsheet, database software, or specialized software, such as WBAT.
3.5.7.5 Considerations for Small and Single-Individual Organizations.
3.5.7.5.1 Single-pilot operations and single-individual organizations pose unique situations when implementing an SMS. As a result of these factors, the following regulatory section is excepted from the implementation requirements for organizations with a single pilot who is the sole individual performing all necessary functions in the conduct and execution related to, or in direct support of, the safe operation of aircraft:
Section 5.71(a)(7): A confidential employee reporting system in which employees can report hazards, issues, concerns, occurrences, incidents, as well as propose solutions and safety improvements, without concern of reprisal for reporting.
3.5.7.5.2 Single-individual organizations are not required to have a confidential reporting system; however, they should have a process for documenting and managing hazards identified in the course of operations.
3.5.8 H azard Notification .
3.5.8.1 References. Sections 5.57 and 5.71(a)(8).
3.5.8.2 Part 5 Requirement.
§ 5.71(a)(8), Safety performance monitoring and measurement.
(a) Any person required to have an SMS under this part must develop and maintain processes and systems to acquire data with respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following:
(8) Investigations of hazard notifications that have been received from external sources.
3-53
3.5.8.3 Discussion.
3.5.8.3.1 When an aviation organization receives notification of a hazard from an external source, they are required to evaluate the hazard utilizing the same safety assurance processes as they would if they received a confidential employee report. The hazard should be evaluated using the SRM processes, and if the aviation organization determines the hazard would best be addressed by an external organization, they must notify that organization by utilizing their hazard notification process as required by § 5.57.
3.5.8.3.2 If the aviation organization is best suited to address the hazard, they should implement their SRM processes and procedures as defined in §§ 5.53 and 5.55.
3.5.9 Analysis of Data.
3.5.9.1 References. Sections 5.25(c)(2), 5.71(b), and 145.209(d)(2).
3.5.9.2 Part 5 Requirement.
§ 5.71(b), Safety performance monitoring and measurement.
(b) Any person required to have an SMS under this part must develop and maintain processes that analyze the data acquired through the processes and systems identified under paragraph (a) of this section and any other relevant data with respect to its operations, products, and services.
3.5.9.3 Discussion. Analysis involves examining data acquired from various sources as specified in § 5.71(a) in order to make inferences about the safety performance of operational systems and the SMS. It is common for organizations to treat each employee report, audit finding, or investigation in isolation. Often, system problems may not be seen if data points are examined in isolation. Thus, analysis processes should also look across individual reports and among various data sources for patterns or trends. The following is a starting point for developing and maintaining a process for analyzing data acquired through the data acquisition processes.
3.5.9.3.1 Establish the Context. Understand the safety performance objectives of the system, operations, and SMS. For system impacts and to analyze risk controls developed under SRM, process owners would also need to review the system analysis conducted under SRM.
3.5.9.3.2 Id entify the Objective of the Analysis. What is being analyzed: the safety performance of a system or an operation or the SMS itself?
3.5.9.3.3 S ecure Appropriate Data. Section 5.71(a) provides a framework for data sources. The data needed may be already on hand, or additional data-gathering, such as conducting a special audit with focus on a specific problem, may be needed.
3-54
3.5.9.3.4 S elect an Appropriate Data Analysis Method. Analysis need not be sophisticated to yield valuable results. For example, analysis of employee reports or qualitative analysis by SMEs may be the best method. If desired, several classification systems exist to help convert subjective, qualitative data into quantitative data for tracking and trend analysis. For routine reporting, analysis may consist of tracking such things as dispatch reliability per month, system or part failure rates, crew utilization/duty time, and events such as minor incidents, diversions, and precautionary engine shutdowns.
3.5.9.3.5 Reco mmendation. At this point, the individual conducting the analysis may compare performance against relevant organizational safety objectives. Unless the decisionmaker is personally conducting the analysis, an assessment recommendation may be made. In the case that a potential regulatory violation is discovered during analysis, the aviation organization may initiate a self-disclosure under voluntary reporting procedures.
3.5.9.3.6 Doc umentation. Prepare reports and records in a format appropriate to the operation.
Note: T he outputs from data analysis could be recorded in a simple recording medium (e.g., a notebook, paper files, a common desktop software, specialized systems, or a third-party software, such as WBAT).
3.5.9.4 Implementation Strategies. The following examples demonstrate ways aviation organizations might choose to implement this requirement based on where the aviation organization may fall on the spectrum of size or complexity. The following are examples only and are not the only ways the requirements could be met.
3.5.9.4.1 In a single-individual aviation organization, data acquisition may consist of making notes about how things did not go as planned, how communications were compromised at low altitude in certain flight route segments, etc. This data could be reviewed on a regular basis (perhaps monthly or quarterly).
Hazards could be analyzed in SRM, and mitigating actions could be recorded as changes to procedures, minimum performance requirements, etc. are identified. Aviation organizations on the higher end of the size or complexity spectrum may have the DOS or other individuals conduct the analysis of data.
This information would be shared with other departments and management during regularly scheduled meetings. Operational departments may have their own data analysis group reviewing data and analyzing the data by SMEs within the respective department, possibly supported and coordinated by a safety department.
3.5.9.5 Considerations for Small and Single-Individual Organizations. In a single-individual organization, the accountable executive will perform this
3-55 function. In a small organization, this may be accomplished by another member of management.
3.5.10 S afety Performance Assessment (SPA).
3.5.10.1 References. Sections 5.23(a)(2)(ii), 5.25(b)(1), and 5.73.
3.5.10.2 Part 5 Requirement.
§ 5.73, Safety performance assessment.
(a) Any person required to have an SMS under this part must conduct assessments of its safety performance against its safety objectives, which include reviews by the accountable executive, to:
(1) Ensure compliance with the safety risk controls established by the person.
(2) Evaluate the performance of the SMS.
(3) Evaluate the effectiveness of the safety risk controls established under § 5.55(c) and identify any ineffective controls.
(4) Identify changes in the operational environment that may introduce new hazards.
(5) Identify new hazards.
(b) Upon completion of the assessment, if ineffective controls or new hazards are identified under paragraphs (a)(2) through (5) of this section, the person must use the safety risk management process described in subpart C of this part.
3.5.10.3 Discussion. Under § 5.71, the aviation organization is required to collect and review safety performance data. This is typically conducted by the process owner of the area being reviewed. Analysis takes place, and informed decision making occurs. When developing SPAs, it is important to ensure the process is appropriate for the size of the organization. The SPA process should consider who makes the decisions regarding whether the organization’s safety performance is effective and whether the organization is meeting its safety objectives and expectations that are identified in the safety policy required by § 5.21. When reviewing the aviation organization’s goals and objectives, it is important to determine if the expected result is being achieved. The conclusions of the SPAs are reported to the accountable executive, who possesses ultimate authority to act on such conclusions, as necessary.
3.5.10.3.1 This requirement does not define a specific interval, but the assessments should be accomplished at a frequency to permit a change in processes or procedures to better align the organization to meet the safety objectives.
Aviation organizations must define the SPA review frequency in their processes and procedures. A small organization will likely conduct less frequent reviews as opposed to a larger organization. The assessment and review frequency should be sufficient to monitor activities so changes can be made in a timely manner.
3.5.10.3.2 The aviation organization must develop processes and procedures to ensure the SMS is working as designed. It should be generating appropriate data to
3-56 aid in identification of hazards and changes in the operational environment. It should also aid process owners in determining if any ineffective processes or procedures are in their areas of responsibility. Processes should be developed to update the SMS if any areas of deficiencies are noted. The SMS should provide information for proactive risk management.
3.5.10.3.3 This section also requires evaluating the effectiveness of safety risk controls developed to address any known hazards in the operational environment. Any new hazards identified in the assessment would need to be evaluated using the SRM processes in §§ 5.53 and 5.55.
3.5.10.3.4 Section 5.97(a) requires aviation organizations to retain outputs of their SRM processes as long as the risk controls are relevant to the operation. This is a good place to start when developing areas that should be monitored to ensure the risk controls remain effective.
3.5.10.3.5 Assessments can have one of the following general outcomes:
1. Performance is acceptable and objectives are being met.
2. Performance is not acceptable, and analysis suggests that the problem lies with conformity of either the regulations or organizational policy and procedures or the necessary resources have not been provided. In the event this occurs, corrective action under § 5.75 would be warranted.
3. Conformity with the risk controls and regulations or organizational policy and procedures appears to be satisfactory; however, desired results are not being obtained. In the event that this occurs, the SRM processes would be triggered.
4. New hazards or changes to the operational environment are discovered.
This may be due to new hazards having arisen since the system was designed or discovery of factors that were overlooked. In this case, as in the previous case, the SRM processes must be followed.
3.5.10.3.6 The results of assessments can be recorded in a paper or electronic medium
(e.g., in a common logbook-style binder, electronic file folder, common desktop software, specialized system, or third-party software program, such as WBAT).
3.5.10.4 Implementation Strategies. The following examples demonstrate ways aviation organizations might choose to implement this requirement based on where the aviation organization may fall on the spectrum of size or complexity. The following are examples only and are not the only ways the requirements could be met.
3.5.10.4.1 As an organization grows in size, it is normal to have additional personnel performing safety, quality, or internal evaluation functions. An SMS does not change the number and types of personnel in these situations as much as it
3-57 may change the way in which these persons and organizations work and interact. For example, in a small or lower complexity aviation organization, safety performance and assessment could be a documented activity performed by the accountable executive or a coordinated activity between the accountable executive and other operational managers, supported and coordinated by the DOS if the organization has this position. Risk acceptance would also normally fall to managers within this group. Aviation organizations on the higher end of the size or complexity spectrum may have the accountable executive, division vice presidents, and other defined leaders or process owners conduct the SPA. At each level, the organization would define who is responsible for making risk acceptance decisions and what actions should be taken to either correct the problem or design new risk controls. Larger organizations typically have standing management committees at the functional organization level (e.g., flight operations, technical operations/maintenance, in-flight services, and dispatch/operational control) and a second body at the corporate level to ensure integration, coordination, and review by the accountable executive.
3.5.10.5 Considerations for Small and Single-Individual Organizations. For single-individual organizations, an internal review (a self-assessment) may be as effective as a formal audit. Networking with other similar organizations and sharing safety information may provide insights into vulnerabilities that may exist in the organization’s processes and procedures. In small organizations, the reactive data gathered may not be statistically significant due to small sample sizes. This can be managed by looking beyond the data within your own organization. Where available, include data from industry organizations and associations, related industries, regulatory bodies, and safety boards.
3.5.11 C ontinuous Improvement .
3.5.11.1 References. Sections 5.25(b)(5) and 5.75.
3.5.11.2 Part 5 Requirement.
§ 5.75, Continuous improvement.
Any person required to have an SMS under this part must establish and implement processes to correct safety performance deficiencies identified in the assessments conducted under § 5.73.
3.5.11.3 Discussion. The final step within safety assurance is continuous improvement.
This process is designed to ensure that the aviation organization corrects substandard safety performance identified during the SPA in order to continuously improve safety.
3.5.11.3.1 In general, the corrective action process required by § 5.75 is triggered when an aviation organization’s employees are not utilizing the established risk controls that are integrated into a developed and implemented process or procedure. It is not always necessary to conduct a new safety risk analysis as
3-58 these deficiencies are typically employee performance-based and do not rise to the level of requiring initiating SRM processes. Quite often, retraining or counseling employees as to the need to follow processes and procedures is sufficient. The key is determining if the problem identified is systemic or employee related.
3.5.11.3.2 If the aviation organization has previously evaluated the process and procedure and determined the risk controls are effective but not properly used, the assessment should attempt to determine why the employee failed to follow the designed process. For example, if it has been found that an Airworthiness Directive (AD) has not been applied to a particular aircraft, the only correct action is to comply with the risk control (in this case, the AD), as that is required by regulation (14 CFR part 39, § 39.7), and an organization does not have discretion to take an action that violates a regulatory requirement. If the organization determined that the AD was missed due to an employee failing to input the records correctly into the tracking software, additional training of the employee may be an appropriate corrective action.
3.5.11.4 Impl ementation Strategies. Continuous improvement decision making is an output of the SPA process. Therefore, corrective actions discussed during the assessment apply to continuous improvement activities. The managers, committees, or working groups that make assessment decisions for the aviation organization would also determine the appropriate corrective actions based on the situation.
3.5.11.5 Safety Assurance Example. Flyslow Aviation becomes aware of an uncalibrated tool being issued to a technician through their confidential employee reporting program (§ 5.71(a)(7)). During a review of the tool control program, the process owner recognized this could be a potential noncompliance with regulatory standards7 (§ 5.71(a)(6)) and uses the VDRP to notify Flyslow Aviation’s FAA Certificate Management Team (CMT) of the incident.
7 The use of calibrated tools is regulatory under 14 CFR part 43, § 4 3.13 and §§ 121.369(b)(5), 135.427(b)(5), and 145.109(b).
3.5.11.5.1 A n investigation of the calibrated tool program is initiated as required by § 5.71(a)(3). Records of previous audits (§ 5.97(b)) were reviewed for any previous calibrated tool findings. It was noted during the records review and interviews with employees that the tool room changed from a manual calibrated tool tracking system to a computer-based tracking system since the previous audit occurred.
3.5.11.5.2 Uncalibrated tools being issued to employees is identified as a new hazard (§ 5.73(c)). Flyslow Aviation initiates their SRM process as required by § 5.51(d). A discussion of the SRM process is located in paragraph 3.4.6.