safety issues. Flight operations quality assurance (FOQA) and Line Operations Safety Audit (LOSA) programs may also be sources of information to monitor operations.
3. Large. Monitoring may involve multiple levels of management, safety professionals, functional area managers (such as DO, DOM, Chief Inspector, and Chief Pilot), trained auditors/analysts, and teams/groups of line managers. As in medium-size carriers, FOQA and LOSA programs may be employed. Operational processes may need to be coordinated across adjacent work function boundaries, so effective monitoring may also need to be coordinated.
(2) Monitoring of Operational Environment.
(a) SRR. Section 5.71(a)(2).
(b) Related Regulatory Requirements. None.
(c) ICAO/SMSPP. Element 3.1/Process 3.1.1.
(d) Objective. To monitor the operational environment to identify new or changed conditions.
(e) Part 5 Requirements.
§ 5.71 Safety performance monitoring and measurement.
(a) The certificate holder must develop and maintain processes and systems to acquire data with
respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following: …
(2) Monitoring of the operational environment to detect changes.
(f) Discussion. Monitoring of the operational environment involves practices that are similar to those of monitoring operational processes. The context for monitoring the operational environment of a system is developed from the system analysis that is conducted under SRM. Once the scope of the operational environment is defined under SRM, the operational environment must be monitored to assess impacts on aviation safety. For example, increases in the price of fuel may require airlines to change their scheduling, routes, and aircraft utilization.
(3) Auditing of Operational Processes and Systems.
(a) SRR. Section 5.71(a)(3).
(b) Related Regulatory Requirements. Section 5.23(a)(2)(ii).
(c) ICAO/SMSPP. Element 3.1/Processes 3.1.2 and 3.1.4.
(d) Objective. To provide the process owners with a means to obtain information about the performance of systems in their area of responsibility.
Par 3-5 Page 31
(e) Part 5 Requirements.
§ 5.71 Safety performance monitoring and measurement.
(a) The certificate holder must develop and maintain processes and systems to acquire data with respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following: …
(3) Auditing of operational processes and systems.
(f) Discussion.
1. Audits are a means of collecting data to confirm whether or not actual practices are being followed within a department. Audits should typically involve the operational management responsible for the system(s) being audited. Procedures for auditing should describe your audit process, criteria, scope, frequency, method for selecting auditors, and methods of documentation and recordkeeping. Audit planning should take into account the safety criticality of the processes to be audited and the results of previous audits. Auditors should not audit their own work, but may audit the work of others around them in the same department. Audit procedures should include the responsibilities and expectations for planning, conducting, reporting results of audits, maintaining records of audit results, and processes for auditing contractors and vendors, as necessary.
2. The results of audits can be recorded in paper format such as a common logbook-style binder, or in electronic media such as a desktop spreadsheet program or a program such as a WBAT.
(g) Scalability.
1. Small. The auditing process could be carried out periodically by the accountable executive/owner, key management person, or a trained employee as a collateral duty. Audits may also be carried out as a subfunction of normal business processes. For example, comparisons of deferred maintenance logs and repair part receipts are a form of safety auditing that are probably already accomplished routinely.
2. Medium. In a medium-size organization, the auditing process can be accomplished by operational departmental personnel, on a periodic basis, as determined by the needs of operational decision makers.
3. Large. In a large organization, the auditing processes are typically fulfilled by divisional auditors on a consistent basis. Large companies may already have safety and safety/quality auditors who perform this function or, in smaller divisions, they may be performed by personnel from inside the divisions.
(4) Evaluation of SMS and Operational Processes and Systems.
(a) SRR. Section 5.71(a)(4).
(b) Related Regulatory Requirements. Section 5.25(c)(3), Responsibilities of Safety Management Personnel.
Par 3-5 Page 32
(c) ICAO/SMSPP. Element 3.1/Process 3.1.3.
(d) Objective. To provide a source of information to the organization regarding the safety performance of operational systems and the SMS.
(e) Part 5 Requirements.
§ 5.71 Safety performance monitoring and measurement.
(a) The certificate holder must develop and maintain processes and systems to acquire data with respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following: …
(4) Evaluations of the SMS and operational processes and systems.
(f) Discussion. An evaluation is typically an independent review of the company’s processes, procedures, and systems. The evaluation process builds on the concepts of audit and inspection. An evaluation is an internal oversight tool that provides the accountable executive with a snapshot of the safety performance of the carrier’s operational processes and systems, as well as SMS processes. The evaluation should include all available data about the organization, including information from the audits conducted by the operational management.
1. Conducting evaluations at planned intervals will help the carrier’s management to determine if its safety management methods and practices are meeting safety objectives and expectations set out in the safety policy. Evaluation planning should take into account the safety criticality of the processes that are being evaluated and the results from previous evaluations. The scope, content, and frequency of evaluations should be based on the decision maker’s need for information to assess the health of operational processes and the SMS.
You will also need to define criteria for selecting evaluators.
2. The results of evaluations can be recorded in a paper or electronic medium in a common logbook-style binder, an electronic file folder, or a secure email account.
(g) Scalability.
1. Small. The evaluation process could be carried out periodically by the accountable executive/owner, a key management person, or designated employees as a collateral duty under the direction of the accountable executive.
2. Medium. This process could be accomplished by the Director of Safety (DOS) or Safety Department on a monthly, quarterly, or other periodic basis, as determined by the information needs of the accountable executive or other senior management decision makers.
Personnel resources to perform the observations and data collection for evaluations could be from a small, dedicated department or selected line personnel as a collateral duty.
3. Large. Evaluations could be accomplished by a Safety Department or an Internal Evaluation Program (IEP) office on a quarterly, annual, or other periodic basis, as determined by the information needs of the accountable executive or other senior management decision makers. Most part 121 companies have IEPs, and their outputs can be integrated into the SMS. Analysis of evaluations is typically performed by a Safety Department. The resulting data Par 3-5 Page 33
would be acted upon by the appropriate operational department with the Safety Department managing the data and assisting the responsible division(s) in resolving their issues. Most large organizations have standing management committees that consider results of evaluations and any corrective action needed.
(5) Investigation of Incidents and Accidents.
(a) SRR. Section 5.71(a)(5).
(b) Related Regulatory Requirements. None.
(c) ICAO/SMSPP. Element 3.1/Process 3.1.5.
(d) Objective. To gather information on accidents and incidents to identify potential weaknesses in operational systems.
(e) Part 5 Requirements.
§ 5.71 Safety performance monitoring and measurement.
(a) The certificate holder must develop and maintain processes and systems to acquire data with
respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following: …
(5) Investigations of incidents and accidents.
(f) Discussion. Investigations should be treated as an opportunity for organizational learning to prevent a repeat of errors and/or change company processes so that mistakes do not recur. Investigations should focus on what went wrong rather than who caused the error and emphasize improvement of safety performance. The organization should include investigation data, if available, from outside sources such as FAA or National Transportation Safety Board (NTSB) investigations and may, where appropriate, participate as a party to official investigations. The results of investigations can be recorded in paper or electronic medium in a common logbook-style binder, an electronic file folder, or other electronic system such as a WBAT or another suitable system.
(g) Scalability.
1. Small. Investigations can be conducted by the accountable executive or assigned employees.
2. Medium. Investigations can be conducted by a Safety Department, with additional assigned line personnel providing technical expertise.
3. Large. Investigations, depending on the situation, can be conducted by safety teams with specialized disciplines.
(6) Investigation of Potential Noncompliance.
(a) SRR. Section 5.71(a)(6).
Par 3-5 Page 34
(b) Related Regulatory Requirements. None.
(c) ICAO/SMSPP. Element 3.1/Process 3.1.5.
(d) Objective. To obtain information to determine compliance with regulations as well as underlying policies and procedures.
(e) Part 5 Requirements.
§ 5.71 Safety performance monitoring and measurement.
(a) The certificate holder must develop and maintain processes and systems to acquire data with respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following: …
(6) Investigations of reports regarding potential non-compliance with regulatory standards or other safety risk controls established by the certificate holder through the safety risk management process established in subpart B of this part.
(f) Discussion. This subject is very similar to the one described above in that the focus of the investigation should reveal information that, when utilized correctly, will concentrate on objective facts to identify system deficiencies, help prevent future recurrences, and improve system reliability. It is not as important to identify “who did it” as it is for you to learn why it happened. Within this process, it is important to distinguish between error and intentional/willful noncompliant actions. Investigations of reports regarding potential noncompliance with regulatory standards or of inadequate safety risk controls established by the certificate holder should be mitigated through the SRM process established in part 5, subpart B.
Instances of noncompliance with an FAA regulation may be reported through the Voluntary Disclosure Reporting Program (VDRP), where applicable. For instances involving individual employee noncompliance with FAA regulations, these employees may use an Aviation Safety Action Program (ASAP), if one is available.
(g) Scalability. Methods of conducting investigations of potential noncompliance can be accomplished in a manner similar to that for investigations of accidents and incidents.
(7) Confidential Employee Reporting System.
(a) SRR. Section 5.71(a)(7).
(b) Related Regulatory Requirements. None.
(c) ICAO/SMSPP. Element 3.1/Process 3.1.6.
(d) Summary. To provide a means for employees to communicate safety information to management.
Par 3-5 Page 35
(e) Part 5 Requirements.
§ 5.71 Safety performance monitoring and measurement.
(a) The certificate holder must develop and maintain processes and systems to acquire data with respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following: …
(7) A confidential employee reporting system in which employees can report hazards, issues, concerns, occurrences, incidents, as well as propose solutions and safety improvements.
(f) Discussion. The workforce is an important information source that should be included in the data-gathering process. Audits and evaluations bring decision makers important information and are essential to a proactive SA approach. However, these tools can be limited by the scope and content of their design. Front-line employees may observe aspects of the operation or environment that were not expected and were not included in audit or evaluation protocols. In this respect, the employee reporting system can fill in important gaps in the company’s data collection process.
1. In order to be effective, the organization needs to establish and maintain an environment in which employees feel comfortable to report hazards, issues, and concerns, as well as occurrences, incidents, etc., and propose safety solutions and improvements. The accountable executive and management team need to encourage employees to report safety issues and not fear reprisals from management. Policies that assure employees of fair treatment and clear standards of behavior are an essential part of the reporting process.
2. A key aspect of the confidential reporting system is that it is confidential.
Therefore, you must define methods for employee reporting and de-identification of sources without losing essential information. As you develop and employ the confidential reporting procedures and include this invaluable input in safety decisionmaking, employees will begin to trust the system to work toward elimination of systemic problems. This, in turn, will stimulate greater participation in employee reporting of safety concerns.
3. ASAPs can be used as part of the employee reporting system for the employee groups covered by the ASAPs. However, the confidential employee reporting system required by part 5 must include all employees in the company. Companies that do not have ASAPs may consider providing employees with a hotline, a suggestion box, or information and forms for the NASA ASRS. If a WBAT is used, this system provides a portal for ASRS reporting. ASRS provides certificated employees with limited immunity in the form of waivers of sanctions for reported events with certain restrictions.
(g) Scalability.
1. Small. An employee reporting system for a small company need not be highly sophisticated to be effective. The employees might report a hazard either orally or in a note or email to their supervisor. Collection tools could be a suggestion box, voice mail “hotline,” etc.
Par 3-5 Page 36
2. Medium and Large. A medium- or large-sized company will most likely have an existing online employee reporting system or ASAPs for some employee groups. Data collection for the reporting system can take many forms, from a simple suggestion box to company Web sites or intranets, or a dedicated email address. Data management can be accomplished with common desktop spreadsheet or database software or specialized software, such as a WBAT.
d. Analysis of Data.
(1) SRR. Section 5.71(b).
(2) Related Regulatory Requirements. Section 5.25(c)(2), Responsibilities of Safety Management Personnel.
(3) ICAO/SMSPP. Element 3.1/Process 3.1.7.
(4) Objective. To organize and examine data to determine its meaningfulness in making safety decisions.
(5) Part 5 Requirements.
§ 5.71 Safety performance monitoring and measurement.
(b) The certificate holder must develop and maintain processes that analyze the data acquired through the processes and systems identified under paragraph (a) of this section and any other relevant data with respect to its operations, products, and services.
(6) Discussion. Analysis involves examining data acquired from various sources in § 5.71(a) in order to make inferences about the safety performance of operational systems and the SMS.
(a) It is common for organizations to treat each employee report, audit finding, or investigation in isolation. Often, system problems may not be seen if data points are examined in isolation. Thus, analysis processes should also look across individual reports and among various data sources for patterns or trends.
(b) For example, in a recent case, examination of data from an airline’s FOQA program showed several instances of unstabilized approaches and exceeding of flap and landing gear speeds on approach. It would be easy to assume that pilot technique was the cause and then counsel or retrain the pilots. However, upon closer analysis and comparison of the events, the airline found that all of the instances were at one specific airport. Contacts with other airlines indicated similar experiences at the same airport. After further conversations with air traffic control (ATC), the parties concluded that, due to the traffic handling practices, it was common for flights to be vectored close and high, which resulted in the approach problems. After further conversations with ATC management, the problem was resolved.
Par 3-5 Page 37
(c) The following is a starting point for an analysis process.
1. Establish the context: Understand the safety performance objectives of the system, operations, or SMS. For system impacts, and to analyze risk controls developed under SRM, you would also need to review the system analysis conducted under SRM.
2. Identify the objective of the analysis: Are you analyzing the safety performance of a system, of an operation, or the SMS itself?
3. Secure appropriate data: Section 5.71(a) provides a framework for data sources. The data needed may be already on hand, or additional data-gathering may be needed, such as conducting a special audit with focus on a specific problem.
4. Select an appropriate data analysis method: Analysis need not be sophisticated to yield valuable results. For example, analysis of employee reports or qualitative analysis by SMEs may be the best method. If desired, several classification systems exist to help convert subjective, qualitative data into quantitative data for tracking and trend analysis. For routine reporting, analysis may consist of tracking such things as dispatch reliability per month, system or part failure rates, crew utilization/duty time, and events such as minor incidents, diversions, and precautionary engine shutdowns.
5. Recommendation: At this point, the person conducting the analysis may compare performance against relevant company safety objectives. Unless the decision maker is personally conducting the analysis, an assessment recommendation may be made. In the case that a potential regulatory violation is discovered during analysis, the carrier may initiate a self-disclosure under voluntary reporting procedures.
6. Documentation: Prepare reports and records in a format appropriate to your operation.
(d) The outputs from data analysis could be recorded in a simple recording medium such as a notebook, paper files, common desktop software, or specialized systems such as a WBAT.
(7) Scalability.
1. Small. Analysis of the data gathered could be done by the accountable executive, the DOS, other individual managers, or other designated employees as a collateral duty.
2. Medium. Analysis of data could be done by the DOS, other individuals within a Safety Department, or a person(s) within each department, and then shared with other departments and management during regularly scheduled meetings.
3. Large. Operational departments may have their own data analysis group reviewing data and analyzing the data by SMEs within the respective department, possibly supported and coordinated by a Safety Department.
Par 3-5 Page 38
e. Safety Performance Assessment.
(1) SRR. Section 5.73.
(2) Related Regulatory Requirements. Sections 5.23(a)(2)(ii) and 5.25(b)(1).
(3) ICAO/SMSPP. Element 3.1/Processes 3.1.8, 3.2, and 3.3.2.
(4) Objective. To make decisions regarding safety performance with reference to safety objectives and regulatory compliance.
(5) Part 5 Requirements.
§ 5.73 Safety performance assessment.
(a) The certificate holder must conduct assessments of its safety performance against its safety objectives, which include reviews by the accountable executive, to:
(1) Ensure compliance with the safety risk controls established by the certificate holder;
(2) Evaluate the performance of the SMS.
(3) Evaluate the effectiveness of the safety risk controls established under § 5.55(c) and identify any ineffective controls.
(4) Identify changes in the operational environment that may introduce new hazards.
(5) Identify new hazards.
(b) Upon completion of the assessment, if ineffective controls or new hazards are identified under
paragraph (a)(2) through (a)(5) of this section, the certificate holder must use the safety risk
management process described in subpart C of this part.
(6) Discussion. Under § 5.71, collected safety performance data is analyzed and the results are used for informed decisionmaking. The assessment process is where these decisions are made. Decisions are made by personnel with assigned responsibility and authority. The SA process should consider who makes the decisions regarding whether the company’s safety performance is effective and whether the company is meeting its safety objectives and expectations that are identified in the safety policy required by § 5.21. The conclusions of the safety assessments are reported to the accountable executive, who possesses ultimate authority to act on such conclusions, as necessary.
(a) Assessments can have one of the following general outcomes:
• Performance is acceptable and objectives are being met.
• Performance is not acceptable, and analysis suggests that the problem lies with conformity with regulations or company policy and procedures, or necessary resources have not been provided. In the event this occurs, corrective action under § 5.75 would be warranted.
• Conformity with the risk controls and regulations appears to be satisfactory;
however, desired results are not being obtained. In the event that this occurs, the SRM processes would be triggered.
• New or uncontrolled hazards are discovered. This may be due to new hazards having arisen since the system was designed or discovery of factors Par 3-5 Page 39
that were overlooked. In this case, as in the previous, the SRM processes must be followed.
(b) The results of assessments can be recorded in a paper or electronic medium in a common logbook-style binder, an electronic file folder, common desktop software, or specialized system, such as a WBAT.
(7) Scalability.
(a) Small and Medium. As an organization grows in size, it is normal to have additional personnel performing safety, quality, or internal evaluation functions. An SMS does not change the number and types of personnel in these situations as much as it may change the way in which these persons and organizations work and interact. For example, safety performance and assessment could be a documented activity performed by the accountable executive or a coordinated activity between the accountable executive and other operational managers, supported and coordinated by the DOS. Risk acceptance would also normally fall to managers within this group.
(b) Large. This process is best addressed at the highest level in the organization and involves the accountable executive, division vice presidents, and other defined leaders and decision makers. At each level, the company would define who is responsible for making risk acceptance decisions and what actions should be taken to either correct the problem or design new risk controls. Larger companies typically have standing management committees at the functional organization level (e.g., flight operations, technical operations/maintenance, in-flight services, dispatch/operational control) and a second body at the corporate level to assure integration, coordination, and review by the accountable executive.
f. Continuous Improvement.
(1) SRR. Section 5.75.
(2) Related Regulatory Requirements. None.
(3) ICAO/SMSPP. Element 3.3/Process 3.3.1.
(4) Objective. To provide the organization’s decision makers with a means to correct safety performance deficiencies identified in the safety performance assessment process.
(5) Part 5 Requirements.
§ 5.75 Continuous improvement.
The certificate holder must establish and implement processes to correct safety performance deficiencies
identified in the assessments conducted under § 5.73.
Par 3-5 Page 40
(6) Discussion.
(a) The final step within SA is continuous improvement. This process is designed to ensure that you are correcting substandard safety performance identified during the safety performance assessment in order to continuously improve safety performance.
(b) As discussed in subparagraph 3-5e, Safety Performance Assessment), the corrective action process of § 5.75 is triggered when conformity with risk controls has been found to be deficient. In this case, it is not necessary to conduct a new safety risk analysis. Risk has already been assessed as being unacceptable without satisfactory completion of the risk control. For example, if it has been found that an Airworthiness Directive (AD) has not been applied to a particular aircraft, the only correct action is to comply with the risk control (in this case, the AD). The risk of flying the airplane without the AD has already been assessed as unacceptable.
(7) Scalability. Continuous improvement decisionmaking is an output of the performance assessment process. Therefore, the arrangements discussed for assessment apply to scalability for continuous improvement activities. The managers, committees, or working groups that make assessment decisions would also lay out courses of corrective action.
g. An SA Example. This example builds on the scenario discussed in the SRM example, which can be found in the SRM section, subparagraph 3-4e. In that example, an airline addressed the unacceptable risk of an airport approach procedure by imposing risk controls consisting of a change to operational procedures, including programming of the FMS and associated approach procedure training for flightcrew members.
(1) As required by §§ 5.71(a)(1) and 5.73(a)(1) and (a)(3), the air carrier specifically monitored the flight operations and the associated controls at the airport by the operational control/dispatch organization as part of their normal activities. This organization was tasked with assuring that crews assigned to the specific flights were appropriately qualified (thus verifying that this part of the risk control was being employed), and that the airplanes scheduled for the flights had appropriate operational equipment. As required by § 5.71(a)(2), the air carrier also monitored the airport to determine whether the altered approach procedure was still necessary.
(2) Another part of the risk control monitoring and auditing was conducted using structured line checks pursuant to § 5.71(a)(3). This method was used to conduct audits of specific flight operations (in addition to the airline’s normal checking procedures), thus providing data to verify that the procedures put in place provided risk controls that are effective, as required by § 5.73(a)(3). At the same time, per § 5.71(a)(7), the airline’s employee reporting system was available for pilots to submit their concerns to the company. Line check and employee reporting data were collected and analyzed under § 5.71, and then assessed by management under § 5.73.
(3) If any deficiencies were found, an analysis would have been conducted to ascertain whether the deficiencies should be addressed through SRM (per § 5.73(b)) or a correction (per § 5.75). If the root cause of the deficiencies was shortfalls in compliance with the designed operational process (e.g., required training was not completed), the deficiency should be Par 3-5 Page 41
addressed through a correction, per § 5.75. However, if the deficiency was associated with the basic design of the control (e.g., inadequate training curriculum), the SRM process would be repeated to revise the procedure in the areas where it was found deficient, per § 5.73(b).
3-6. SUBPART E: SAFETY PROMOTION.
a. Introduction.
(1) SRR. Part 5, subpart E, Safety Promotion
(2) Related Regulatory Requirements. Section 5.23(a)(2)(ii), Accountabilities of Members of Management.
(3) ICAO/SMSPP. Component 4.0.
(4) Objective. To provide employees with effective SMS training commensurate with their safety responsibilities, and to create a means to deliver organization-wide safety communication.
(5) Discussion. Because a key component of SMSs is the effective control of risk, every member in your organization must understand and take responsibility for the role they play in controlling risk by their actions and behavior. These members should also have access to up-to-date safety information so that they can properly fulfill their roles.
b. Competencies and Training.
(1) SRR. Section 5.91.
(2) Related Regulatory Requirements. None.
(3) ICAO/SMSPP. Element 4.1/Processes 4.1.1 and 4.1.2.
(4) Objective. To assure that employees are competent to perform their SMS-related duties.
(5) Part 5 Requirements.
§ 5.91 Competencies and training.
The certificate holder must provide training to each individual identified in § 5.23 to ensure the individuals
attain and maintain the competencies necessary to perform their duties relevant to the operation and
performance of the SMS.
(6) Discussion. Your organization will need to train its employees. Your employees may receive initial safety training for them to understand and perform their safety duties.
Recurrent training may also be necessary to reinforce these skills. For example, a Safety Department employee will probably need more detailed SMS training (such as safety risk analysis, system evaluation, system assessment, and data mining, auditing, and inspections) than will a baggage handler. Line managers may need to know the potential consequences of safety Par 3-5 Page 42