Par 3-4 Page 24
4. The type of consequence (e.g., error, failure, accident, or incident) that is envisioned normally drives the estimate of severity. For example, if the hazard could result in controlled flight into terrain (CFIT), the severity of this outcome is normally major, if not catastrophic. Conversely, tire failures, while potentially leading to a fatal accident, more often lead only to aircraft damage.
5. Even where the best estimate has to be based on reasonable expert judgment, effective risk management can be accomplished by applying a disciplined analysis. The outputs could be recorded in a simple recording medium such as a notebook, basic desktop software, a WBAT,5 or a spreadsheet application.
(g) Scalability.
1. Small. Risk analysis could be performed by the owner/manager, and/or another employee(s). It might be performed in conjunction (by the same individual/group) with system description and analysis, hazard identification, risk assessment, and risk control.
2. Medium. Risk analysis could be performed by a member of management and/or the designated management representative with a small workgroup of company SMEs and stakeholders.
3. Large. Risk analysis might be performed at multiple organizational levels,
(e.g., the corporate, division, or department levels) and facilitated by the Safety Department or specially trained analytical personnel shared with other departments. The organization might have standing committees of SMEs and stakeholders participating at various levels.
(2) Safety Risk Assessment.
5 The Web-Based Analysis Tool (WBAT) is a federally developed and funded software system that may be used to assist air carriers with data management. WBAT also contains an SMS implementation plan manager module, which supports the air carrier’s implementation of SMS.
Par 3-4 Page 25
(a) SRRs. Section 5.55(b).
(b) Related Regulatory Requirements.
• Section 5.23(a)(2)(i), Accountabilities of Members of Management.
• Section 5.23(b), Decision Authority Regarding Safety Risk Acceptance.
(c) ICAO/SMSPP. Element 2.2/Process 2.2.2.
(d) Objective. To make a decision regarding the acceptability of operation in the presence of an identified hazard.
(e) Part 5 Requirements.
(f) Discussion. Once the risk is analyzed, you must assess whether the risk is acceptable. A common tool used in risk assessment decisions is a risk matrix. A risk matrix provides you with a way to integrate the effect of severity of the outcome and the probability of occurrence, which enables you to assess risks, compare potential effectiveness of proposed risk controls, and prioritize risks where multiple risks are present.
1. If a risk matrix is used, the carrier should develop criteria for severity and likelihood that are appropriate for their type of operations and their operational scenario. For example, severity levels are sometimes defined in terms of a dollar value of potential damage. In this case, different types of airplanes operated and their relative values would dictate different scales between carriers. Likewise, the method that the carrier uses to estimate likelihood will have an effect on how likelihood scales are defined. If the carrier prefers to use quantitative estimates (e.g., probability), the scales would be different than one that prefers to use qualitative estimates. Figure 3-2 below depicts a sample risk matrix.
§ 5.55 Safety risk assessment and control.
(b) The certificate holder must define a process for conducting risk assessment that allows for the determination of acceptable safety risk.
Par 3-4 Page 26 FIGURE 3-2. SAMPLE RISK MATRIX
2. Risk assessment is based on judgment, experience, and input from data collection tools (DCT) and previous processes. If the risk is acceptable, it may be done with the SRM process. If this is the case, then the system may be placed into operation and monitored in the SA process. If you decide the risk is not acceptable, you will need to follow the next step in SRM, which is developing risk controls.
3. Risk assessments must also involve the levels of management with the authority to make risk decisions, deciding what is or is not an acceptable risk for the systems within their area of operational responsibility. For example, dispatching a flight that presents a medium or high risk might require the Chief Pilot or DO to approve or authorize the flight. For large scale operational decisions, the accountable executive may be the only appropriate person to make these risk acceptance decisions. Thus, the person responsible for making these risk acceptance decisions will depend on the scope of the proposed change to the operation and the level of risk presented to the operation.
4. The outputs of this process could be recorded on paper or via an electronic medium, such as a WBAT.
(g) Scalability.
1. Small. Risk assessment could be performed by the owner/manager, and/or another employee(s) making the risk decisions. Risk acceptance would also probably be conducted by this individual/group. These processes could be similar to a flight risk management process, or you could use a risk matrix.
2. Medium and Large. SRM should be coordinated across the divisional and geographic units of the company to assure integrated decisionmaking and communication.
Decisions involving multiple systems may require joint decisionmaking among departments or managers responsible for those systems. Many companies have standing committees made up of senior managers, who are the decision makers, supported by working groups of technical
Par 3-4 Page 27 personnel. For example, the accountable executive could make company-level decisions and department managers could make the decisions for their operations. A risk matrix may be useful to determine who makes the risk decision, whether the risk is acceptable, or to determine the priority for mitigating risk.
(3) Safety Risk Control.
(a) SRRs. Sections 5.55(c) and (d).
(b) Related Regulatory Requirements. None.
(c) ICAO/SMSPP. Element 2.2/Process 2.2.3.
(d) Objective. To develop a risk control that reduces risk to an acceptable level.
(e) Part 5 Requirements.
(f) Discussion.
1. After hazards and associated risk are fully understood, risk controls must be designed for risks that the carrier deems unacceptable. This is accomplished using their risk assessment process, as specified in § 5.55(b). Examples of risk controls include new processes, equipment, training, new supervisory controls, new equipment or hardware, new software, changes to staffing arrangements, or any of a number of other system changes. In short, anything that would lessen the likelihood or severity of a potential incident/accident.
2. Next, look at the system with the proposed control in place to determine if the level of risk is now acceptable and the proposed control does not introduce unintended consequences or new hazards. This is commonly referred to as substitute risk. Section 5.55(d) requires you to evaluate whether the risk will be acceptable with the proposed safety risk control applied. The risk that remains is often called residual risk.
(g) Scalability.
1. Small. The risk control process could be a documented activity performed by the owner/manager and/or another employee(s) designing and evaluating the risk controls. It might be performed in conjunction (by the same individual/group) with system description and analysis, hazard identification, risk analysis, and risk assessment.
2. Medium and Large. The risk control process could be performed by a member of management or SMS management representatives with a small workgroup of company SMEs and stakeholders to design the risk controls. There would be interdepartmental § 5.55 Safety risk assessment and control.
(c) The certificate holder must develop and maintain processes to develop safety risk controls that are necessary as a result of the safety risk assessment process under paragraph (b) of this section.
(d) The certificate holder must evaluate whether the risk will be acceptable with the proposed safety risk control applied, before the safety risk control is implemented.
Par 3-4 Page 28 coordination before the controls are implemented. After the control is approved, it is implemented and documented through the company’s publication system. Implementation of risk controls may include distribution of manual revisions and training of company personnel.
e. SRM Example. The scenario below illustrates how these SRM processes would be applied in an operational context.
(1) An airline receives notification that the instrument landing system (ILS) at one of its destination airports is scheduled to be out of service for a period of time and, therefore, the flights will have to use a less familiar Nonprecision Approach (NPA) procedure. Concern is expressed by the flightcrews about the procedure and the differences between fleets that will affect operational procedures and the currency of training. Under § 5.51, this triggers the SRM processes and procedures.
(2) Pursuant to § 5.53(a) and (b), the SRM process starts with an analysis of the systems involved, with the addition of a new procedure for the aircraft fleet. This would include analysis of the flight operations and training program. In this case, the company has two fleets of airplanes that are operated to the destination airport in question. The equipment, and therefore the procedures and flightcrew training in these two fleets, differs. By analyzing the system, they identify a hazard (per § 5.53(c)), in that the pilots are not properly trained to fly the procedures that are required for that specific airport.
(3) To analyze the risk associated with this hazard (as required by § 5.55(a)), a team is assembled with representatives from management, the training and standardization organization, the flight safety organization, the organization responsible for dispatch and operational control, and the pilot’s union. Representatives of the FAA Certificate Management Team (CMT) also attend the work team’s meetings. The risk analysis consists of documented discussions among the assembled SMEs.
(4) As a result of the risk analysis, a risk assessment is done (per § 5.55(b)). As part of the assessment, it is determined that the existing operational procedures and training used by one of the fleets are not acceptable for safe operation into the airport without revision and additional training. A management decision is made that the procedures and associated training currently in use will not provide an acceptable level of safety (ALoS) for operations into the airport.
(5) After determining that the risk is unacceptable, risk controls are designed under § 5.55(c). The risk control in this case consists of a change to operational procedures, including programming of the flight management system (FMS) and associated approach procedure training for flightcrew members. Prior to implementation of the risk controls, the company checks the resulting procedures in flight simulators to ensure that the risk control will mitigate risk to an acceptable level, as required by § 5.55(d).
(6) Per § 5.23(b), the solution is submitted to the fleet Chief Pilots for management review and acceptance of the mitigated risk. Division-level management and top management are briefed on the operation at regular management reviews. Once the controls are approved by all concerned, they are implemented operationally. Per the SA requirements, these controls are
Par 3-4 Page 29 specifically targeted for performance monitoring by the operational control/dispatch organization and the flight operations organization line checks.
3-5. SUBPART D: SAFETY ASSURANCE.
a. Introduction.
(1) SRR. Part 5, subpart D.
(2) Related Regulatory Requirements.
• Section 5.23(a)(2), Accountabilities of Members of Management.
• Section 5.25(a)(4), Accountable Executive—Ultimate Responsibility for Safety Performance.
• Section 5.25(b)(1), Accountable Executive—Responsibility to Ensure SMS Implementation and Performance.
(3) ICAO/SMSPP. Component 3.0.
(4) Summary. SA consists of the following five processes, divided into three sections of the regulation.
(a) Safety Performance Monitoring and Measurement § 5.71.
• System monitoring,
• Data acquisition, and
• Analysis.
(b) Safety Performance Assessment § 5.73. System assessment.
(c) Continuous Improvement § 5.75. Continuous improvement.
NOTE: Figure 3-3, Safety Assurance Processes and Regulatory Requirements, depicts the SA processes and their associated regulatory requirements.
FIGURE 3-3. SAFETY ASSURANCE PROCESSES AND REGULATORY
REQUIREMENTS SA 5.71 Safety (Subpart c) System Performance Inputs From SRM: 5.71 (a) Monitoring Operation/ * 5.71 (a)(1)&(2) Monitoring Monitoring and Measurement Data Acquisition 5.71 (a)(3) Internal Audits & Process * 5.71 (a)(4) Internal Evaluation
5.71 (a)(5)&(6) Investigations
5.71(a)(7) Employee Reporting Analysis
5.71 (b) Analysis of Data - How is this going to be _ analyzed? By whom?
5.75 Continuous Improvement System 5.73(a) System Assessment 5.73 Safety Assmt, 5.73(a) Management Review Performance Outputs To SRM:
5.73(b) Assessment Preventive/ Corrective * Note: Each data source should be traceable Action 5.75 through analysis 5.71(b) assessment 5.73(a) And Corrective Action 5.75 where necessary.
b. Safety Performance Monitoring and Measurement: Overview. This section of the regulation consists of the following processes: Monitoring of Operational Processes (subparagraph 3-5c(1)), Monitoring of the Operational Environment (subparagraph 3-5c(2), Data Acquisition (subparagraphs 3-5c(3) (7)), and Analysis of Data (subparagraph 3-5d).
(1) SRR. Section 5.71.
(2) Related Regulatory Requirements. None.
(3) ICAO/SMSPP. Element 3.1.
c. Safety Performance Monitoring and Measurement: Required Data Acquisition Processes. This subsection consists of the following seven data acquisition processes:
Monitoring of Operational Processes, Monitoring of the Operational Environment, Auditing of Operational Processes, Evaluation of the SMS and Operational Processes, Investigations:
Accidents and Incidents, and Investigation: Potential Noncompliance and Confidential Employee Reporting System.
(1) Monitoring of Operational Processes.
(a) SRR. Section 5.71(a)(1).
Par 3-5 Page 30
Par 3-5 Page 31
(b) Related Regulatory Requirements.
• Part 121, § 121.703, Service Difficulty Reports.
• Section 121.705, Mechanical Interruption Summary Reports.
(c) ICAO/SMSPP. Element 3.1/Process 3.1.1.
(d) Objective. To observe the day to day, flight by flight, job by job performance of operational systems and their associated risk controls.
(e) Part 5 Requirements.
(f) Discussion. Monitoring operational processes is what supervisors do on a day-to-day basis (e.g., direct supervision of employee activities, monitoring of pilot currency, and monitoring Minimum Equipment List (MEL) status). Monitoring also involves reviewing data that is collected for operational purposes to look for anything of safety significance (e.g., duty logs, crew reports, work cards, process sheets, and reports from the employee safety feedback system). This may include monitoring products and services from outside sources that are used in the certificate holder’s operations.
(g) Scalability.
1. Small. Most of the data/information-gathering for monitoring of operational processes will likely occur as a normal business process by the management personnel who are directly involved in the day-to-day operations. For example, regularly reviewing (e.g., weekly, monthly, or quarterly) the flight dispatch logs and crewmember duty records is a form of monitoring and could be conducted during the normal course of duties.
2. Medium. Line managers and departmental or key management personnel may observe and review day-to-day activity, noting work task inconsistencies and potential safety issues. Flight operations quality assurance (FOQA) and Line Operations Safety Audit (LOSA) programs may also be sources of information to monitor operations.
3. Large. Monitoring may involve multiple levels of management, safety professionals, functional area managers (such as DO, DOM, Chief Inspector, and Chief Pilot), trained auditors/analysts, and teams/groups of line managers. As in medium-size carriers, FOQA and LOSA programs may be employed. Operational processes may need to be coordinated across adjacent work function boundaries, so effective monitoring may also need to be coordinated.
§ 5.71 Safety performance monitoring and measurement.
(a) The certificate holder must develop and maintain processes and systems to acquire data with respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following:
(1) Monitoring of operational processes.
Par 3-5 Page 32
(2) Monitoring of Operational Environment.
(a) SRR. Section 5.71(a)(2).
(b) Related Regulatory Requirements. None.
(c) ICAO/SMSPP. Element 3.1/Process 3.1.1.
(d) Objective. To monitor the operational environment to identify new or changed conditions.
(e) Part 5 Requirements.
(f) Discussion. Monitoring of the operational environment involves practices that are similar to those of monitoring operational processes. The context for monitoring the operational environment of a system is developed from the system analysis that is conducted under SRM. Once the scope of the operational environment is defined under SRM, the operational environment must be monitored to assess impacts on aviation safety. For example, increases in the price of fuel may require airlines to change their scheduling, routes, and aircraft utilization.
(3) Auditing of Operational Processes and Systems.
(a) SRR. Section 5.71(a)(3).
(b) Related Regulatory Requirements. Section 5.23(a)(2)(ii).
(c) ICAO/SMSPP. Element 3.1/Processes 3.1.2 and 3.1.4.
(d) Objective. To provide the process owners with a means to obtain information about the performance of systems in their area of responsibility.
(e) Part 5 Requirements.
§ 5.71 Safety performance monitoring and measurement.
(a) The certificate holder must develop and maintain processes and systems to acquire data with respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following: …
(2) Monitoring of the operational environment to detect changes.
§ 5.71 Safety performance monitoring and measurement.
(a) The certificate holder must develop and maintain processes and systems to acquire data with respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following: …
(3) Auditing of operational processes and systems.
Par 3-5 Page 33
(f) Discussion.
1. Audits are a means of collecting data to confirm whether or not actual practices are being followed within a department. Audits should typically involve the operational management responsible for the system(s) being audited. Procedures for auditing should describe your audit process, criteria, scope, frequency, method for selecting auditors, and methods of documentation and recordkeeping. Audit planning should take into account the safety criticality of the processes to be audited and the results of previous audits. Auditors should not audit their own work, but may audit the work of others around them in the same department. Audit procedures should include the responsibilities and expectations for planning, conducting, reporting results of audits, maintaining records of audit results, and processes for auditing contractors and vendors, as necessary.
2. The results of audits can be recorded in paper format such as a common logbook-style binder, or in electronic media such as a desktop spreadsheet program or a program such as a WBAT.
(g) Scalability.
1. Small. The auditing process could be carried out periodically by the accountable executive/owner, key management person, or a trained employee as a collateral duty. Audits may also be carried out as a subfunction of normal business processes. For example, comparisons of deferred maintenance logs and repair part receipts are a form of safety auditing that are probably already accomplished routinely.
2. Medium. In a medium-size organization, the auditing process can be accomplished by operational departmental personnel, on a periodic basis, as determined by the needs of operational decision makers.
3. Large. In a large organization, the auditing processes are typically fulfilled by divisional auditors on a consistent basis. Large companies may already have safety and safety/quality auditors who perform this function or, in smaller divisions, they may be performed by personnel from inside the divisions.
(4) Evaluation of SMS and Operational Processes and Systems.
(a) SRR. Section 5.71(a)(4).
(b) Related Regulatory Requirements. Section 5.25(c)(3), Responsibilities of Safety Management Personnel.
(c) ICAO/SMSPP. Element 3.1/Process 3.1.3.
(d) Objective. To provide a source of information to the organization regarding the safety performance of operational systems and the SMS.
Par 3-5 Page 34
(e) Part 5 Requirements.
(f) Discussion. An evaluation is typically an independent review of the company’s processes, procedures, and systems. The evaluation process builds on the concepts of audit and inspection. An evaluation is an internal oversight tool that provides the accountable executive with a snapshot of the safety performance of the carrier’s operational processes and systems, as well as SMS processes. The evaluation should include all available data about the organization, including information from the audits conducted by the operational management.
1. Conducting evaluations at planned intervals will help the carrier’s management to determine if its safety management methods and practices are meeting safety objectives and expectations set out in the safety policy. Evaluation planning should take into account the safety criticality of the processes that are being evaluated and the results from previous evaluations. The scope, content, and frequency of evaluations should be based on the decision maker’s need for information to assess the health of operational processes and the SMS.
You will also need to define criteria for selecting evaluators.
2. The results of evaluations can be recorded in a paper or electronic medium in a common logbook-style binder, an electronic file folder, or a secure email account.
(g) Scalability.
1. Small. The evaluation process could be carried out periodically by the accountable executive/owner, a key management person, or designated employees as a collateral duty under the direction of the accountable executive.
2. Medium. This process could be accomplished by the Director of Safety (DOS) or Safety Department on a monthly, quarterly, or other periodic basis, as determined by the information needs of the accountable executive or other senior management decision makers.
Personnel resources to perform the observations and data collection for evaluations could be from a small, dedicated department or selected line personnel as a collateral duty.
3. Large. Evaluations could be accomplished by a Safety Department or an Internal Evaluation Program (IEP) office on a quarterly, annual, or other periodic basis, as determined by the information needs of the accountable executive or other senior management decision makers. Most part 121 companies have IEPs, and their outputs can be integrated into the SMS. Analysis of evaluations is typically performed by a Safety Department. The resulting data would be acted upon by the appropriate operational department with the Safety Department managing the data and assisting the responsible division(s) in resolving their issues. Most large organizations have standing management committees that consider results of evaluations and any corrective action needed.
§ 5.71 Safety performance monitoring and measurement.
(a) The certificate holder must develop and maintain processes and systems to acquire data with respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following: …
(4) Evaluations of the SMS and operational processes and systems.
Par 3-5 Page 35
(5) Investigation of Incidents and Accidents.
(a) SRR. Section 5.71(a)(5).
(b) Related Regulatory Requirements. None.
(c) ICAO/SMSPP. Element 3.1/Process 3.1.5.
(d) Objective. To gather information on accidents and incidents to identify potential weaknesses in operational systems.
(e) Part 5 Requirements.
(f) Discussion. Investigations should be treated as an opportunity for organizational learning to prevent a repeat of errors and/or change company processes so that mistakes do not recur. Investigations should focus on what went wrong rather than who caused the error and emphasize improvement of safety performance. The organization should include investigation data, if available, from outside sources such as FAA or National Transportation Safety Board (NTSB) investigations and may, where appropriate, participate as a party to official investigations. The results of investigations can be recorded in paper or electronic medium in a common logbook-style binder, an electronic file folder, or other electronic system such as a WBAT or another suitable system.
(g) Scalability.
1. Small. Investigations can be conducted by the accountable executive or assigned employees.
2. Medium. Investigations can be conducted by a Safety Department, with additional assigned line personnel providing technical expertise.
3. Large. Investigations, depending on the situation, can be conducted by safety teams with specialized disciplines.
(6) Investigation of Potential Noncompliance.
(a) SRR. Section 5.71(a)(6).
(b) Related Regulatory Requirements. None.
(c) ICAO/SMSPP. Element 3.1/Process 3.1.5.
§ 5.71 Safety performance monitoring and measurement.
(a) The certificate holder must develop and maintain processes and systems to acquire data with respect to its operations, products, and services to monitor the safety performance of the organization. These processes and systems must include, at a minimum, the following: …
(5) Investigations of incidents and accidents.