Appendix 2 modeling safety standard must be defined, based on the prevailing prescriptive rules, that serves as the baseline for evaluation of fatigue or performance estimates under the FRMS operation.
(3) The collection and analysis of data are a prerequisite for the approval of the FRMS and issuance of Operations Specification (OpSpec) A318, Authorization to Conduct Operations under an FRMS. Once the FRMS is approved, similar measurements will be obtained periodically as part of the SA process to confirm the continued effectiveness of the FRMS relative to the established safety standards.
c. Process 3.0: Identify Hazards. Identify the specific nature of the fatigue hazard(s) associated with the proposed operations where FRMS will be applied that may need to be managed. The current edition of ICAO Annex 6, Part I, Appendix 8 requires that a certificate holder develop, maintain, and document three types of processes for fatigue hazard identification. These are referred to as predictive, proactive, and reactive processes. A certificate holder shall develop and maintain the following three fundamental and documented processes for fatigue hazard identification as part of the approved FRMS:
(1) Predictive. The predictive process shall identify fatigue hazards by examining crew scheduling and taking into account factors known to affect sleep and fatigue and their effects on performance. Methods of examination may include but are not limited to:
Certificate holder or industry operational experience and data collected on similar types of operations;
Evidence-based scheduling practices; and Biomathematical models.
(2) Proactive. The proactive process shall identify fatigue hazards within current flight operations.
(a) Methods. Methods of examination may include but are not limited to:
Self-reporting of fatigue risks;
Crew fatigue surveys;
Relevant flight and cabin crew performance data;
Relevant flight and cabin crew sleep data;
Available safety databases and scientific studies; and Analysis of planned versus actual time worked.
(b) Crewmember Fatigue Reports. Reports citing high fatigue levels or fatigue-related performance issues are vital to keep the FSAG informed about fatigue hazards in day-to-day FRMS operations. A series of fatigue reports on a particular route can be a trigger for further investigation by the FSAG. An effective fatigue reporting system requires an effective reporting culture. It needs to:
1. Use forms that are easy to access, complete, and submit;
Appendix 2
2. Have clearly understandable rules about confidentiality of reported information;
3. Have clearly understandable voluntary reporting protection limits;
4. Include regular analysis of the reports; and
5. Provide regular feedback to crewmembers about decisions or actions taken based on the reports and lessons learned.
(c) Crew Fatigue Surveys. Crew fatigue surveys fall into two basic types:
1. Retrospective surveys that ask crewmembers about their sleep and fatigue in the past. These can be relatively long and are usually completed only once, or at long time intervals (e.g., once a year); and
2. Prospective surveys that ask crewmembers about their sleep and fatigue right now. These are typically short and are often completed multiple times to monitor fatigue across a duty period, trip, or schedule. They usually include measures such as sleepiness, fatigue, and mood ratings.
(d) Performance Measurements. Performance measurements provide objective data that can be used to supplement the subjective data collected in fatigue reports and survey responses. Currently, there are three main approaches to monitoring crewmember performance:
1. Simple tests, developed in the laboratory, which measure aspects of an individual’s performance (e.g., reaction time, vigilance, and short-term memory). These performance measurements used to monitor the effectiveness of the FRMS may be the same as those utilized to obtain FAA approval of the FRMS;
2. Flight data analysis (FDA), which examines the relationship between identified elements of aircraft performance and pilot performance; and
3. Having trained flight deck observers rating the performance of crewmembers on the flight deck (e.g., Line-Oriented Safety Audit).
(e) Sleep Measurements. Sleep measurements provide objective data that verify that the sleep assumptions that were the foundation of the FRMS approval remain valid during the implementation of the FRMS. Sleep measurement technology used for ongoing hazard identification may be similar to those used for the FRMS approval.
(3) Reactive. The reactive process shall identify the contribution of fatigue hazards to reports and events associated with potential negative safety consequences in order to determine how the impact of fatigue could have been minimized. At a minimum, the process may be triggered by any of the following:
Fatigue reports, Confidential reports, Page 28
Appendix 2 Audit reports, Incidents, and Flight data analysis events.
d. Process 4.0: Assess Safety Risk. Conduct a safety risk assessment for each identified hazard. Decide which risks require mitigation. A certificate holder shall develop and implement risk assessment procedures that determine the probability and potential severity of fatigue-related events and identify when the associated risks require mitigation. The risk assessment procedures shall review identified hazards and link them to:
Operational processes, Their probability, Possible consequences, and The effectiveness and controls of existing safety barriers and controls.
e. Process 5.1: Select and Implement Controls and Mitigations (Risk Management (RM)). Implement appropriate mitigation strategies and communicate them to all relevant personnel. When it is decided that a particular fatigue hazard requires action, then controls and mitigations must be identified and implemented. The specific expertise of the FSAG should be used in the selection of these controls and mitigations.
(1) All involved personnel should clearly understand the hazard and the controls and mitigations designed to reduce the associated risk. The certificate holder shall develop and implement risk mitigation procedures that:
(a) Select the appropriate mitigation strategies,
(b) Implement the mitigation strategies, and
(c) Monitor the strategies’ implementation and effectiveness.
(2) If the controls and mitigations perform to an established safety standard, they become part of normal operations and are monitored by the FRMS SA processes. If, however, the controls and mitigations do not perform to an acceptable standard, then it will be necessary to reenter the FRM processes at the Process 2.0, Collect and Analyze Data. This will require gathering additional information and data; and/or re-evaluation of the fatigue hazard and the associated risks; and/or identification, implementation, and evaluation of new or revised controls and mitigations.
f. Process 5.2: Set Safety Performance Indicators. Set safety performance indicator(s) to be able to assess whether the mitigation strategies are delivering the required level of risk reduction. Data collected during the first 4 months of the FRMS operation will be compared to the respective model predictions and the performance safety standards from the FRMS validation (Process 2.0) to establish whether crewmember performance and fatigue are in the acceptable range. By the fourth month of the FRMS operation, the fatigue reporting rate (reports per flight segment) and average fatigue report risk level should be comparable to operations under the prevailing prescriptive rules. No “intolerable” fatigue reports should be received.
Appendix 2
g. Process 5.3: Develop Flightcrew FRMS Operations Procedures. Flightcrew FRMS operations procedures, are developed by the certificate holder and approved by the FAA to provide operational guidance for flightcrew members, dispatchers, schedulers, persons holding operational control, and individuals with oversight management of these areas for operations where an FAA-approved FRMS is applied. Therefore, flightcrew FRMS operations procedures serve as the foundation for disseminating safety-related guidance and procedural information for operations where an FRMS is proposed. FRMS operations procedures should be designed with supporting validated data collected and analyzed from various sources associated with the FRMS operation. Such data sources includes, but not limited to, actigraphy, performance tests, biomathematical modeling, and fatigue monitoring and reporting. Whenever FRMS is applied to any operation flightcrew members, dispatchers, schedulers, persons holding operational control, and individuals with oversight management of these areas for operations, all must comply with the FRMS operations procedures for that operation.
(1) As mitigations and controls are identified and validated to reduce fatigue risks associated with a specific FRMS operation, the certificate holder will be responsible for developing operating procedures that support those mitigations and controls. Specific limitations and conditions outlined in the certificate holder’s OpSpec A318, Authorization to Conduct Operations under an FRMS, must also be reflected in the FRMS operating procedures. The certificate holder should use FRM and SA to validate and update their FRMS operation procedures. This process adds an additional layer of safety with regard to demonstrating the certificate holder’s FRMS operation procedures support their AMOC.
(2) As part of the FRMS continuous monitoring and improvement process, the certificate holder will be responsible for evaluating the effectiveness of the FRMS and updating these procedures to support the data results. This data shall be submitted with the certificate holder’s reporting interval data. The certificate holder must make available to each flightcrew member, dispatcher, scheduler, person holding operational control, and individual with oversight management of these areas for operations a current copy of the flightcrew FRMS operations procedures. When an FRMS is being applied to any part of the certificate holder’s operation, each person involved with the FRMS shall comply with the flightcrew FRMS operations procedures.
h. Process 6.0: Set Up Processes for Monitoring the Effectiveness of Controls and Mitigations. If mitigation strategies perform to an acceptable standard, they become part of normal operations and are monitored by the FRMS SA processes. If mitigation strategies do not perform to an acceptable standard, then reenter the FRM processes at the appropriate step.
(1) There is a defined validation period for the first 4 months of the operation that involves more intensive monitoring. The FSAG will have regular oversight of all data and fatigue reports coming in and will act in a timely manner when issues arise. Copies of all data collected and associated monitoring data, along with actions taken by the certificate holder to mitigate fatigue events and occurrences, shall be submitted to the FAA during regular scheduled reporting intervals. A report will be compiled and routine processes will be defined for fatigue risk monitoring and management. This report will be available to all interested parties, including the FAA. If the performance indicators are acceptable, the operation will revert to routine monitoring.
Appendix 2
(2) If the controls and mitigations perform to an acceptable standard, they become part of normal operations and are monitored by the FRMS SA processes. If the controls and mitigations do not perform to an acceptable standard, then it will be necessary to reenter the FRM processes at the appropriate step. This could require:
Gathering additional information and data;
Reevaluation of the fatigue hazard and the associated risks; and/or Identification, implementation, and evaluation of new or revised controls and mitigations.
Appendix 2 SECTION 6. FATIGUE RISK MANAGEMENT SYSTEM SAFETY ASSURANCE
PROCESSES
1. INTRODUCTION TO FRMS SA PROCESSES. The FRMS processes is the part of the day-to-day FRMS operations focusing on identifying fatigue hazards, assessing safety risks, putting in place controls and mitigation strategies, and monitoring their effectiveness. This section works through the basic steps in FRMS SA processes, which form another layer in a certificate holder’s defenses against fatigue-related risk.
a. SA Processes. Figure 2-5, Fatigue Risk Management and Safety Assurance Processes, illustrates the combined elements of the FRM and FRMS SA processes. FRMS SA processes are also part of the routine operation of the FRMS, and they monitor how well the entire FRMS is functioning. The FRMS SA processes:
(1) Check that the FRMS is functioning as intended;
(2) Check that it is meeting the safety objectives defined in the FRMS policy;
(3) Check that it is meeting regulatory requirements;
(4) Identify where changes in the operating environment have the potential to increase fatigue risk; and
(5) Identify areas for improvement in the management of fatigue risk (continuous improvement of the FRMS).
b. Continuous Monitoring. The certificate holder shall develop and maintain FRMS SA processes to provide for continuous FRMS performance monitoring, analysis of trends, and measurement to validate the effectiveness of the fatigue safety risk controls. The sources of data may include, but are not limited to:
(1) Hazard reporting and investigations,
(2) Operational data related to fatigue,
(3) Audits and surveys, and
(4) Reviews and fatigue studies.
c. Management Process. The certificate holder shall provide a formal process for the management of changes which shall include, but is not limited to:
(1) Identification of changes in the operational environment that may affect the FRMS;
(2) Identification of changes within the organization that may affect the FRMS; and
(3) Consideration of available tools that could be used to maintain or improve FRMS performance prior to implementing changes.
Appendix 2
d. Continuous Improvement. The FRMS SA process must provide for the continuous improvement of the FRMS. This shall include, but is not limited to:
(1) The elimination and/or modification of risk controls that have had unintended consequences or that are no longer needed due to changes in the operational or organizational environment;
(2) Routine evaluations of facilities, equipment, documentation, and procedures;
(3) Trending of safety performance indicators to determine if there is a need to introduce new processes and procedures to mitigate emerging fatigue-related risks; and
(4) As an option, retrospective and prospective biomathematical modeling of schedules to assess potential schedule-related fatigue risk.
e. Safety Performance Indicators. The FRMS SA processes use a variety of data and information as safety performance indicators that can be measured and monitored over time.
Having a variety of safety performance indicators, plus a safety target for each, is expected to give better insight into the overall performance of the FRMS than having a single measure.
Safety performance targets must fall in the tolerable region defined in the risk assessment process and they may need to be revised as operational circumstances change. The information, data, and safety performance indicators from the FRMS processes provide a source of information for the FRMS SA processes. In addition, the FRMS SA processes:
(1) Use information and expertise from other sources, both from within the certificate holder’s organization and external to it, to evaluate the functioning of the FRMS;
(2) Evaluate trends in safety performance indicators to identify emerging or changed hazards and refer these back to the FRM processes;
(3) Identify changes in the operating environment that could affect fatigue risk and refer these back to the FRM processes; and
(4) Provide input for improving the operation of the FRMS.
2. FRMS SA FIVE-STEP PROCESS. The FRMS SA processes consist of a five-step process that focuses on identifying fatigue hazards, assessing safety risks, putting in place controls and mitigation strategies, and monitoring their effectiveness. The objectives of each individual step are outlined below.
a. Step 1: Collect and Review Information.
(1) Safety Performance Indicators. This process involves collecting and reviewing information gained through the FRMS processes to examine the overall performance of the FRMS. Performance of the FRMS should be examined through indentifying a variety of safety performance indicators. This should include information specific to the FRMS as well as safety performance indicators.
Appendix 2
(2) Examples. Examples of safety performance indicators specific to an FRMS will include measures obtained through the FRMS processes, such as:
The number of exceeded maximum duty days in operations covered by the FRMS, The number of voluntary fatigue reports per month, The average “fatigue call” rate by flightcrews on a specific pairing (trip), The ratio of fatigue reports from operations covered by the FRMS to fatigue reports from operations covered by the prescriptive flight and duty time regulations, Attendance at FRMS training sessions, Results on FRMS training assessments, The level of crewmember participation in fatigue-related data collection; and The number of times fatigue is identified as an organizational factor contributing to an event.
(3) Sources of Data. The sources of data for monitoring the safety performance of the FRMS may include (but are not limited to):
Hazard reporting and investigations,
Audits and surveys, and Reviews and fatigue studies.
(4) Hazard Reporting and Investigations. Trends in voluntary fatigue reports by crewmembers and others can provide valuable insights into the effectiveness of the FRMS.
Safety events in which crewmember fatigue has been identified as a contributing factor should be less common than fatigue reports. However, regular review of these events may also highlight areas where functioning of the FRMS could be improved. The value of both these sources of information depends on appropriate methods to identify the role of fatigue.
(5) Audits and Surveys. Audits and surveys can provide measures of the effectiveness of the FRMS without having to rely on fatigue levels being high enough to trigger fatigue reports or fatigue-related safety events (both of which are relatively rare events).
(a) Audits focus on the integrity of, and adherence to, the FRMS processes. These audits should answer questions such as:
Are all departments implementing the recommendations of the FSAG?
Are crewmembers using mitigation strategies as recommended by the FSAG?
Is the FSAG maintaining the required documentation of its activities?
(b) Audits can also periodically assess the effectiveness of the FRMS (e.g., by looking at the status of FRMS safety performance indicators and targets). Audits are external to the FSAG, but may still be internal to the certificate holder (i.e., conducted by other groups within the organization). In addition, feedback from regulatory audits can provide useful information for FRMS safety performance monitoring. Another type of audit that can be used in this context is a review by an independent scientific review panel that periodically reviews the Page 34
Appendix 2 activities of the FSAG and the scientific integrity of their decisions. A scientific review panel can also provide the FSAG with periodic updates on new scientific developments relevant to the FRMS.
(c) Surveys can provide information on the effectiveness of the FRMS. For example, they can document how schedules and assignments are affecting crewmembers, either by asking about their recent experiences (retrospective) or tracking them across time (prospective). Surveys for this purpose should include validated measures, such as standard rating scales for fatigue and sleepiness and standard measures of sleep timing and quality. Remember that a high response rate (ideally, more than 70%) is needed for survey results to be considered representative of the entire group, and response rates tend to decline when people are surveyed too frequently (participant fatigue).
(6) Reviews and Fatigue Studies. In general, safety reviews are used to ensure that safety performance is adequate during times of change (e.g., during the introduction of a new type of operation or a significant change to an existing operation covered by the FRMS).
(a) A review would start by identifying the change (e.g., moving a trip to a crew base in a different time zone, changes in on-board crew rest facilities, significant changes in the total trip, or a change of equipment being used for the trip). It would then evaluate the appropriateness and effectiveness of the FRMS activities relative to the change (for example, proposed methods for fatigue hazard identification, the risk assessment process, proposed controls and mitigations to address the fatigue hazard(s), and measures of their effectiveness to be used during the implementation of the change).
(b) Fatigue studies, as part of FRMS SA processes, are undertaken when a certificate holder is concerned about a broad fatigue-related issue for which it is appropriate to look at external sources of information. These could include the experience of other certificate holders, industry-wide or nation-wide studies, and scientific studies. External sources of information are particularly helpful when an internal consensus for a course of action cannot be reached or to supplement the limited experience and knowledge within that certificate holder’s organization. Fatigue studies in this context are mainly used for gathering information about large-scale issues related to the FRMS, rather than for identifying specific fatigue hazards.
(7) FRMS Safety Performance Indicators. Trends in FRMS safety performance indicators are also an important source of information about the effectiveness of the FRMS. This may include indicators identified by the FSAG as part of the FRMS processes. They may also include indicators that capture more global aspects of the safety performance of the FRMS.
b. Step 2: Evaluate FRMS Performance.
(1) This process is intended to validate the effectiveness of the fatigue controls and mitigations by analyzing the data collected and reviewed in Step 1 of the FRMS SA processes to determine whether:
(a) All specified FRMS safety performance targets are being met;
Appendix 2
(b) All specified FRMS safety performance indicators remain in the tolerable region defined in the risk assessment process;
(c) The FRMS is meeting the safety objectives defined in the FRMS policy; and
(d) The FRMS is meeting all regulatory requirements.
(2) The following are examples of safety performance targets and indicators that could be used in FRMS SA processes and that correspond with the safety performance indicators identified above:
(a) The length of the maximum duty days in operations covered by the FRMS does not exceed the limits defined in the FRMS policy. This is reviewed monthly by a computer algorithm and trends across time are evaluated every 3 months.
(b) By the fourth month after the introduction of a new operation, there must be a stable, low number of voluntary fatigue reports per month or a clear downward trend in the number per month (allowing time for crewmembers and other affected personnel to adjust to the new operation).
(3) The FSAG is responsible for providing a written report on the validation phase of the new operation, including analysis of all fatigue-related events and voluntary fatigue reports, and documentation of the corresponding adjustments made in fatigue controls and mitigations.
Additionally, the following information that should be included in the report:
(a) The rate of fatigue calls under the FRMS operation in comparison to the rate under non-FRMS operations.
(b) Actions taken when the rate of fatigue reports for operations conducted under the FRMS exceed the rate of fatigue reports for non-FRMS operations. The actions by the certificate holder taken must be reported to the FAA.
(c) Evidence that, in the last quarter, designated management has provided adequate resourcing for the FRMS, as specified in the FRMS policy.
(d) Evidence that, in the last quarter, the FSAG has met as often as is required in the FRMS policy and has maintained all the documentation of its activities required for internal and regulatory auditing.
(e) Evidence that all personnel responsible for schedule design and assignments have met annual FRMS training requirements as specified in the FRMS promotion processes.
(f) Measures of the effectiveness of FRM training and education programs.
(g) Evidence that quarterly levels of absenteeism are below the target specified for each operation covered by the FRMS.
Appendix 2
(4) When FRMS safety performance targets are not met or when safety performance indicators are not at an acceptable level, the controls and mitigations in use may need to be modified by reentering the FRMS processes at Step 2 or beyond. It may also be appropriate to seek additional information from outside the organization (e.g., by looking at fatigue studies). It may be necessary to undertake a review of compliance of crewmembers and other departments with the recommendations of the FSAG. It may also sometimes be necessary to review the functioning of the FSAG itself to find out why the FRMS is not working as intended.
c. Step 3: Identify Emerging Hazards.
(1) Analysis of trends in safety performance indicators may indicate the emergence of fatigue hazards that have not previously been recognized through the FRMS processes. For example, changes in one part of the organization may increase workload and fatigue risk in another part of the organization. Identifying emerging fatigue-related risks is an important function of FRMS safety performance processes, which take a broader system perspective than FRMS processes.
(2) Any newly identified fatigue risk, or combination of existing risks for which current controls are ineffective, should be referred back to the FSAG for evaluation and management using FRMS processes (risk assessment and design and implementation of effective controls and mitigations).
d. Step 4: Identify Changes Affecting the FRMS.
(1) In our dynamic aviation environment, changes are a normal part of flight operations.
They may be driven by external factors (e.g., new regulatory requirements, changing security requirements, or changes to air traffic control) or by internal factors (e.g., management changes or new routes, aircraft, equipment, or procedures). Changes can introduce new fatigue hazards into an operation, which need to be managed. Changes may also reduce the effectiveness of controls and mitigations that have been implemented to manage existing fatigue hazards.
(2) During Step 4, the objective of FRMS SA processes is to identify new hazards that may be a result of change. The current edition of ICAO Annex 6, Part I, Appendix 8 requires that a certificate holder has FRMS SA processes that provide a formal methodology for the management of change. These must include (but are not limited to):
(a) Identification of changes in the operational environment that may affect the FRMS;
(b) Identification of changes within the organization that may affect the FRMS; and
(c) Consideration of available tools that could be used to maintain or improve FRMS performance prior to implementing changes.
(3) A change management process is a documented strategy to proactively identify and manage the safety risks that can accompany significant change. When a change is planned, the following steps can be followed.
Appendix 2
(a) Use the FRM processes to identify fatigue hazards, assess the associated risk, and propose controls and mitigations; and
(b) Obtain appropriate management and/or regulatory signoff that the level of residual risk is acceptable.
(4) During the period of implementation of the change, use the FRMS SA processes to provide periodic feedback to line managers that the FRMS is functioning as intended in the new conditions. Documentation of the change management strategy in relation to fatigue management is also the responsibility of the FSAG. Changes in the operational environment may also necessitate changes in the FRMS itself. Examples include bringing new operations under the scope of the FRMS, collecting different types of data, and adjusting training programs. The FSAG should propose such changes and obtain approval for them from the FAA for implementation.
e. Step 5: Improve Effectiveness of the FRMS.
(1) Ongoing Evaluations. Ongoing evaluation by the FRMS SA processes not only enables the FRMS to be adapted to meet changing operational needs: it also allows the FRMS to continuously improve the management of fatigue risk. In doing so, risk controls that have unintended consequences or that are no longer needed due to changes in the operational or organizational environment can be identified and then modified or eliminated through the FRMS processes. Examples include:
(a) Routine evaluations of facilities, equipment, documentation, and procedures; and
(b) The determination of the need to introduce new processes and procedures to mitigate emerging fatigue-related risks.
(2) Documentation. It is important that changes made to the FRMS are documented by the FSAG so that they are available for internal and regulatory audits.
(3) Assigning Responsibility for FRMS SA Processes.
(a) To deliver effective oversight of the functioning of the FRMS, the FRMS SA processes need to operate in close communication with the FSAG, but with a degree of independence. The objective is to avoid the FSAG from reviewing its own performance.
(b) Primary responsibility for the FRMS SA processes is assigned to a quality assurance (QA) person or team (as appropriate) that is accountable to the executive management team. In smaller operations, responsibility for the FRMS SA processes might reside with an individual rather than a team. This individual may also have a variety of other QA responsibilities.