Par 9 Page 13 framework, data from a LOSA can be cross-referenced with data from the ASAP and/or FOQA programs, if the airline operates one or both of those voluntary programs. Each data source provides unique yet complementary evidence of the airline’s safety status. In the above example, the airline might track unstable approaches through its FOQA program using new flight parameters decided by the action committee and then implemented into procedures and training.
To see if pilots are incurring problems with the new procedure, the FOQA aircraft data can be cross-referenced with ASAP reports of events resulting from unstable approaches. This way, the airline does not have to wait until the next LOSA to learn if its interventions are being successful.
c. LOSA data are useful in another way. LOSA presents a broad view of operations; a repeat LOSA can maintain that broad focus. For example, did the changes that were introduced after the first LOSA improve results in one area, only to cause problems in another? Checklist adherence may have improved, but did error detection—the superordinate goal of improving checklist adherence—actually improve or is the new adherence simply cosmetic?
10. SUMMARY: THE 10 OPERATING CHARACTERISTICS OF LOSA.
a. Ten operating characteristics define and summarize the LOSA process. It is recommended that a LOSA observe all 10 characteristics to ensure the integrity of the LOSA process and the quality of the final product.
b. The 10 Operating Characteristics Are:
(1) Jumpseat Observations During Normal Flight Operations. LOSA observations are limited to regularly scheduled flights. Line checks, initial line indoctrination or other training flights are off-limits due to the extra level of stress put on pilots during this type of situation.
Having another observer onboard adds to an already high stress level, thus providing an unrealistic picture of performance. In order for the data to be representative of normal operations, LOSA observations should be collected on regular and routine flights.
(2) Joint Management/Pilots’ Association Sponsorship. In order for LOSA to succeed as a viable safety project, there needs to be support not only from the management side but also from the pilots. The joint sponsorship provides a “check and balance” for the project to ensure that change, as necessary, will be made as a result of LOSA data. When considering whether to conduct a LOSA, the first question to be asked by airline management is whether the pilots’ association (or pilot group representatives) endorses the project. If the answer is “No,” the project should not be initiated until endorsement is obtained.
(3) Voluntary Crew Participation. Maintaining the integrity of LOSA within an airline and the industry as a whole is extremely important for long-term success. To accomplish this goal, all LOSA observations are collected with voluntary crew participation. Before conducting LOSA observations, observers should first ask the flightcrew for permission to be observed. If the crew declines, the observer takes another flight with no questions asked. If an airline conducting a LOSA has an unreasonably high number of declines, this should serve as an indicator that there are critical “trust” issues to be resolved.
Page 14 Par 10
(4) Deidentified, Confidential, and Nondisciplinary Data Collection. LOSA observers are required to not record names, flight numbers, dates, or any other information that can identify a crew or individual. The purpose of LOSA is to collect safety data, not to punish pilots. Airlines cannot allow themselves to squander a unique opportunity to gain insight into their operations by having pilots fearful that a LOSA observation could be used against them for disciplinary reasons. If a LOSA observation is ever used for disciplinary reasons, the credibility of the entire safety program may be irreparably compromised.
(5) Targeted Observation Form. The LOSA observation form in Appendix 3 is predicated on the TEM framework described in Appendix 1. However, other conceptual frameworks can be used for LOSA data collection. Whatever framework is used, it should generate meaningful data on a variety of topics, including what the crews did well, what they did poorly, and how they managed each phase of flight. A narrative written by the observer should have sufficient detail to allow others to understand the flight and all its events. The observers need to describe the environmental conditions and events surrounding the pilots’ behavior so that the crews’ performance can be understood in full context.
(6) Trained and Calibrated Observers. Primarily, pilots conduct LOSA. Observation teams will typically include line pilots, instructor pilots, safety pilots, management pilots, and representatives of the pilots’ safety committee. It is critical to select observers that are respected and trusted within the airline to ensure line acceptance of LOSA. After observers are selected, they are trained and calibrated in the LOSA methodology, including the use of the LOSA observation form. Observers’ training in the concepts and methodology of LOSA will ensure that the observations will be conducted in the most standardized manner (see Appendix 2).
(7) Trusted Data Repository. In order to maintain confidentiality, airlines should have a trusted data repository. This site can be in-house, such as that used for other confidential data, or it can be offsite. The goal is that no individual observations will be misplaced or improperly disseminated through the airline.
(8) Data Verification. Data-driven programs like LOSA require quality data management procedures and consistency checks. For LOSA, these checks are done at dataverification roundtables. A roundtable consists of three or four department and pilots’ association representatives who review all the raw data for possible inaccuracies. The end product is a database that is validated for consistency and accuracy according to the airline’s standards and manuals before any statistical analysis is performed.
(9) Targets for Enhancement. The final product of a LOSA is the data-derived targets for enhancement based on emergent patterns in the data. It is then up to the airline to develop an action plan based on these targets, using experts from within the airline to analyze the targets and implement appropriate change strategies.
Par 10 Page 15 (and 16)
(10) Feedback Results to the Line Pilots. In order to ensure long-term success of LOSA, airlines should communicate the results back to the line pilots. Pilots will want to see not only the results of the audit, but also management’s plan for improvement.
ORIGINAL SIGNED BY CAROL E.GILES (for)
James J. Ballough Director, Flight Standards Service
Appendix 1 APPENDIX 1. THREAT AND ERROR MANAGEMENT
1. Introduction.
The Threat and Error Management (TEM) model is a conceptual framework for understanding operational performance in complex environments. Originally created to capture the flightcrew’s task in commercial aviation, the model is generic and can be applied to numerous work situations. The added value that TEM brings to other performance models is that it focuses simultaneously on the operating environment and the humans working in that environment.
Because the model captures ongoing performance in its “natural” or normal operating context, the resulting description is realistic, dynamic, and holistic. Because the model can also quantify the specifics of the environment and the effectiveness of performance in that environment, it is also highly diagnostic1.
There are several ways of using the TEM model, from focusing on a single event (as is the case with accident/incident analysis) to understanding systemic patterns in a large set of events (as with Line Operations Safety Audits (LOSA)). As a training tool, TEM can help individuals clarify their performance needs and vulnerabilities; and as part of a safety management system, TEM can help an organization measure and improve the effectiveness of its organizational defenses and safeguards.
2. The Model.
This section defines and provides examples of the various components of the TEM model.
a. Threats.
A threat is defined as an event or error that occurs outside the influence of the flightcrew
(i.e., it was not caused by the crew), increases the operational complexity of a flight, and requires crew attention and management if safety margins are to be maintained.
There are threats from the environment—adverse weather, airport conditions, terrain, traffic, and air traffic control (ATC)—and threats emanating from within the airline—aircraft malfunctions and master equipment list (MEL) items, problems, interruptions, or errors from dispatch, cabin, ground, maintenance, and the ramp. The crew, for example, may anticipate threats by briefing a thunderstorm in advance; or they may be unexpected, occurring suddenly and without warning such as in flight aircraft malfunctions. Some threats are easily resolved and quickly dismissed from the crew’s workload, while other threats require greater attention and management. A mismanaged threat is defined as a threat that is linked to or induces flightcrew error.
b. Errors.
1 The rest of this appendix will talk about TEM exclusively in the aircraft piloting environment so as to provide specific examples and situations. However, it is important to remember that the TEM framework can be applied to other high-risk jobs. To do so requires task and situational analyses by subject matter experts to develop the relevant threat and error taxonomies.
Appendix 1
Crew error is defined as action or inaction that leads to a deviation from crew or organizational intentions or expectations. Errors in the operational context tend to reduce the margin of safety and increase the probability of adverse events.
Broadly speaking, there are handling errors (flight controls, automation), procedural errors (checklists, briefings, callouts) and communication errors (with ATC, ground, or pilot-to-pilot).
See the error management worksheet in the sample observation form, Appendix 3, for a more complete list of errors.
Understanding how the error was managed is as important, if not more important, than understanding the prevalence of different types of error. It is of interest then if and when the error was detected and by whom, as well as the response(s) upon detecting the error, and the outcome of the error. As with threats, some errors are quickly detected and resolved, leading to an inconsequential outcome, while others go undetected or are mismanaged. A mismanaged error is defined as an error that is linked to or induces additional error or an undesired aircraft state.
c. Undesired Aircraft States.
An UAS is defined as a position, condition, or attitude of an aircraft that clearly reduces safety margins and is a result of actions by the flightcrew. It is a safety-compromising state that results from ineffective error management. Examples include unstable approaches, lateral deviations, firm landings, and proceeding towards wrong taxiway/runway (more examples are noted on the observation form in Appendix 3). Events such as equipment malfunctions or ATC command errors can also place the aircraft in a compromised position, but these would be considered threats.
As with errors, UASs can be managed effectively, returning the aircraft to safe flight, or the crew action or inaction can induce an additional error, incident, or accident.
d. Threat and Error Countermeasures.
A description of a flight is not complete without noting what the crew was doing to anticipate threats and avoid errors, as well as managing those that occurred. The following crew behaviors are considered threat and error countermeasures:
• Planning countermeasures—planning, preparation, briefings, contingency management— are essential for managing anticipated and unexpected threats
• Execution countermeasures—monitor/cross-check, taxiway/runway management, workload, and automation management—are essential for error detection and error response
• Review/modify countermeasures—evaluation of plans, inquiry—are essential for managing the changing conditions of a flight
Appendix 1 In addition to crew behaviors, TEM countermeasures also include equipment and procedural countermeasures. Warning systems such as ground proximity warning systems (GPWS) and weather alerts can be considered threat countermeasures, just as checklists and well-written procedures provide the means for error avoidance and error detection.
In sum, the TEM model captures the dynamic activity that is a flightcrew planning and executing a flight in real time and under real conditions. The use of the model is that it can be applied proactively or reactively, at the individual, organizational, and/or systemic levels.
3. Practical Applications of the TEM Model.
a. TEM as a Training Tool.
TEM has been found to be an effective tool for human factors training programs at a number of airlines throughout the world2. TEM training emphasizes the value of threat anticipation and management, error avoidance, and error detection and recovery. The model allows pilots to analyze their own performance strengths and vulnerabilities.
TEM concepts can be trained effectively in the classroom in the absence of LOSA.
However, TEM training can be enhanced if an airline has also conducted a LOSA. The LOSA results can help shape the training curriculum, and pilots can discuss the findings during training.
Pilots are always interested in and respond well to data derived from their own operation.
It is important to clarify that TEM is not crew resource management (CRM) and should not be considered a replacement for it. TEM and CRM refer to overlapping but not equivalent activities. CRM refers specifically to activities conducted by the crew to optimize performance.
These activities include threat and error countermeasures such as briefing, contingency planning, and monitor/cross-checking, but they also include higher-order concepts such as leadership and establishing open communication in the cockpit. Similarly, TEM includes crew countermeasures, but it also encompasses equipment, procedural, and regulatory countermeasures.
As a training tool, TEM can help individuals clarify their performance needs and vulnerabilities from a different perspective. Hence, threat and error management concepts could be introduced and explored as one component of CRM training.
b. TEM as a Reporting Tool for Incidents.
Reporting forms structured to the TEM framework instruct the pilots to describe the event at the level of threats and errors. The TEM format prompts pilots to report information about the threats that were present, the errors they may have made, how well the event was managed, and how the event may have been avoided or handled better. Preliminary work has shown that even pilots who have not had training in the TEM model are able to complete the reporting form, a fact that speaks to the intuitive nature of the TEM framework.
2 The International Civil Aviation Organization (ICAO) has adopted the TEM model in its Human Factors Training Manual (ICAO Document 9683, 2002)
Appendix 1
In the ASAP environment, TEM can “go inside the pilot’s head” in a way that LOSA as an objective observational tool cannot do and most assuredly does not want to do. With ASAP, pilots can report personal or historical factors that contributed to the event – information that is not privy to an observer. With LOSA, the benefit is that observers may detect threats and errors that the crews themselves do not detect. This is one example of how LOSA and ASAP data can complement each other at the system level.
c. TEM as a Systematic Observation Tool.
The TEM model was first conceived in conjunction with the development of LOSA; hence, its original application was as an observation tool. As of the publication of this advisory circular (AC), feasibility studies are currently underway to explore the transfer of the methodology to airline flight dispatch and air traffic control (ATC). In 2002, one major airline began exploring an adapted version of LOSA called Dispatch Operations Safety Audit (DOSA). Early results demonstrated that such a transfer of methodology is possible and could ultimately provide a 360-degree perspective on the interaction between pilots and dispatchers. In addition, ICAO has instituted a formal group of ATC subject matter experts from across the world to develop the Normal Operations Safety Survey (NOSS), a formal protocol to observe normal operations in ATC, based on the TEM model and LOSA methodology.
d. TEM as a Reactive Analysis Tool for Accidents and Incidents.
TEM can be used as an analysis tool to understand rare events, such as accidents and serious incidents. For example, the International Air Transport Association (IATA) Safety Committee has adopted the TEM model as an analysis framework for its incident review meetings, based on its ease of use and utility of the extracted data.
e. TEM as a Proactive Analysis Tool.
When TEM is used as the framework for safety data collection, a wealth of information can be extracted. An airline can use the data to understand patterns at the organizational level. The data can also be collected across the industry and analyzed for systemic trends.
An analysis based on TEM can:
• Quantify those aspects of the working environment that can pose a problem for the efficiency or safety of the operation (threat prevalence)
• Quantify the management of those threats as either effective or ineffective (threat management)
Appendix 1 Page 5 (and 6)
• Recognize high rates of threat prevalence and mismanagement as systemic vulnerabilities
• Codify and quantify the errors that crews commit (error prevalence)
• Codify and quantify the error management process from diagnosis to response and outcome (error management)
• Recognize high rates of error prevalence and error mismanagement as systemic flaws in procedures, policies, training, aircraft design, and or interagency coordination, and
• Locate strengths as well as vulnerabilities in organizational safeguards
4. Conclusion.
The TEM model is intuitive, practical, and versatile. More and more airlines are realizing the use of TEM, as exemplified in the following quote from an airline manager:
“Since our LOSA, we have introduced a 1-day TEM course for our pilots. The overwhelming response has been positive, and there’s a recognition among pilots that in TEM, the academic community is using language we understand and feel.
Additionally we are using TEM as a debriefing tool for training events. Once again this makes sense to the pilots, and the common reaction is one of dawning;
penny-dropping; eureka!”
Appendix 2 APPENDIX 2. TRAINING AND STANDARDIZING LOSA OBSERVERS
This appendix details the training and standardizing of Line Operations Safety Audits (LOSA) observers. Some of this material appears throughout the advisory circular (AC); this appendix draws that information together and provides more in-depth information.
1. Observer Training.
Observer training typically requires 2 to 3 days of classroom training with a followup session after one or two line observations. There are five topics that need to be covered in observer training:
• LOSA rationale and etiquette
• Company policies and procedures
• Observation form
• Threat and Error Management (TEM) concepts
• Narratives
a. LOSA Rationale and Etiquette.
Observers will likely have a rudimentary understanding of LOSA when selected for the project; however, they will need to fully understand the safety rationale for conducting a LOSA at their airline. A “big picture” perspective will help observers understand the “why” of LOSA and will underscore the importance of their role in the LOSA process. Also, the observers will be ambassadors for LOSA while observing on the line, and it is important that they be able to explain the process fully, to answer any questions that the line pilots may have, and allay any fears or concerns.
Specifically, the observers need to understand the safety rationale for normal operations monitoring—a discussion of proactive vs. reactive safety strategies is recommended. The observers also need to know how the data collected from the LOSA will be used to understand strengths and weaknesses in the operations. An overview of the whole process from observations to data cleaning and analysis, to the diagnostic report and the development of targets for enhancement is recommended.
Of course, the observers will also need to know the “how” of LOSA, specifically the etiquette associated with being a LOSA observer. An observer needs to learn how to approach a crew, how to ask permission to observe a flight, how to walk away so that the crew can discuss it, and to accept without question any crew’s decision to deny access to the cockpit. The observer should also carry a copy of the letter of endorsement jointly signed by management and pilots’ association to show any interested crew.
On the jumpseat, the observer’s behavior is best summarized as “a fly on the wall.” The observer needs to be unobtrusive, yet responsive to any queries the crew may have. The observer does not complete the observation form on the jumpseat—this would be a disconcerting distraction to the crew. Instead, the observer is encouraged to carry a small pocket notebook or
Appendix 2 legal pad to jot down minimalist notes, just enough to jolt the memory when outside the cockpit and completing the full observation form.
LOSA observers should be trained to accept their role as observers, not evaluators—they are not check airmen. LOSA observers will observe errors and undesired aircraft states as part of their observations; however, they should only interrupt if they perceive the safety of the flight to be seriously and immediately endangered. A helpful rule of thumb is to ask observers to think of themselves as a guest riding in the jumpseat of another airline. This seems to help distance the observer from the flightcrew while still being able to politely point out safety concerns if they come about.
The LOSA observer is the only person in the whole LOSA process who has access to crew identities. It is essential, therefore, that observers are reminded throughout the training of their responsibilities in this regard. Confidentiality is paramount and observed crew behavior should not be discussed with anyone—not even other observers. Experience has shown that at the end of a flight, crews will often ask the observer to “debrief” their performance. In these circumstances, it is essential that the observer politely decline the invitation. This emphasizes the concept that the observer is not there to evaluate the crew, merely to record events.
LOSA observers should act in an unobtrusive and consistent manner so that line pilots have a similarly positive experience of LOSA, which in turn will favorably affect their receptivity to the final results and outcomes.
b. Company Policies and Procedures.
Observers need to be current with company policies and procedures so as to observe procedural adherence on the line and detect any deviations. Selecting active line pilots from the airline is one way to ensure this. Spending some time in the classroom reviewing procedures across fleets allows all observers to get “up to speed” on the fleets they will be observing. All observers should also be encouraged to review the manuals as homework.
c. Observation Form.
This subsection refers to the mechanics of correctly completing and submitting a LOSA observation form. During the training, the observers work to develop the needed competencies as defined by the observation form. Hence, observers should see the observation form as soon as possible after the training begins so that they have a clear sense of what is expected of them.
With the observation form in hand, the observers can be led through the various sections, and then practice using case studies (see next subsection).
If the observation form is software-based, time should be spent ensuring that all observers have the necessary computer skills to open the form, enter and edit data, and submit the observation. Mastering these skills in the classroom will avoid potential loss of data later in the LOSA due to computer error. Observers should have the name of a contact person in case of computer problems.