Appendix 2
d. TEM Concepts.
If the airline is using TEM as the basis of its LOSA, observers need to receive training in the framework. Specifically, observers need to be able to define, distinguish, and identify threats, errors, and undesired aircraft states. This is best achieved with a mixture of lecture, case studies, and review. Lecture material should include multiple examples of each type of threat, error, and undesired aircraft state, and case studies can take the form of scripted vignettes and/or actual accident and incident report excerpts. The distinction between threats, errors, and undesired aircraft states becomes clear with practice and observers are usually able to correctly distinguish examples of all three categories in 2 days or fewer of classroom involvement. Some training examples are provided in Appendix 5.
e. Narratives.
Observers need training in writing the flight narrative. If they understand the concepts underlying the observation form as well as the diagnostic rationale for conducting a LOSA, the observers will realize that a good-quality narrative is imperative. The observation form should contain several prompts to help the observer provide sufficient detail and observers should be encouraged to “overwrite” the flight rather than provide too little detail. In particular, observers need to record events that happened, such as threats, errors, or undesired aircraft states, the context in which they happened, and the crew’s response and management of the event.
Observers are selected because they are experts at understanding flight operations and this expertise is best expressed in detailed narratives. As long as the observer provides a detailed narrative of the flight, any coding oversights can be remedied later in the data-cleaning process.
f. Training Objectives.
In sum, at the end of LOSA observer training, an observer should be able to:
(1) Knowledgeably and confidently explain the rationale and process for conducting a LOSA at the airline;
(2) Enact the LOSA observer etiquette in a professional and consistent manner;
(3) Demonstrate knowledge of company policies and procedures;
(4) Use the observation form accurately and comprehensively;
(5) Understand the theoretical framework of the observation form. If the tool is based on TEM concepts, the observer should be able to define, distinguish, and identify threats, errors, and undesired aircraft states; and
(6) Write detailed and comprehensive flight narratives from which others will be able to understand the full context of the flight and related events.
Appendix 2 Members of the LOSA steering committee may want to attend LOSA observer training at another airline or attend an industry-sponsored LOSA conference for further guidance before designing the training.
2. Observer Standardization.
Standardization refers to the need to be sure that flight details are recorded in a systematic and consistent fashion. In LOSA, standardization is a multi-step process that involves standardizing the observers and conducting followup data-cleaning and coding of completed observations.
The first step in any observer standardization is good-quality training. To be sure observers understand the concepts, group discussions are encouraged. Focusing on the finer points of the model and the observation form, these discussions help calibrate the observers to a common standard. A test can be administered at the end of the training to be sure that all observers have grasped the necessary knowledge and can demonstrate the required competencies as specified by the training objectives. If observers complete this test satisfactorily, they can be released to the line to complete one or two “trial” observations.
The LOSA project coordinator or trainer should schedule time with each observer to discuss their trial observations. If the observer does a good job, as evidenced particularly by the quality of the narrative, the observations can be retained and used in the LOSA, and the observer can be sent back out to complete their observations. If the quality of the narrative is poor (e.g., lacking sufficient detail), the trainer can work with the observer to help draw out missing information. If the observer has forgotten details and cannot recreate the flight, the observation should be discarded, and the observer sent out to complete another trial observation. It is the LOSA coordinator’s decision to drop any observer from the observation team if that person fails to meet the required standard. For this reason, it can be a good idea to initially recruit and train more observers than needed, to allow for attrition, illness, and scheduling conflicts.
a. Data Standardization. There are several supplementary techniques that ensure goodquality standardized data are used in a LOSA.
First, observers are not asked to evaluate performance, but simply to observe it. From a data standpoint, this is the distinction between subjective judgment and objective observation. For example, observers are asked to note threats without any subjective judgment—if there is a thunderstorm, record it; if there is an aircraft malfunction or ground maintenance problem, record it. It is the same for errors and undesired aircraft states. Observers also note the crew’s response to the threats, errors, and undesired aircraft states and the outcome. The observer is not assessing crew performance or providing a subjective evaluation—the observer is telling the story of the flight.
Appendix 2 Page 5 (and 6) Second, the data analyst always checks observation forms submitted by observers against the narrative. Observers are expert at describing a flight—they are not necessarily expert at assigning codes to the various threats, errors, etc., especially if it is their first time as a LOSA observer. While training is given in the codes, nonetheless it is realistic to accept that observers won’t necessarily retain this information perfectly. The narrative is the “fail-safe” in the system in that it allows the analyst to read the events of the flight and match them to the observer’s codes. Because the analyst is the expert with the codes, he or she can add any codes that were missed and recode anything that might not be quite correct. Hence, a good-quality narrative is the ultimate key to standardized data. Observers provide a comprehensive narrative, and the analyst ensures consistent and accurate coding.
A third step in standardizing the LOSA data prior to analysis involves verifying the data with a team of local experts—airline personnel familiar with the operation of each fleet (possibly fleet managers or representation of the airline’s operations, member of the steering committee, but not any of the observers). The group’s task is to review and verify the observations against current manuals, policies, and procedures. For example, an observer might log a procedural error for failure to make an approach callout when, in fact, there is no written procedure in the airline’s flight operations manual. The “error” would then be deleted from the database. The data verification group acts as a check on the analyst’s coding, ensuring events are correctly recorded in line with each fleet’s procedures and policies. It also builds ownership in the results and dispels any later criticism that the coding was not accurate.
In sum, there are several methods that ensure that LOSA data are consistently and accurately recorded:
• Observers are trained, calibrated, tested, and recalibrated
• Objective observation, not subjective evaluation, is the basis and outcome of the observation
• The narrative is stressed as key to high-quality data
• The data analyst applies consistent coding to the observations
• The data verification group checks the analyst’s coding against specific procedures
Following the above steps will ensure reliability and validity of the data analyzed from LOSA observations.
Appendix 3 APPENDIX 3. SAMPLE LOSA OBSERVATION FORM LOSA Observation Form Observer Information Observer ID (Employee number) 3059 Observation Number #1
(e.g.. "1 of 2" indicates segment one for a crew that you observed across two segments) Crew Observation Number 1 of Flight Demographics City Pairs (e.g., PIT-CLT) PIT - LAX A/C Type (e.g., 737-300) B-757 Pilot flying (Check one) CA FO Time from Pushback to Gate Arrival (Hours: Minutes) 4:55 Local Arrival Time (Use 24 hour time) 09:55 Late Departure? (Yes or No) Yes Predeparture / Taxi Narrative Your narrative should provide a context. What did the crew do well? What did the crew do poorly? How did the crew manage threats, crew errors, and significant events? Also, be sure to justify your behavioral ratings.
The CA established a great team climate - positive with open communication. However, he seemed to be in a rush and not very detail oriented. The FO, who was relatively new to the A/C, tried to keep up but fell behind at times. The CA did not help the cause by interrupting the FO with casual conversation (marginal workload management).
All checklists were rushed and poorly executed. The CA was also lax verifying paperwork. This sub-par behavior contributed to an undetected error - the FO failed to set his airspeed bugs for T/O (poor monitor/cross-check).
The Before Takeoff Checklist should have caught the error, but the crew unintentionally skipped over that item The FO noticed the error upon commencing the takeoff roll and said, "Missed that one."
The Captain's brief was interactive but not very thorough (marginal SOP briefing). He failed to note the closure of the final 2000' of their departing runway (28R) due to construction. Taxiways B7 and B8 at the end of the runway were also out. The crew was marked "poor" in contingency management because there were no plans in place on how to deal with this threat in the case of a rejected takeoff. Lucky it was a long runway.
Appendix 3 APPENDIX 3. SAMPLE LOSA OBSERVATION FORM (Continued) Takeoff / Climb Narrative Your narrative should provide a context. What did the crew do well? What did the crew do poorly? How did the crew manage threats, crew errors, and significant events? Also, be sure to justify your behavioral ratings.
Takeoff was normal. ATC granted a right turn VFR climb which was commenced at 600 ft. Climb to flight level 20000 with step climbs to 35000 ft. Eventually leveled at 31000 ft about 90 miles North. When established at FL200, ATC cleared the crew to FL270. They accepted and the First Officer dialed 230 instead of 270 in the MCP. The Captain caught the error on cross-verification.
Cruise Narrative Your narrative should provide a context. What did the crew do well? What did the crew do poorly? How did the crew perform during the handover?
Crew stayed attentive to aircraft position throughout cruise.
Appendix 3 APPENDIX 3. SAMPLE LOSA OBSERVATION FORM (Continued) Descent / Approach / Land / Taxi Narrative Your narrative should provide a context. What did the crew do well? What did the crew do poorly? How did the crew perform during the handover?
Briefing to TOD - The approach brief much better than their takeoff brief. They expected runway 25L from the Civet Arrival for a straight-in visual approach. Jepp charts were out, contingencies talked about, and everything was by the book (outstanding SOP brief and plans stated).
10000'to slowing and configuring - ATC cleared the crew to 25L, but at 8000', ATC changed us to the Mitts Arrival for runway 24R due to a slow moving A/C on 25L. The CA changed the arrival and approach in the FMC, tuned the radio, and quickly briefed 24R. As soon as everything was clean, A TC called back and told the crew they could either land on 25L or 24R at their discretion. Since time was a factor, the crew discussed and decided to stick with the approach into 24R. The crew was flexible and the CA did a nice job assigning workload.
FO flew the plane while the CA checked everything over one more time (outstanding evaluation of plans). The crew was also better monitors and cross checkers. However, their execution of checklists was still a little sloppy - late and rushed (marginal monitor and cross check) Bottom lines to Flare / Touchdown - The approach was stable, but the FO let the airplane slip left, which resulted in landing left of centerline. Since the FO was new to this aircraft (1 month flying time), the observer chalked it up to a lack of stick and rudder proficiency.
Taxi-in - The crew did a great job navigating taxiways and crossing the active 24L runway. Charts were out and both heads looking for traffic. (outstanding taxiway / runway management). However, there were no wing walkers meeting the aircraft in a congested ramp area.. A common problem in LAX.
Overall Flight Narrative This narrative should include your overall impressions of the crew.
Overall, the crew did a marginal job with planning and review/modify plans during predeparture. However, during the descent/approach/land phase, it was excellent. Their execution behaviors were marginal to good for the entire flight.
While the takeoff brief was marginal, the CA made an outstanding approach brief. Open communication was not a problem. Good flow of information when the flight's complexity increased with the late runway change. They really stepped it up During predeparture, the CA introduced an unnecessary element of being rushed, which compromised workload management. However, his decisiveness and coordination in the descent /approach/land phase kept his leadership from being marked "marginal."
The big knock against this crew involved checklists, cross verifications, and all monitoring in general. They were a little too complacent during low workload periods (e.g., No altitude verifications during climb). The CA set a poor example in this regard. When the workload increased, the crew did a good job.
Appendix 3 APPENDIX 3. SAMPLE LOSA OBSERVATION FORM (Continued) Threat Management Worksheet Threat Description Threat Management Phase of Flight Linked to Describe the threat Threat Type 1 Predepart / Taxi 2 Takeoff/Climb flight crew error? How did the crew manage or mismanage the threat?
3 Cruise 4 Des/App/Land (Yes / No) 5 Taxi- in T1 Runway and taxiway construction on their departing runway (final 2000) 103 1 No and closures in his brief. No plans were made in the event of a rejected takeoff, which is required by airline SOP. Threat mismanaged - CA failed to include the construction T2 Late A TC runway change - changed moving aircraft on 25L runway to 24R from 25L due to a slow 101 4 Yes radios, and placed emphasis on the FO to fly the aircraft Threat managed - CA reprogrammed the FMC, handled the After a late runway change, ATC called Threat managed - CA asked for the FO's preference. They T3 back and told the crew that it was at 101 4 Yes mutually decided to continue the approach into 24R their discretion to land on 24R or 25L because it was already in the FMC T4 ramp area in LAX On taxi-in, there were no wing walkers meeting the aircraft in a congested 204 5 Yes walkers were dispatched to the airplane Threat managed - The crew called ground ops and wing T5 T6 Threat Codes Environmental Threats Airline Threats 100 Adverse Weather 103 Airport Conditions 200 Airline Operational Pressure 204 Ground / Ramp 101 ATC 104 Heavy traffic (air or ground) 201Cabin 205 Dispalch / Paperwork 102 Terrain 199 Other Environmental Threats 202 Aircraft Malfunctions / MEL Items 206 Manuals / Charts 200 Ground Maintenance 299 Other Airline Threats Page 4
Appendix 3 APPENDIX 3. SAMPLE LOSA OBSERVATION FORM (Continued) Error Management Worksheet Error Description Error Response / Outcome Error Management Phase of Linked flight to Crew Error Error Describe the crew error 1 Predepart / Taxi threat? Error Response Outcome How did the crew manage or 2 Takeoff/Climb 3 Cruise 4 Des/App/Land (If Yes Type 1 Detected 2 No response 2 Undesired state 3 Additional error 1 Inconsequential mismanage the error?
5 Taxi-in Threat ID) CA failed to brief a rejected E1 takeoff for shortened departing 1 T1 403 2 1 No error management.
runway due to construction.
E2 FO failed to set his airspeed bugs. 1 304 2 3 Linked to error #3 In running the Before Takeoff ЕЗ Checklist, the FO skipped the 1 401 2 2 Linked to UAS #1 takeoff data item.
At FL200, the crew was cleared to E4 FL270. They accepted and the FO dialed 230 instead of 270 in the 2 302 1 Error managed - Captain caught the error on cross-verification.
Mode Control Panel.
FO, hand flying, let the airplane slip E5 a little to the left during the final 4 300 2 2 Linked to UAS #2 approach.
Error Type Codes Aircraft Handling Procedural Communication 300 Manual Flying 400 SOP Cross-verification 500 Crew to External Communication 301 Flight Control 401 Checklist 501 Crew to Crew Communication 302 Automation 402 Callout 599 Other Communication 309 Ground Handling 403 Briefing 304 Systems / Instruments / Radios 404 Documentation 399 Other Aircraft Handling 499 Other Procedural Page 5
Appendix 3 APPENDIX 3. SAMPLE LOSA OBSERVATION FORM (Continued) Undesired Aircraft State (UAS) Management Worksheet UAS Description UAS Response / Outcome UAS Management Linking Error? Undesired aircraft state description Code UAS Crew UAS Response UAS Outcome 1 Inconsequential How did the crew manage or mismanage the undesired aircraft state?
(Enter the Error ID) 2 No response 2 Additional error Errors mismanaged - The bug error should have UAS E2 Wrong airspeed bugs on takeoff roll 1 1 1 been caught with the Before Takeoff Checklist, but the FO missed the item. The FO detected and corrected the error on the roll.
UAS FO landed left of the centerline. 86 1 1 Error mismanaged - FO tried to correct but still landed left of the centerline. Approach was 2 stable and made the first high-speed taxiway UAS Undesired Aircraft State Type Codes Configuration States Aircraft Handling States - All Phases Approach / Landing States 1 Incorrect A/C configuration - flight controls, brakes, thrust reversers, landing SURVET CREDE EIDI 30 Crew induced deviation above G/8 or FM3 path gear- 41 Lateral deviation 81 Crew induced deviation below G/S or FM9 path 2 Incorrect A/D configuration - systems (fuel, electrical, hydraulics, pneumatics, air-conditioning pressurization, instrumentation) 2 Unnecessary WX penetra Bo 82 Unstable approach 3 Incorred A/C configuration - automation 3 Unauthorized airspace penetration 83 Continued landng - unstable approach 4 Incorrect AVC configuration - engines 84 Firm landing 14 Speed too high 85 Ficated landing Ground States 36 Landing off CL 20 Proceeding towards wrong runway 37 Long landing outside TDZ 21 Runway incursion 46 Abrupt aircraft control (attitude) 38 Landing short of TDZ 22 Proceeding towards wrong taxiway / ramp 47 Excessive banking 23 Tariway / ramp incursion 48 Operation outside aircraf limitations 99 Other Undesired States 24 Wrong gate 25 Wrong hold spot 26 Abrupt aircraft control - taxi Page 6
Appendix 3 APPENDIX 3. SAMPLE LOSA OBSERVATION FORM (Continued) Crew Performance Marker Worksheet 2 4 Poor Marginal Good Outstanding Observed performance had safety implications Observed performance was adequate but Observed performance was effective Observed performance was truly noteworthy.
needs improvement Phase of Flight Ratings Descent / Approach / Planning Performance Markers Predeparture / Taxi Takeoff / Climb Land / Taxi SOP BRIEFING The required briefing was interactive and operationally thorough 2 PLANS STATED Operational plans and decisions were communicated and acknowledged 3 4 CONTINGENCY MANAGEMENT Crew members developed effective strategies to manage threats to safety 1 Execution Performance Markers MONITOR / CROSS-CHECK Crew members actively monitored and cross-checked systems and other 1 1 2 WORKLOAD MANAGEMENT Operational tasks were prioritized and properly managed to handle primary flight duties 2 3 3 VIGILANCE Crew members remained alert of the environ aircraft osition of the 3 3 AUTOMATION MANAGEMENT Automation was properly managed to balan requirements al and/or workload 3 AXIWAY / RUNWAY MANAGEMENT Crew members used caution and kept watch outside wher taxinays anu rumnays navigating 3 4 Review / Modify Performance Markers EVALUATION OF PLANS Existing plans were reviewed and modified when necessary 4 INQUIRY Crew members not afraid to ask questions to investigate and/or clanty current plans of action 3 3 Overall Performance Markers Ratings COMMUNICATION ENVIRONMENT Environment for open communication was e LEADERSHIP Captain showed leadership and coordinated c activities Page 7 (and 8)