Par 7-8 Page 41
(a) These event problems have specific rules or procedures for resolution.
(b) The book procedure for problem solution resolves or reverses the abnormal condition.
(c) After diagnosis, this situation requires no crew decision. The crew selects and uses the appropriate rule (e.g., manual gear extension).
(d) The nature of this type of problem is unlikely to require a high level of CRM skill, unless the event set is already particularly demanding or time compressed, requiring multitask prioritization. In this case, the events are interrelated in the same timeframe.
(2) SOP Event Trigger Management.
(a) This type of event often entails continuous monitoring or system compensation.
(b) The defining characteristic is that the corrective procedure does not solve the problem.
(c) In the case requiring continuous monitoring tasks, the crew will need to prioritize tasks and reduce the effects of distraction (e.g., monitoring a constant speed drive (CSD) outlet temperature in the caution zone).
(d) Problem management should require more inherent CRM skills than SOP problem solution.
(e) On a continuum of difficulty, these types of problems lie between simple and complex problems.
(3) Knowledge-Based Solution/Management.
(a) A book procedure or solution is not available to the crew.
(b) Crews are required to brainstorm a solution or management strategy.
(c) Knowledge-based solutions and management strategies require a decision-making process that often engages multiple crewmembers.
(d) An example of a knowledge-based solution is the selection of an alternate airport when weather or other conditions prohibit landing at the planned destination. There may be no SOP for alternate selection.
c. Summary. Awareness of the types of problems raised by each event set within a LOS scenario will help to ensure that scenario objectives are met and selected CRM skills are used.
Par 7-8 Page 42 TABLE 7-2. SELECTED SCENARIO EVENT SET INDEX WITH PHASES OF FLIGHT
AND PROFICIENCY OBJECTIVES
SCENARIO EVENT
SET NUMBER
PHASES OF FLIGHT
TERMINAL PROFICIENCY
OBJECTIVES Scenario Event Set One Predeparture and Pushback Dispatch–Winter Preflight–with Malfunctions Start and Pre-taxi–Hung Start Scenario Event Set Two Taxi Taxi–Low Visibility Taxi–Winter Conditions Deicing Scenario Event Set Three Takeoff Takeoff–Winter Conditions Climb to Cruise Altitude–Winter Conditions Scenario Event Set Four Climb Climb to Cruise Altitude–Winter Conditions Scenario Event Set Five Cruise En Route Cruise–Winter Conditions, with Malfunctions–Severe Compressor Stall Scenario Event Set Six Descent Descent from Cruise–Winter Conditions, with Malfunction Engine Out Driftdown–Winter Conditions Scenario Event Set Seven Approach and Landing Engine-Out Instrument Landing System (ILS)–Winter Conditions Engine-Out Landing–Winter Conditions Taxi In–Winter Conditions Scenario Event Set Eight Taxi/Parking Parking–Winter Conditions Shutdown–with Auxiliary Power Unit (APU) Postshutdown NOTE: The general theme of the entire scenario is winter operations occurring across all event sets, and the scenario contains a simple problem (hung start in event set one) and a complex one (compressor failure crossing event sets five through eight).
7-9. THE EVENT SET MATRIX.
a. Event Set Matrices. An event set matrix will provide a quick reference source for specific items to be accomplished during the LOS, and will help to ensure that all proficiency objectives identified in the training program are accomplished. In addition, the matrix can be used to categorize the problems as simple to complex, in order to identify demands that will be placed on the crew.
b. Performance Indicators. CRM performance indicators for each event set must also be developed. This will require the integration and validation of the CRM skills to produce a similar matrix. The validation process is discussed in paragraph 7-11.
Par 7-10 Page 43
7-10. LEVEL OF DIFFICULTY. It is important to control the level of difficulty for all LOS scenarios, but it is imperative for the LOE. Students should not have to struggle because their evaluation is more difficult than another student’s evaluation. The most common methodology used in AQPs is to develop a difficulty scale for each task, subtask, and element (if necessary) on the job task list. Operators training using traditional means can develop a similar inventory of flight elements to rate. These values vary from aircraft type to aircraft type, so separate analyses must be conducted for each fleet type. One carrier uses a 5-point scale, where noncritical events are scored 1–3 and critical events 4–5. For example:
• 5 – Wind shear, ditching, evacuation, emergency descent, landing with flaps or slats jammed, etc.
• 4 – Bomb on board, incapacitation, manual Nonprecision Approach (NPA), rejected takeoff below 100 knots, etc.
• 3 – Manual precision approach, manual holding, instrument meteorological conditions (IMC) takeoff, runway change, operations out of limits, unreliable speed indication, etc.
• 2 – Visual meteorological conditions (VMC) takeoff, abnormal descent, two-engine manual landing, missing documentation during ground operations, etc.
• 1 – Normal descent, two-engine autolanding, FMS normal workload, normal after landing operations, etc.
NOTE: Most event set worksheets have a field for the event set trigger, the distractor(s), the supporting event(s), and the difficulty rating for each particular event set. Once the scenario is drafted all of the difficulty values for all of the events are added together. This number is then compared to what is considered an acceptable range of difficulty such as 35–45. If the scenario is out of range, events may be switched out to bring the overall scenario within range.
7-11. INTEGRATION AND VALIDATION OF CRM SKILLS TO EVENT SETS.
a. Observable Behavior. The matrix in Table 7-3, Selected Scenario Event Set Index With Phases of Flight and Crew Resource Management Behaviors, shows how to link observable CRM behaviors to each of the scenario event sets based upon the defined CRM objectives and crew tasks of the event set. A subset of these linked observable behaviors will be selected for validation based upon agreement between the experts.
b. Validation. Management and fleet subject matter experts (SME) should be used to validate the links between observable crew behaviors and scenario event sets. The participants can include instructors, check pilots/check Flight Engineers (FE), FAA inspectors, and managers from CRM departments. This relatively small group of participants should be familiar with the CRM process and flight training program.
c. Rating. Validation data should be collected using a rating system. The form should be a standalone form used by pilots who have flown at least one of the series of event sets and, therefore, have some experience with the event sets. The form should present sufficient background information to explain the scenario event set approach to assessment and should be limited to a set of observable behaviors that can be rated in about 1 hour. The ratings may use
Par 7-11 Page 44 a 5-point scale, where 1 signifies that there was a very low probability that the observable crew behavior was important in the assessment of tasks being performed, and 5 signifies that there was a very high probability that it was important.
d. The Validation Process. The validation process begins with the presentation of the selected scenario event sets. The selected event sets should be representative of the range of CRM assessment categories and should provide sufficient material to develop a number of scenarios. As modules, scenario event sets can be thought of as building blocks in scenario development, and the group of selected scenario event sets defines the range and boundaries. The validation results for the CRM assessment categories and observable crew behaviors are then presented, together with their links to the scenario event sets.
e. Assessment. These two sources of data, the ratings of the CRM categories and the ratings of the observable crew behaviors, can be used to generate complementary representations or profiles of a group of event sets, individual scenarios, or individual event sets. These ratings results also demonstrate that there can be agreement about the primary observable behaviors among those making CRM assessments. When scenario event sets are specified and listed with likely crew behaviors, experienced evaluators can show substantial agreement on the primary observable behaviors to properly assess the related tasks. Therefore, it is likely that making CRM assessments based on observable behaviors will produce reliable assessments.
f. Conclusion. The end result of the validation phase is an event set matrix that lists the events, technical requirements, and CRM behaviors for each event set (see Table 7-4, The Event Set Matrix). With the event sets defined, the proficiency objectives assigned, and the CRM objectives validated, the design team is now ready to develop the scripts and fly the scenarios.
Flying the scenarios is a critical step in the final determination that the training objectives are being met. From this final step, instructor training and the development of supporting documentation for the LOS can be developed.
Par 7-11 Page 45 TABLE 7-3. SELECTED SCENARIO EVENT SET INDEX WITH PHASES OF FLIGHT
AND CREW RESOURCE MANAGEMENT BEHAVIORS
SCENARIO
EVENT SET NUMBER
SITUATIONAL
AWARENESS
WORKLOAD
MANAGEMENT PLANNING DECISION MAKING Event Set One- Predeparture Pilot flying (PF) planned deice for winter operations standard operating procedure (SOP) PF briefed rising terrain PF analyzed departure weather radar (WX) and requests takeoff alternate Event Set Two-Taxi Crew discussed route and holdover times Taxi – Low Visibility Deicing pad Surface Movement Guidance and Control System (SMGCS) plan Pad coordination Event Set Three-Takeoff Crew discussed icing issue before it could become a problem Crew set clear priorities for tasks and their order
Event Set Four-Climb PF requested higher altitude
Event Set Five-Cruise PF directed pilot monitoring (PM) to deal with engine problem PM performed needed checklists and announced compliance Crew assessed one engine landing with WX at diversion field PF calculated time and distance to Mahlon Sweet Field Airport (EUG) PF stated that they cannot go back to Seattle-Tacoma International Airport (SEA) Event Set Six-Descent PF prioritized tasks and got ready for approach PF reviewed single engine approach procedures and aircraft evacuation
Event Set Seven- Approach and Landing PF properly prioritized PM provides backup for PF on all his tasks PF briefed cabin crew PF planed and briefed Safety Enhancement (SE) instrument landing system (ILS)
Event Set Eight-Taxi In
NOTE: The four CRM challenges are spread across the eight event sets, such that only two of the eight event sets presents more than two CRM challenges. One of the strengths of the event set methodology is that it allows the facilitator to focus on a limited set of CRM issues at a time.
Par 7-11 Page 46 TABLE 7-4. THE EVENT SET MATRIX
EVENT SET PHASE OF FLIGHT TECHNICAL REQUIREMENTS KEY EVENTS CRM BEHAVIORS
EVENT SET #1– Predeparture the crew must consider winter operations.
Predeparture Pushback Deicing procedures must be followed.
Takeoff alternate is required.
Departure, en route, and arrival in winter conditions.
Destination weather is at CAT IIIa minimums.
During engine start there is no N1 indication on Engine #1.
OR The #2 engine has a hung start, but starts on the second attempt.
OR When turning the engine anti-ice on, one valve fails to open.
COMMUNICATION:
Open, interactive crew climate established, crew asks questions and seeks answers on operational issues they are concerned about.
DECISION MAKING:
Captain asks and receives input, but makes decisive final decisions affecting mission. Crew continually assesses changing conditions to improve operations.
WORKLOAD MANAGEMENT:
Efficient workload distribution so no one is overtaxed.
EVENT SET #2–Taxi Taxi Takeoff from short runway in winter conditions with takeoff gross weight near runway limit.
Flaps 5/15 takeoff required Engine run-up required in takeoff position Cycle gear after takeoff Taxi via slippery and congested ramps and taxiways in low visibility.
The takeoff runway limited, low visibility and icing conditions near runway limit.
There is rapidly rising terrain to the south of the departure runway.
Complex departure in icing conditions.
COMMUNICATION:
Air traffic control (ATC) interaction, problem definition about deicing and rising terrain.
WORKLOAD MANAGEMENT:
Prioritize tasks for deicing and departure.
DECISION MAKING:
Captain decisive about rising terrain issues, with crew input.
Par 7-11 Page 47 TABLE 7-5. THE EVENT SET ASSESSMENT/GRADE SHEET Rate crew 1-5 Not Obs CRUISE Event Set # 5 Pilot Flying: ❒Left Seat ❒Right Seat
a. Proficient in use of FMS and AFDS. (4.1.2.2)(4.1.2.3)
b. All normal/non-normal procedures accomplished in accordance with standard operating procedure (SOP). (2.1.1)(2.1.2)(2.1.3)(2.1.4)
c. Evaluates options and determines a suitable airport for landing. (4.1.1.7)(4.1.3.6)(5.1.2)
d. Briefs F/As using TEST procedures in preparation for landing. (10.1.3.6)(BC 1.6)
e. Crew manages automated systems to increase SA and avoid work overload. (AT 6.6)
f. Roles, tasks, and responsibilities clearly assigned. Guidelines established. (LT 2.3)
g. Communications with cabin crew/ATC/company clear and timely. (SA 1.6)(SA 1.7)(LT 2.3)
h. Determine plan, and discuss aircraft configurations, airport-specific procedures, and performance. (SA 3.1)
i. Establish and brief “bottom lines” and “backup” plans. (DM4.3)
Pilot #1 Technical CRM Pilot #2 Technical CRM
COMMENTS: REQUIRED FOR ALL ITEMS GRADED 1, 2 OR NOT OBS.
Rate crew 1-5 Not Obs DESCENT Event Set # 6 Pilot Flying: ❒Left Seat ❒Right Seat
a. Proficient in use of FMS and AFDS. (5.1.1.2)(5.1.1.3)
b. Accomplishes descent/arrival, approach briefing, and normal checklists procedures in accordance with SOP. (5.1.1)
c. Ensures all non-normal checklists are completed prior to commencing approach. (10.1.3)
d. Ensures that F/As and passengers are prepared for landing and possible evacuation.
(10.1.3.6)
e. Crew verbalizes a plan, to include bottom lines and a backup plan. (DM 4.4)
f. Crew is assertive when voicing concerns, deviations from the original plan, and when nearing/reaching bottom lines. (LT 2.7)
g. Crew plans and briefs automation modes and configurations. (AT 6.1)
h. Informs appropriate personnel of emergency situation; keeps cabin crew and passengers informed and updated. (BC 1.6)
Pilot #1 Technical CRM Pilot #2 Technical CRM
COMMENTS: REQUIRED FOR ALL ITEMS GRADED 1, 2 OR NOT OBS.
NOTE: This example has two separate job task analyses: one for technical tasks and one for CRM tasks. Technical tasks are identified in parentheses with no letters in front of the numbers. CRM tasks are identified in parentheses with letters in front of the numbers. In this example, DM stands
Par 7-11 Page 48 for decision making, LT for leadership and teamwork, SA for situational awareness, BC for briefings and communications, and AT for automation and technology. Event set 5 has nine events, three drawn from the list of technical tasks, five from the list of CRM tasks, and one event (d) drawn from both technical and CRM tasks. Thus, event set 5 integrates technical and CRM tasks into a single graded event set.
7-12. SCRIPTING AND VALIDATING THE LOS SCENARIO.
a. Scripting. Experience has shown that the effectiveness of LOS relies on script detail and following the programmed script during the LOS session. To accomplish this, the LOS scenario should be carefully scripted, including ensuring that all ATC communications use correct terminology, timing, and routing. Attention to detail in communications scripting will enhance the realism of the LOS FSTD session. Below are some guidelines for developing effective scripts:
(1) Realism. Scripting should include events and conditions that will take place during the flight. The script should concentrate on realistic inputs that support the flight profile as well as variations determined during scenario development.
(2) Crew Autonomy. Scripting must allow for crew decisions other than the expected response. The crew must be given the flexibility to play out their hand to a logical conclusion.
These diversions can normally be identified as a result of a key event or within an event set
(e.g., scripting possible diversions and changes in routing). The script should include event set number, phase of flight, communications (including frequency and radio call), key events, and expected actions. With these categories, weather conditions and general environmental conditions should be presented as they would occur in an actual situation.
(3) Defined Criteria. If the script is being developed as an LOE, detailed success criteria must be established. Technical performance criteria are documented in applicable regulations or in company documentation, and only an overview is presented in the event set documentation. CRM performance criteria are presented for each event set and are divided by CRM behavior areas that have been integrated and validated by the design team. The desired behaviors are presented with a brief statement of what constitutes unsatisfactory behavior.
Examples of criteria for the sample LOS shown in Tables 7-2, 7-3, and 7-4 include:
(a) Technical Skills. The crew will be proficient in the knowledge and execution of all required takeoff data, analysis of terrain issues, winter operations, systems procedures, and performance limitations of the aircraft.
(b) Communication. The crew will accomplish a predeparture briefing, to include the entire crew (cabin and flightcrew). The briefing will establish the crew climate by emphasizing the importance of interactive decision making and participation of the entire crew.
The crew is encouraged to voice concerns they may have. Crewmembers will ask questions and seek information from each other about operational issues and decisions. Crewmembers will advocate issues until an acceptable solution is achieved. All problems should be recognized and decisions for their solutions made.
Par 7-12 Page 49
(c) Decision Making. The captain asks for and considers crew inputs, but the captain makes the final decision for the aircraft configuration as dictated by weather, performance, and fuel requirements. The crew continually assesses the changing conditions to improve the operation of the flight.
(d) Workload Management. The crew will distribute the workload to ensure that each crewmember is used while no one is overtaxed. The crew will use available resources to analyze the required tasks for this complex departure.
(e) Unsatisfactory Performance. Unsatisfactory performance of this event set includes a crew that is completely unaware of winter operations and the ramification on performance operating considerations. Also judged unsatisfactory is a crew that is not prepared for the complex departure, including the issue of the rapidly rising terrain. Other issues the evaluator observes during this event set may be result in a judgment of unsatisfactory performance.
b. Evaluations. Using these success criteria for the LOE, the evaluation is based on the outcome of the event set much like the current evaluation of the outcome of a maneuver. Within the event set, specific objectives are assigned, any one of which could be involved in the unsuccessful outcome of the event set.
c. Validation. A systematic approach to validating scenarios in terms of their training objectives should be adopted. Formal and informal review panels, analysis of data on scenario attributes, and feedback from instructors, check pilots/check FEs, flightcrew members, and FAA inspectors provide the information needed to validate or modify the scenario.
d. Test Flights. After the LOS is represented by the script, the LOS should be flown by at least two different crews, and if possible, the POI or designated representative. If possible, crews flying the LOS should be recorded for viewing. These tapes will serve as a useful tool for training the instructors/evaluators prior to implementation. When the crews have finished the LOS, they should complete the same validations forms used to integrate the CRM. Quite often, the crews flying the event sets will have a different rating for the CRM behaviors as compared to the experts. From this final validation, the LOS will be modified prior to instructor training and implementation.
e. Scenario Validation. There should also be a scenario validation accomplished by a facilitator not conducting the LOS. The scenario will be evaluated for its value in meeting the training objectives and for determining the level of facilitator skill required to administer the LOS. The evaluation of the LOS will also provide an opportunity to note any errors that may exist in the facilitator’s guide or in the flight documentation.
f. FAA Approval. The FAA must approve LOE scenarios prior to their use. However, if LOE event sets are designed to allow for their recombination in a mix-and-match configuration to create different LOEs, then event sets may be individually approved by the FAA for that purpose. When preapproved event sets are employed in new combinations, the resulting new LOE scenarios do not require additional FAA approval.
7-13. SCENARIO FLEXIBILITY.
a. Real-Time Flexibility. Students should not know exactly what to expect when they enter a scenario-based training or evaluation session. Therefore, it is desirable to provide the scenario with some flexibility, allowing the facilitator to switch events around, often in real time.
One option is to switch out airports. For example, an operator may use scenarios that involve departing from one of three airports to a hub, and then departing to, again, one of three airports.
Other than the hub, the flightcrew members do not know which airport they will be departing or arriving, as the facilitator can change the airport in real time.
b. Branching Options. The most common approach to building flexibility into scenarios involves branching options. The flightcrew members may know that there will be one of three electrical emergencies in cruise phase, but they do not know in advance which electrical emergency will occur. When using this approach, it is important that the difficulty rating of each emergency is similar. The event set matrix in Table 7-4 shows three alternate possible key events in event set 1. This is an example of the branching strategy built into an event set.
c. Interchangeable Events. One of the most creative uses of scenario flexibility requires careful planning, but it provides maximum flexibility and minimal cognitive load on the facilitator. This approach involves designing two separate scenarios in such a way that the aircraft configuration at the beginning and end of each matching event set is identical. So, for example, the aircraft configuration at the beginning and end of event set two is identical across both scenarios. This allows the event sets within those two scenarios to be switched out at any point. So if there are two scenarios, A and B, one crew receives AAAAAA, the next BBBBBB, the next ABABAB, the next AABBAA, and so on. The facilitators must master only the contents of two full scenarios, yet from these elements they can generate a large number of somewhat difference scenarios.
7-14. GRADING EVENT SETS/GRADING PHASES OF FLIGHT. As originally conceived by the Airlines for America (A4A) LOS Design Focus Group, event sets were to be independent of the phases of flight. A hung start or a compressor stall would be an event set, and would be graded as such. However, operators have found that approach awkward and have switched to assigning grades to each phase of flight instead. Using the scenario in Table 7-2, the original idea was that the severe compressor stall that occurs in phase five and carries through phase eight would be a single event set. Instead, each phase of flight is now graded according to how the event set activities within each phase of flight are handled, even if the event set carries across multiple phases of flight. So although the event set is the linchpin of the design and development of the LOS, flightcrew member grades are assigned according to phase of flight. What this means is that instead of the crew receiving a single grade for the compressor stall response across four phases of flight, each of the last four phases of flight receives a grade, and that grade is based on how the crew manages the compressor stall within that phase of flight. What that means is that while scenarios are developed from event sets, they may be graded either as event sets or as phases of flight.