(c) In the case requiring continuous monitoring tasks, the crew will need to prioritize tasks and reduce the effects of distraction (e.g., monitoring a CSD outlet temperature in the caution-zone).
(d) Problem management should require more inherent CRM skills than SOP problem solution.
(e) On a continuum of difficulty, these types of problems lie between simple and complex problems.
(3) Knowledge-Based Solution/Management.
(a) A book procedure or solution is not available to the crew.
(b) Crews are required to brainstorm a solution or management strategy.
(c) Knowledge-based solutions and management strategies require a decision making process that often engages multiple crewmembers.
(d) An example of a knowledge-based solution is the selection of an alternate airport when weather or other conditions prohibit landing at the planned destination. There may be no SOP for alternate selection.
c. Awareness of the types of problems raised by each event set within a LOS scenario will help to ensure that scenario objectives are met and selected CRM skills are used.
Page 34 Par 607
TABLE 6-2 Selected Scenario Event Set Index With Phases of Flight and Proficiency Objectives
SCENARIO EVENT
SET NUMBER
PHASES OF FLIGHT TERMINAL PROFICIENCY
OBJECTIVES Scenario Event Set One Pre Departure, and Push Back, Dispatch - Winter Preflight - with Malfunctions Start and Pre-Taxi - Hung Start Scenario Event Set Two Taxi Taxi – Low Vis Taxi – Winter conditions De-icing Scenario Event Set Three Takeoff Takeoff - Winter Conditions Climb to Cruise Altitude - Winter Conditions
Scenario Event Set Four Climb Climb to Cruise Altitude - Winter Conditions
Scenario Event Set Five Cruise En route Cruise - Winter Conditions, with Malfunctions – Severe Compressor Stall
Scenario Event Set Six
Descent Descent from Cruise - Winter Conditions, with Malfunction Engine Out Driftdown - Winter Conditions Scenario Event Set Seven Approach and Landing Engine Out ILS - Winter Conditions Engine Out Landing - Winter Conditions Taxi In - Winter Conditions
Scenario Event Set Eight
Taxi/Parking Parking - Winter Conditions Shutdown - with APU Post Shutdown
608. THE EVENT SET MATRIX.
a. An event set matrix will provide a quick reference source for specific items to be accomplished during the LOS, and will help to ensure that all proficiency objectives identified in the training program are accomplished. In addition, the matrix can be used to categorize the problems as simple to complex in order to identify demands that will be placed on the crew.
Par 607 Page 35
b. CRM performance indicators for each event set must also be developed. This will require the integration and validation of the CRM skills to produce a similar matrix. The validation process is discussed in the next section.
609. INTEGRATION AND VALIDATION OF CRM SKILLS TO EVENT SETS.
a. The matrix in Table 6-3 shows how to link observable CRM behaviors to each of the scenario event sets based upon the defined CRM objectives and crew tasks of the event set. A subset of these linked observable behaviors will be selected for validation based upon agreement between the experts.
b. Management and fleet subject matter experts should be used to validate the links between observable crew behaviors and scenario event sets. The participants can include instructors, check airmen, FAA inspectors, and managers from CRM departments. This relatively small group of participants should be familiar with the CRM process and flight training program.
c. Validation data should be collected using a rating system. The form should be a standalone form used by pilots who have flown at least one of the series of event sets and, therefore, have some experience with the event sets. The form should present sufficient background information to explain the scenario event set approach to assessment and should be limited to a set of observable behaviors that can be rated in about 1 hour. The ratings may use a five-point scale where “1” signifies that there was a very low probability that the observable crew behavior was important in the assessment of tasks being performed, and the number “5” signifies that there was a very high probability that it was important.
d. The validation process begins with the presentation of the selected scenario event sets.
The selected event sets should be representative of the range of CRM assessment categories and should provide sufficient material to develop a number of scenarios. As modules, scenario event sets can be thought of as building blocks in scenario development, and the group of selected scenario event sets defines the range and boundaries. The validation results for the CRM assessment categories and observable crew behaviors are then presented, together with their links to the scenario event sets.
e. These two sources of data, the ratings of the CRM categories, and the ratings of the observable crew behaviors, can be used to generate complementary representations or profiles of a group of event sets, individual scenarios, or individual event sets. These ratings results also demonstrate that there can be agreement about the primary observable behaviors among those making CRM assessments. When scenario event sets are specified and listed with likely crew behaviors, experienced evaluators can show substantial agreement on the primary observable behaviors to properly assess the related tasks. Therefore, it is likely that making CRM assessments based on observable behaviors will produce reliable assessments.
Page 36 Par 608
f. The end result of the validation phase is an event set matrix that lists the events, technical requirements, and CRM behaviors for each event set (see Table 6-4). With the event sets defined, the proficiency objectives assigned, and the CRM objectives validated, the design team is now ready to develop the scripts and fly the scenarios. Flying the scenarios is a critical step in the final determination that the training objectives are being met. From this final step, instructor training and the development of supporting documentation for the LOS can be developed.
Par 609 Page 37
TABLE 6-3 Selected Scenario Event Set Index With Phases of Flight and CRM Behaviors
SCENARIO
EVENT SET NUMBER
SITUATIONAL
AWARENESS
WORKLOAD
MANAGEMENT
PLANNING DECISION MAKING Event Set One - Pre Departure PF planned de-ice for winter operations SOP PF briefed rising terrain PF analyzed departure WX and requests takeoff alternate Event Set Two - Taxi Crew discussed route and hold over times Taxi – Low Vis De-icing pad SMGS plan Pad coordination Event Set Three - Takeoff Crew discussed icing issue before it could become a problem
Crew set clear priorities for tasks and their order
Event Set Four - Climb PF requested higher altitude
Event Set Five - Cruise PF directed PM to deal with engine problem PM performed needed checklists and announced compliance Crew assessed one engine landing with WX at diversion field PF calculated time and distance to EUG
PF stated that they cannot go back to SEA
Event Set Six
- Descent PF prioritized tasks and got ready for approach PF reviewed single engine approach procedures and A/C evacuation
Event Set Seven - Approach and Landing PF properly prioritized PM provides backup for PF on all his tasks PF briefed cabin crew PF planed and briefed SE ILS
Event Set Eight – Taxi In
Page 38 Par 609
TABLE 6-4 The Event Set Matrix
EVENT SET PHASE OF FLIGHT TECHNICAL
REQUIREMENTS
KEY EVENTS CRM BEHAVIORS EVENT SET #1 - Pre-departure The crew must consider winter operations.
Pre Departure Push Back
Deicing procedures must be followed.
Takeoff alternate is required.
Departure, enroute and arrival in winter conditions.
Destination WX is at CAT IIIa minimums.
During preflight crew may have a During engine start there is no N1 indication on Engine #1.
OR The #2 engine has a hung start, but starts on the second attempt or when turning the engine anti-ice on, one valve fails to open.
.
COMMUNICATION:
Open, interactive crew climate established, crew asks questions and seeks answe on operational issues they are concerned about.
DECISION MAKING:
Captain asks and receives input, but makes decisive final decisions affecting mission. Crew continually assesses changing conditions to improve operations.
WORKLOAD MANAGEMENT:
Efficient workload distribution so no one is over taxed.
EVENT SET #2 - Taxi Taxi Takeoff from short runway in winter conditions with takeoff gross weight near runway limit.
Flaps 5/15 takeoff required Engine run up required in takeoff position Engine run up required in takeoff position Cycle gear after takeoff Taxi via slippery and congested ramps and taxiways in low visibility The takeoff runway limited, low visibility and icing conditions near runway limit.
There is rapidly rising terrain to the south of the departure runway.
Complex departure in icing conditions.
COMMUNICATION:
ATC interaction, problem definition about deicing and rising terrain.
WORKLOAD MANAGEMENT:
Prioritize tasks for de-icing and departure.
DECISION MAKING:
Captain decisive about rising terrain issues, with crew input.
Par 609 Page 39
TABLE 6-5 The Event Set Assessment/Grade Sheet
Rate crew 1-5 Not Obs CRUISE Event Set # 5 Pilot Flying: ❒ Left Seat ❒ Right Seat
a. Proficient in use of FMS and AFDS. (4.1.2.2)(4.1.2.3)
b. All normal/non-normal procedures accomplished in accordance with SOP
(2.1.1)(2.1.2)(2.1.3)(2.1.4)
c. Evaluates options and determines a suitable airport for landing. (4.1.1.7)(4.1.3.6)(5.1.2)
d. Briefs flight attendants using TEST procedures in preparation for landing (10.1.3.6)(BC 1.6)
e. Crew manages automated systems to increase SA and avoid work overload (AT 6.6)
f. Roles, tasks, and responsibilities clearly assigned. Guidelines established (LT 2.3)
g. Communications with cabin crew/ATC/company clear and timely (SA 1.6)(SA 1.7)(LT 2.3)
h. Determine plan, and discuss aircraft configurations, airport specific procedures and performance. (SA 3.1)
i. Establish and brief “bottom lines” and “back up” plans (DM4.3)
Pilot #1 Technical CRM Pilot #2 Technical CRM
COMMENTS: Required for all items graded 1, 2 or not Obs.
Rate crew 1-5 Not Obs DESCENT Event Set # 6 Pilot Flying: ❒ Left Seat ❒ Right Seat
a. Proficient in use of FMS and AFDS. (5.1.1.2)(5.1.1.3)
b. Accomplishes descent/arrival, approach briefing, normal checklists procedures IAW SOP.
(5.1.1)
c. Ensures all non-normal checklists are completed prior to commencing approach. (10.1.3)
d. Ensures that flight attendants and passengers are prepared for landing and possible evacuation (10.1.3.6)
e. Crew verbalizes a plan, to include bottom lines and a backup plan. (DM 4.4)
f. Crew is assertive when voicing concerns, deviations from the original plan and when nearing/reaching bottom lines (LT 2.7)
g. Crew plans and briefs automation modes and configurations (AT 6.1)
h. Inform appropriate personnel of emergency situation; keeps cabin crew and passengers informed and updated (BC 1.6)
Pilot #1 Technical CRM Pilot #2 Technical CRM
COMMENTS: Required for all items graded 1, 2 or not Obs.
Page 40 Par 609
610. SCRIPTING AND VALIDATING THE LOS SCENARIO.
a. Experience has shown that the effectiveness of LOS relies on script detail and following the programmed script during the LOS session. To accomplish this, the LOS scenario should be carefully scripted, including ensuring that all ATC communications use correct terminology, timing and routing. Attention to detail in communications scripting will enhance the realism of the LOS simulator session. Below are some guidelines for developing effective scripts:
(1) Scripting should include events and conditions that will take place during the flight.
The script should concentrate on realistic inputs that support the flight profile as well as variations determined during scenario development
(2) Scripting must allow for crew decisions other than the “expected” response. The crew must be given the flexibility to “play out their hand” to a logical conclusion. These diversions can normally be identified as a result of a key event or within an event set (e.g., scripting possible diversions and changes in routing). The script should include: Event set number, phase of flight, communications (including frequency and radio call), key events, and expected actions. With these categories, weather conditions and general environmental conditions should be presented as they would occur in an actual situation.
(3) If the script is being developed as an LOE, detailed success criteria must be established. Technical performance criteria are documented in applicable regulations or in company documentation, and only an overview is presented in the event set documentation.
CRM performance criteria are presented for each event set and are divided by CRM behavior areas that have been integrated and validated by the design team. The desired behaviors are presented with a brief statement of what constitutes unsatisfactory behavior. Examples of criteria for the sample LOS shown in tables 6-2, 6-3, and 6-4 include:
(a) Technical Skills. The crew will be proficient in the knowledge and execution of all required takeoff data, analysis of terrain issues, winter operations, systems procedures, and performance limitations of the aircraft.
(b) Communication. The crew will accomplish a pre-departure briefing to include the entire crew (cabin and flightcrew). The briefing will establish the crew climate by emphasizing the importance of interactive decision-making and participation of the entire crew.
The crew is encouraged to voice concerns they may have. Crewmembers will ask questions and seek information from each other about operational issues and decisions. Crewmembers will advocate issues until an acceptable solution is achieved. All problems should be recognized and decisions for their solutions made.
(c) Decisionmaking. The captain asks for and considers crew inputs, but the captain makes the final decision for the aircraft configuration as dictated by weather, performance and fuel requirements. The crew continually assesses the changing conditions to improve the operation of the flight.
Par 610 Page 41
(d) Workload Management. The crew will distribute the workload to ensure that each member is used while no one is over-taxed. The crew will use available resources to analyze the required tasks for this complex departure.
(e) Unsatisfactory Performance. Unsatisfactory performance of this event set includes a crew that is completely unaware of winter operations and the ramification on performance operating considerations. Also judged unsatisfactory is a crew that is not prepared for the complex departure, including the issue of the rapidly rising terrain. Other issues the evaluator observes during this event set may be result in a judgment of unsatisfactory performance.
b. Using these success criteria for the LOE, the evaluation is based on the outcome of the event set much like the current evaluation of the outcome of a maneuver. Within the event set, specific objectives are assigned, any one of which could be involved in the unsuccessful outcome of the event set.
c. A systematic approach to validating scenarios in terms of their training objectives should be adopted. Formal and informal review panels, analysis of data on scenario attributes, and feedback from Instructors, Check Airmen, line pilots, and FAA inspectors provide the information needed to validate or modify the scenario.
d. After the LOS is represented by the script, the LOS should be flown by at least two different crews and if possible the POI or designated representative. If possible, crews flying the LOS should be taped for viewing. These tapes will serve as a useful tool for training the instructors/evaluators prior to implementation. When the crews have finished the LOS, they should complete the same validations forms used to integrate the CRM. Quite often, the crews flying the event sets will have a different rating for the CRM behaviors as compared to the experts. From this final validation, the LOS will be modified prior to instructor training and implementation.
e. There should also be a scenario validation accomplished by an instructor not conducting the LOS. The scenario will be evaluated for its value in meeting the training objectives and for determining the level of facilitator skill required to administer the LOS.
The evaluation of the LOS will also provide an opportunity to note any errors that may exist in the facilitator’s guide or in the flight documentation.
f. The FAA must approve LOE scenarios prior to their use. However, if LOE event sets are designed to allow for their recombination in a “mix and match” configuration to create different LOEs, then event sets may be individually approved by the FAA for that purpose.
When pre-approved event sets are employed in new combinations, the resulting new LOE scenarios do not require additional FAA approval.
Page 42 Par 610