f. The end result of the validation phase is an event set matrix that lists the events, technical requirements, and CRM behaviors for each event set (see Table 6-4). With the event sets defined, the proficiency objectives assigned, and the CRM objectives validated, the design team is now ready to develop the scripts and fly the scenarios. Flying the scenarios is a critical step in the final determination that the training objectives are being met. From this final step, instructor training and the development of supporting documentation for the LOS can be developed.
Par 609 Page 37
TABLE 6-3 Selected Scenario Event Set Index With Phases of Flight and CRM Behaviors
SCENARIO
EVENT SET NUMBER
SITUATIONAL
AWARENESS
WORKLOAD
MANAGEMENT
PLANNING DECISION MAKING Event Set One - Pre Departure PF planned de-ice for winter operations SOP PF briefed rising terrain PF analyzed departure WX and requests takeoff alternate Event Set Two - Taxi Crew discussed route and hold over times Taxi – Low Vis De-icing pad SMGS plan Pad coordination Event Set Three - Takeoff Crew discussed icing issue before it could become a problem
Crew set clear priorities for tasks and their order
Event Set Four - Climb PF requested higher altitude
Event Set Five - Cruise PF directed PM to deal with engine problem PM performed needed checklists and announced compliance Crew assessed one engine landing with WX at diversion field PF calculated time and distance to EUG
PF stated that they cannot go back to SEA
Event Set Six
- Descent PF prioritized tasks and got ready for approach PF reviewed single engine approach procedures and A/C evacuation
Event Set Seven - Approach and Landing PF properly prioritized PM provides backup for PF on all his tasks PF briefed cabin crew PF planed and briefed SE ILS
Event Set Eight – Taxi In
Page 38 Par 609
TABLE 6-4 The Event Set Matrix
EVENT SET PHASE OF FLIGHT TECHNICAL
REQUIREMENTS
KEY EVENTS CRM BEHAVIORS EVENT SET #1 - Pre-departure The crew must consider winter operations.
Pre Departure Push Back
Deicing procedures must be followed.
Takeoff alternate is required.
Departure, enroute and arrival in winter conditions.
Destination WX is at CAT IIIa minimums.
During preflight crew may have a During engine start there is no N1 indication on Engine #1.
OR The #2 engine has a hung start, but starts on the second attempt or when turning the engine anti-ice on, one valve fails to open.
.
COMMUNICATION:
Open, interactive crew climate established, crew asks questions and seeks answe on operational issues they are concerned about.
DECISION MAKING:
Captain asks and receives input, but makes decisive final decisions affecting mission. Crew continually assesses changing conditions to improve operations.
WORKLOAD MANAGEMENT:
Efficient workload distribution so no one is over taxed.
EVENT SET #2 - Taxi Taxi Takeoff from short runway in winter conditions with takeoff gross weight near runway limit.
Flaps 5/15 takeoff required Engine run up required in takeoff position Engine run up required in takeoff position Cycle gear after takeoff Taxi via slippery and congested ramps and taxiways in low visibility The takeoff runway limited, low visibility and icing conditions near runway limit.
There is rapidly rising terrain to the south of the departure runway.
Complex departure in icing conditions.
COMMUNICATION:
ATC interaction, problem definition about deicing and rising terrain.
WORKLOAD MANAGEMENT:
Prioritize tasks for de-icing and departure.
DECISION MAKING:
Captain decisive about rising terrain issues, with crew input.
Par 609 Page 39
TABLE 6-5 The Event Set Assessment/Grade Sheet
Rate crew 1-5 Not Obs CRUISE Event Set # 5 Pilot Flying: ❒ Left Seat ❒ Right Seat
a. Proficient in use of FMS and AFDS. (4.1.2.2)(4.1.2.3)
b. All normal/non-normal procedures accomplished in accordance with SOP
(2.1.1)(2.1.2)(2.1.3)(2.1.4)
c. Evaluates options and determines a suitable airport for landing. (4.1.1.7)(4.1.3.6)(5.1.2)
d. Briefs flight attendants using TEST procedures in preparation for landing (10.1.3.6)(BC 1.6)
e. Crew manages automated systems to increase SA and avoid work overload (AT 6.6)
f. Roles, tasks, and responsibilities clearly assigned. Guidelines established (LT 2.3)
g. Communications with cabin crew/ATC/company clear and timely (SA 1.6)(SA 1.7)(LT 2.3)
h. Determine plan, and discuss aircraft configurations, airport specific procedures and performance. (SA 3.1)
i. Establish and brief “bottom lines” and “back up” plans (DM4.3)
Pilot #1 Technical CRM Pilot #2 Technical CRM
COMMENTS: Required for all items graded 1, 2 or not Obs.
Rate crew 1-5 Not Obs DESCENT Event Set # 6 Pilot Flying: ❒ Left Seat ❒ Right Seat
a. Proficient in use of FMS and AFDS. (5.1.1.2)(5.1.1.3)
b. Accomplishes descent/arrival, approach briefing, normal checklists procedures IAW SOP.
(5.1.1)
c. Ensures all non-normal checklists are completed prior to commencing approach. (10.1.3)
d. Ensures that flight attendants and passengers are prepared for landing and possible evacuation (10.1.3.6)
e. Crew verbalizes a plan, to include bottom lines and a backup plan. (DM 4.4)
f. Crew is assertive when voicing concerns, deviations from the original plan and when nearing/reaching bottom lines (LT 2.7)
g. Crew plans and briefs automation modes and configurations (AT 6.1)
h. Inform appropriate personnel of emergency situation; keeps cabin crew and passengers informed and updated (BC 1.6)
Pilot #1 Technical CRM Pilot #2 Technical CRM
COMMENTS: Required for all items graded 1, 2 or not Obs.
Page 40 Par 609
610. SCRIPTING AND VALIDATING THE LOS SCENARIO.
a. Experience has shown that the effectiveness of LOS relies on script detail and following the programmed script during the LOS session. To accomplish this, the LOS scenario should be carefully scripted, including ensuring that all ATC communications use correct terminology, timing and routing. Attention to detail in communications scripting will enhance the realism of the LOS simulator session. Below are some guidelines for developing effective scripts:
(1) Scripting should include events and conditions that will take place during the flight.
The script should concentrate on realistic inputs that support the flight profile as well as variations determined during scenario development
(2) Scripting must allow for crew decisions other than the “expected” response. The crew must be given the flexibility to “play out their hand” to a logical conclusion. These diversions can normally be identified as a result of a key event or within an event set (e.g., scripting possible diversions and changes in routing). The script should include: Event set number, phase of flight, communications (including frequency and radio call), key events, and expected actions. With these categories, weather conditions and general environmental conditions should be presented as they would occur in an actual situation.
(3) If the script is being developed as an LOE, detailed success criteria must be established. Technical performance criteria are documented in applicable regulations or in company documentation, and only an overview is presented in the event set documentation.
CRM performance criteria are presented for each event set and are divided by CRM behavior areas that have been integrated and validated by the design team. The desired behaviors are presented with a brief statement of what constitutes unsatisfactory behavior. Examples of criteria for the sample LOS shown in tables 6-2, 6-3, and 6-4 include:
(a) Technical Skills. The crew will be proficient in the knowledge and execution of all required takeoff data, analysis of terrain issues, winter operations, systems procedures, and performance limitations of the aircraft.
(b) Communication. The crew will accomplish a pre-departure briefing to include the entire crew (cabin and flightcrew). The briefing will establish the crew climate by emphasizing the importance of interactive decision-making and participation of the entire crew.
The crew is encouraged to voice concerns they may have. Crewmembers will ask questions and seek information from each other about operational issues and decisions. Crewmembers will advocate issues until an acceptable solution is achieved. All problems should be recognized and decisions for their solutions made.
(c) Decisionmaking. The captain asks for and considers crew inputs, but the captain makes the final decision for the aircraft configuration as dictated by weather, performance and fuel requirements. The crew continually assesses the changing conditions to improve the operation of the flight.
Par 610 Page 41
(d) Workload Management. The crew will distribute the workload to ensure that each member is used while no one is over-taxed. The crew will use available resources to analyze the required tasks for this complex departure.
(e) Unsatisfactory Performance. Unsatisfactory performance of this event set includes a crew that is completely unaware of winter operations and the ramification on performance operating considerations. Also judged unsatisfactory is a crew that is not prepared for the complex departure, including the issue of the rapidly rising terrain. Other issues the evaluator observes during this event set may be result in a judgment of unsatisfactory performance.
b. Using these success criteria for the LOE, the evaluation is based on the outcome of the event set much like the current evaluation of the outcome of a maneuver. Within the event set, specific objectives are assigned, any one of which could be involved in the unsuccessful outcome of the event set.
c. A systematic approach to validating scenarios in terms of their training objectives should be adopted. Formal and informal review panels, analysis of data on scenario attributes, and feedback from Instructors, Check Airmen, line pilots, and FAA inspectors provide the information needed to validate or modify the scenario.
d. After the LOS is represented by the script, the LOS should be flown by at least two different crews and if possible the POI or designated representative. If possible, crews flying the LOS should be taped for viewing. These tapes will serve as a useful tool for training the instructors/evaluators prior to implementation. When the crews have finished the LOS, they should complete the same validations forms used to integrate the CRM. Quite often, the crews flying the event sets will have a different rating for the CRM behaviors as compared to the experts. From this final validation, the LOS will be modified prior to instructor training and implementation.
e. There should also be a scenario validation accomplished by an instructor not conducting the LOS. The scenario will be evaluated for its value in meeting the training objectives and for determining the level of facilitator skill required to administer the LOS.
The evaluation of the LOS will also provide an opportunity to note any errors that may exist in the facilitator’s guide or in the flight documentation.
f. The FAA must approve LOE scenarios prior to their use. However, if LOE event sets are designed to allow for their recombination in a “mix and match” configuration to create different LOEs, then event sets may be individually approved by the FAA for that purpose.
When pre-approved event sets are employed in new combinations, the resulting new LOE scenarios do not require additional FAA approval.
Page 42 Par 610