Par 7-3 Page 37 profile and matrix methods to represent the frequency of CRM categories over the entire scenario).
4.2 Fly the LOS scenario using at least two different crews. Invite the Principal Operations Inspector (POI)/Aircrew Program Manager (APM) to participate.
Consider recording some of these sessions for use in developing facilitator training materials.
4.3 Administer the LOS validation instrument form to crews and facilitators instructors that fly the scenario.
4.4 Make required modifications to the revised LOS scenario.
4.5 Submit LOFT or Line Operational Evaluation (LOE) scenario to the POI for approval.
5) Instructor Training Implementation and Evaluation of LOS Scenarios.
5.1 Develop the final representation of LOS for facilitators, with the emphasis on event sets.
5.2 Develop the training plan and materials for recurrent training of facilitators and train the instructors/evaluators.
5.3 Implement the LOS scenario at the fleet level, and evaluate using actual facilitator and crew feedback.
7-4. IDENTIFICATION OF PRIMARY CRM TRAINING OBJECTIVES.
a. Identify Concepts and Definitions. Before an operator can develop a meaningful LOS, it must identify the CRM concepts and definitions that are meaningful within its own culture (refer to the current edition of Advisory Circular (AC) 120-51, Crew Resource Management Training). Several different organizations of CRM concepts can be used, but all of them follow a similar structure:
(1) High-level categories or elements, such as situational awareness and workload management.
(2) Supporting each category are knowledge and skills that should be trained. For example, in workload management, the knowledge would include how to prioritize tasks, while the skill would be the clear assignment of tasks that are understood by all crewmembers.
b. Measure Proficiency. To measure proficiency in the LOS, there should be a set of observable behaviors to look for. For example, in workload management, communicating task priorities is an observable behavior.
7-5. IDENTIFICATION OF PRIMARY TECHNICAL OBJECTIVES. Each carrier’s flight operations department develops technical training objectives. Identification of these objectives,
Par 7-5 Page 38 in the form of proficiency objectives, is a specific requirement of Advanced Qualification Program (AQP). A subset of these objectives can be selected as the technical objectives for each LOS. These objectives will serve an important role in selecting the event sets that will comprise the LOS scenario. Technical objectives might include:
• Origin, routing, and destination.
• Revised departure or arrival procedures.
• Alternate operation of flight management systems (FMS).
• Partial or full loss of integrated FMSs.
• Abnormal and emergency events.
• Adverse weather and environmental conditions.
7-6. DEVELOPING LOS SCENARIOS WITH AUTOMATION THEMES.
a. Automation. Automation is the replacement of a human function, either manual or cognitive, with a machine function. This definition applies to all levels of automation in all aircraft. Effective use of automation means using that level most appropriate to support the priorities of safety, economy, and stated flight operations policies of the individual air carrier.
b. Automation Proficiency. Flightcrew members must be proficient in operating their aircraft in all levels of automation. They must be knowledgeable in the selection of the appropriate degree of automation, and must have the skills needed to move from one level of automation to another.
c. Automation within LOS Scenarios. When developing LOS scenarios with an automation theme, the following items should be considered:
(1) The unique workload distribution between the pilot flying (PF) and pilot monitoring (PM).
(2) The effects of varying levels of automation on situational awareness and workload distribution.
(3) Flightcrew members’ proficiency at dealing with ATC communications, clearance, and weather changes in the automated flight deck.
(4) Company policy and guidelines on high technology procedures.
(5) The effects of lowering levels of automation with decreased levels of crew situational awareness.
(6) Flightcrew members’ proficiency at recognizing automation failures (e.g., autopilot or autothrottle) and, if necessary, taking action to manually control the aircraft.
d. Scenario Design. Scenario designs should be guided by the skills necessary for the individual flightcrew members, as well as the skills necessary for the fully integrated crew.
Scenarios should attempt to engage all crewmembers in CRM activities and should be based on specific training and performance objectives.
Par 7-7 Page 39
7-7. IDENTIFICATION OF POSSIBLE INCIDENTS THAT WILL PRODUCE THE TRAINING OBJECTIVES.
a. Internal Resources. Candidate incidents can be identified through a search of the ASRS database, company incident reports, or company ASAP reports and FOQA events. These incidents can be divided into broad categories of events and these categories then sampled to identify primary issues to be included in new LOS scenarios. Examples of primary issues include rerouting/amended clearance incidents, low fuel during excessive vectoring, and airborne conflicts attributed to flightcrew workload (e.g., delayed approaches, similar call signs, and autoflight incidents). It is common for AQP curriculums to base next year’s scenarios on last year’s ASAP findings. Actual events also provide a template for an event set or even an entire LOS. This prevents the scenario developers from having to begin with a blank sheet each year, and assures the realism of the event.
b. Other Resources. Some other excellent sources for candidate incidents are:
(1) Frequently misused or misunderstood sections of the flight manuals.
(2) Incident reports from other databases, including the International Civil Aviation Organization (ICAO) incident database.
(3) Maintenance difficulty areas identified on the line, during proficiency checks and during training.
(4) Poor performance areas identified on the line, during proficiency checks and during training.
c. Triggers. These sources focus primarily on operational abnormalities and emergencies.
However, CRM skills are required in all aspects of flight operations, including normal operations. By focusing on a specific set of event triggers and associated crew behaviors, a more useful breakdown of these behaviors is obtained. Each type of trigger, whether occurring during normal or abnormal operations, has its own unique CRM requirements:
(1) Normal Operations. CRM behaviors should appear during briefings, crew formation/team building, communications (e.g., inquiry and advocacy), contingency planning, and workload distribution.
(2) Abnormal Operations. Once operations become abnormal or excessively demanding, the required CRM skills will be altered. Some examples include:
(a) Detection of an Abnormal Event.
1. To detect abnormalities, the crew must maintain workload and situational awareness at acceptable levels.
2. Knowledge of checklists, systems, and procedures is required.
(b) Diagnosis and Assessment of an Abnormal Event.
Par 7-7 Page 40
1. Once detected, the abnormal event must be correctly diagnosed and appraised.
2. Appropriate information must be integrated.
3. Essential and nonessential information must be recognized, requiring ongoing situational awareness.
4. The assessment must be communicated to, and acknowledged by, other crewmembers. Challenges should be made when appropriate and all information should be shared.
7-8. EVENT SET SOLUTION VERSUS MANAGEMENT AND EVENT TRIGGER SELECTION.
a. Event Set Selection. Selection of event sets should take into account the types of problems they raise. A mix of simple and complex problems increases the benefits offered by an LOS.
(1) Simple Problems:
(a) Have no further consequences on the conduct of the flight once they have been diagnosed and corrected.
(b) If overused, will detract from realism. Use of one or two of these events can set a proper stage for CRM LOS, but including a number of these events, without logical connection or reason, detracts from the training. If the LOS training objectives are stated properly, they will help to preclude excessive use of nuisance events in the scenario.
(2) Complex Problems:
(a) Have ongoing consequences that must be dealt with in flight, but cannot be fixed.
(b) Add sufficient complexity to the scenario to require the coordinated action of all crewmembers for successful completion, but not to the extent that they induce complete crew failure such as a crash.
(c) Can be compounded by other events, such as weather or ATC-induced complications.
b. Event Set Impacts. The impact of an event set is also influenced by the extent to which the solution to the problem can be solved by means of established standard operating procedures (SOP). The combination of simple versus complex and proceduralized versus nonproceduralized problem characteristics produces three basic categories likely to be found in realistic event sets:
(1) SOP Event Trigger Solution.
Par 7-8 Page 41
(a) These event problems have specific rules or procedures for resolution.
(b) The book procedure for problem solution resolves or reverses the abnormal condition.
(c) After diagnosis, this situation requires no crew decision. The crew selects and uses the appropriate rule (e.g., manual gear extension).
(d) The nature of this type of problem is unlikely to require a high level of CRM skill, unless the event set is already particularly demanding or time compressed, requiring multitask prioritization. In this case, the events are interrelated in the same timeframe.
(2) SOP Event Trigger Management.
(a) This type of event often entails continuous monitoring or system compensation.
(b) The defining characteristic is that the corrective procedure does not solve the problem.
(c) In the case requiring continuous monitoring tasks, the crew will need to prioritize tasks and reduce the effects of distraction (e.g., monitoring a constant speed drive (CSD) outlet temperature in the caution zone).
(d) Problem management should require more inherent CRM skills than SOP problem solution.
(e) On a continuum of difficulty, these types of problems lie between simple and complex problems.
(3) Knowledge-Based Solution/Management.
(a) A book procedure or solution is not available to the crew.
(b) Crews are required to brainstorm a solution or management strategy.
(c) Knowledge-based solutions and management strategies require a decision-making process that often engages multiple crewmembers.
(d) An example of a knowledge-based solution is the selection of an alternate airport when weather or other conditions prohibit landing at the planned destination. There may be no SOP for alternate selection.
c. Summary. Awareness of the types of problems raised by each event set within a LOS scenario will help to ensure that scenario objectives are met and selected CRM skills are used.
Par 7-8 Page 42 TABLE 7-2. SELECTED SCENARIO EVENT SET INDEX WITH PHASES OF FLIGHT
AND PROFICIENCY OBJECTIVES
SCENARIO EVENT
SET NUMBER
PHASES OF FLIGHT
TERMINAL PROFICIENCY
OBJECTIVES Scenario Event Set One Predeparture and Pushback Dispatch–Winter Preflight–with Malfunctions Start and Pre-taxi–Hung Start Scenario Event Set Two Taxi Taxi–Low Visibility Taxi–Winter Conditions Deicing Scenario Event Set Three Takeoff Takeoff–Winter Conditions Climb to Cruise Altitude–Winter Conditions Scenario Event Set Four Climb Climb to Cruise Altitude–Winter Conditions Scenario Event Set Five Cruise En Route Cruise–Winter Conditions, with Malfunctions–Severe Compressor Stall Scenario Event Set Six Descent Descent from Cruise–Winter Conditions, with Malfunction Engine Out Driftdown–Winter Conditions Scenario Event Set Seven Approach and Landing Engine-Out Instrument Landing System (ILS)–Winter Conditions Engine-Out Landing–Winter Conditions Taxi In–Winter Conditions Scenario Event Set Eight Taxi/Parking Parking–Winter Conditions Shutdown–with Auxiliary Power Unit (APU) Postshutdown NOTE: The general theme of the entire scenario is winter operations occurring across all event sets, and the scenario contains a simple problem (hung start in event set one) and a complex one (compressor failure crossing event sets five through eight).
7-9. THE EVENT SET MATRIX.
a. Event Set Matrices. An event set matrix will provide a quick reference source for specific items to be accomplished during the LOS, and will help to ensure that all proficiency objectives identified in the training program are accomplished. In addition, the matrix can be used to categorize the problems as simple to complex, in order to identify demands that will be placed on the crew.
b. Performance Indicators. CRM performance indicators for each event set must also be developed. This will require the integration and validation of the CRM skills to produce a similar matrix. The validation process is discussed in paragraph 7-11.
Par 7-10 Page 43
7-10. LEVEL OF DIFFICULTY. It is important to control the level of difficulty for all LOS scenarios, but it is imperative for the LOE. Students should not have to struggle because their evaluation is more difficult than another student’s evaluation. The most common methodology used in AQPs is to develop a difficulty scale for each task, subtask, and element (if necessary) on the job task list. Operators training using traditional means can develop a similar inventory of flight elements to rate. These values vary from aircraft type to aircraft type, so separate analyses must be conducted for each fleet type. One carrier uses a 5-point scale, where noncritical events are scored 1–3 and critical events 4–5. For example:
• 5 – Wind shear, ditching, evacuation, emergency descent, landing with flaps or slats jammed, etc.
• 4 – Bomb on board, incapacitation, manual Nonprecision Approach (NPA), rejected takeoff below 100 knots, etc.
• 3 – Manual precision approach, manual holding, instrument meteorological conditions (IMC) takeoff, runway change, operations out of limits, unreliable speed indication, etc.
• 2 – Visual meteorological conditions (VMC) takeoff, abnormal descent, two-engine manual landing, missing documentation during ground operations, etc.
• 1 – Normal descent, two-engine autolanding, FMS normal workload, normal after landing operations, etc.
NOTE: Most event set worksheets have a field for the event set trigger, the distractor(s), the supporting event(s), and the difficulty rating for each particular event set. Once the scenario is drafted all of the difficulty values for all of the events are added together. This number is then compared to what is considered an acceptable range of difficulty such as 35–45. If the scenario is out of range, events may be switched out to bring the overall scenario within range.
7-11. INTEGRATION AND VALIDATION OF CRM SKILLS TO EVENT SETS.
a. Observable Behavior. The matrix in Table 7-3, Selected Scenario Event Set Index With Phases of Flight and Crew Resource Management Behaviors, shows how to link observable CRM behaviors to each of the scenario event sets based upon the defined CRM objectives and crew tasks of the event set. A subset of these linked observable behaviors will be selected for validation based upon agreement between the experts.
b. Validation. Management and fleet subject matter experts (SME) should be used to validate the links between observable crew behaviors and scenario event sets. The participants can include instructors, check pilots/check Flight Engineers (FE), FAA inspectors, and managers from CRM departments. This relatively small group of participants should be familiar with the CRM process and flight training program.
c. Rating. Validation data should be collected using a rating system. The form should be a standalone form used by pilots who have flown at least one of the series of event sets and, therefore, have some experience with the event sets. The form should present sufficient background information to explain the scenario event set approach to assessment and should be limited to a set of observable behaviors that can be rated in about 1 hour. The ratings may use
Par 7-11 Page 44 a 5-point scale, where 1 signifies that there was a very low probability that the observable crew behavior was important in the assessment of tasks being performed, and 5 signifies that there was a very high probability that it was important.
d. The Validation Process. The validation process begins with the presentation of the selected scenario event sets. The selected event sets should be representative of the range of CRM assessment categories and should provide sufficient material to develop a number of scenarios. As modules, scenario event sets can be thought of as building blocks in scenario development, and the group of selected scenario event sets defines the range and boundaries. The validation results for the CRM assessment categories and observable crew behaviors are then presented, together with their links to the scenario event sets.
e. Assessment. These two sources of data, the ratings of the CRM categories and the ratings of the observable crew behaviors, can be used to generate complementary representations or profiles of a group of event sets, individual scenarios, or individual event sets. These ratings results also demonstrate that there can be agreement about the primary observable behaviors among those making CRM assessments. When scenario event sets are specified and listed with likely crew behaviors, experienced evaluators can show substantial agreement on the primary observable behaviors to properly assess the related tasks. Therefore, it is likely that making CRM assessments based on observable behaviors will produce reliable assessments.
f. Conclusion. The end result of the validation phase is an event set matrix that lists the events, technical requirements, and CRM behaviors for each event set (see Table 7-4, The Event Set Matrix). With the event sets defined, the proficiency objectives assigned, and the CRM objectives validated, the design team is now ready to develop the scripts and fly the scenarios.
Flying the scenarios is a critical step in the final determination that the training objectives are being met. From this final step, instructor training and the development of supporting documentation for the LOS can be developed.
Par 7-11 Page 45 TABLE 7-3. SELECTED SCENARIO EVENT SET INDEX WITH PHASES OF FLIGHT
AND CREW RESOURCE MANAGEMENT BEHAVIORS
SCENARIO
EVENT SET NUMBER
SITUATIONAL
AWARENESS
WORKLOAD
MANAGEMENT PLANNING DECISION MAKING Event Set One- Predeparture Pilot flying (PF) planned deice for winter operations standard operating procedure (SOP) PF briefed rising terrain PF analyzed departure weather radar (WX) and requests takeoff alternate Event Set Two-Taxi Crew discussed route and holdover times Taxi – Low Visibility Deicing pad Surface Movement Guidance and Control System (SMGCS) plan Pad coordination Event Set Three-Takeoff Crew discussed icing issue before it could become a problem Crew set clear priorities for tasks and their order
Event Set Four-Climb PF requested higher altitude
Event Set Five-Cruise PF directed pilot monitoring (PM) to deal with engine problem PM performed needed checklists and announced compliance Crew assessed one engine landing with WX at diversion field PF calculated time and distance to Mahlon Sweet Field Airport (EUG) PF stated that they cannot go back to Seattle-Tacoma International Airport (SEA) Event Set Six-Descent PF prioritized tasks and got ready for approach PF reviewed single engine approach procedures and aircraft evacuation
Event Set Seven- Approach and Landing PF properly prioritized PM provides backup for PF on all his tasks PF briefed cabin crew PF planed and briefed Safety Enhancement (SE) instrument landing system (ILS)
Event Set Eight-Taxi In
NOTE: The four CRM challenges are spread across the eight event sets, such that only two of the eight event sets presents more than two CRM challenges. One of the strengths of the event set methodology is that it allows the facilitator to focus on a limited set of CRM issues at a time.
Par 7-11 Page 46 TABLE 7-4. THE EVENT SET MATRIX
EVENT SET PHASE OF FLIGHT TECHNICAL REQUIREMENTS KEY EVENTS CRM BEHAVIORS
EVENT SET #1– Predeparture the crew must consider winter operations.
Predeparture Pushback Deicing procedures must be followed.
Takeoff alternate is required.
Departure, en route, and arrival in winter conditions.
Destination weather is at CAT IIIa minimums.
During engine start there is no N1 indication on Engine #1.
OR The #2 engine has a hung start, but starts on the second attempt.
OR When turning the engine anti-ice on, one valve fails to open.
COMMUNICATION:
Open, interactive crew climate established, crew asks questions and seeks answers on operational issues they are concerned about.
DECISION MAKING:
Captain asks and receives input, but makes decisive final decisions affecting mission. Crew continually assesses changing conditions to improve operations.
WORKLOAD MANAGEMENT:
Efficient workload distribution so no one is overtaxed.
EVENT SET #2–Taxi Taxi Takeoff from short runway in winter conditions with takeoff gross weight near runway limit.
Flaps 5/15 takeoff required Engine run-up required in takeoff position Cycle gear after takeoff Taxi via slippery and congested ramps and taxiways in low visibility.
The takeoff runway limited, low visibility and icing conditions near runway limit.
There is rapidly rising terrain to the south of the departure runway.
Complex departure in icing conditions.
COMMUNICATION:
Air traffic control (ATC) interaction, problem definition about deicing and rising terrain.
WORKLOAD MANAGEMENT:
Prioritize tasks for deicing and departure.
DECISION MAKING:
Captain decisive about rising terrain issues, with crew input.
Par 7-11 Page 47 TABLE 7-5. THE EVENT SET ASSESSMENT/GRADE SHEET Rate crew 1-5 Not Obs CRUISE Event Set # 5 Pilot Flying: ❒Left Seat ❒Right Seat
a. Proficient in use of FMS and AFDS. (4.1.2.2)(4.1.2.3)
b. All normal/non-normal procedures accomplished in accordance with standard operating procedure (SOP). (2.1.1)(2.1.2)(2.1.3)(2.1.4)
c. Evaluates options and determines a suitable airport for landing. (4.1.1.7)(4.1.3.6)(5.1.2)
d. Briefs F/As using TEST procedures in preparation for landing. (10.1.3.6)(BC 1.6)
e. Crew manages automated systems to increase SA and avoid work overload. (AT 6.6)
f. Roles, tasks, and responsibilities clearly assigned. Guidelines established. (LT 2.3)
g. Communications with cabin crew/ATC/company clear and timely. (SA 1.6)(SA 1.7)(LT 2.3)
h. Determine plan, and discuss aircraft configurations, airport-specific procedures, and performance. (SA 3.1)
i. Establish and brief “bottom lines” and “backup” plans. (DM4.3)
Pilot #1 Technical CRM Pilot #2 Technical CRM
COMMENTS: REQUIRED FOR ALL ITEMS GRADED 1, 2 OR NOT OBS.
Rate crew 1-5 Not Obs DESCENT Event Set # 6 Pilot Flying: ❒Left Seat ❒Right Seat
a. Proficient in use of FMS and AFDS. (5.1.1.2)(5.1.1.3)
b. Accomplishes descent/arrival, approach briefing, and normal checklists procedures in accordance with SOP. (5.1.1)
c. Ensures all non-normal checklists are completed prior to commencing approach. (10.1.3)
d. Ensures that F/As and passengers are prepared for landing and possible evacuation.
(10.1.3.6)
e. Crew verbalizes a plan, to include bottom lines and a backup plan. (DM 4.4)
f. Crew is assertive when voicing concerns, deviations from the original plan, and when nearing/reaching bottom lines. (LT 2.7)
g. Crew plans and briefs automation modes and configurations. (AT 6.1)
h. Informs appropriate personnel of emergency situation; keeps cabin crew and passengers informed and updated. (BC 1.6)
Pilot #1 Technical CRM Pilot #2 Technical CRM
COMMENTS: REQUIRED FOR ALL ITEMS GRADED 1, 2 OR NOT OBS.
NOTE: This example has two separate job task analyses: one for technical tasks and one for CRM tasks. Technical tasks are identified in parentheses with no letters in front of the numbers. CRM tasks are identified in parentheses with letters in front of the numbers. In this example, DM stands
Par 7-11 Page 48 for decision making, LT for leadership and teamwork, SA for situational awareness, BC for briefings and communications, and AT for automation and technology. Event set 5 has nine events, three drawn from the list of technical tasks, five from the list of CRM tasks, and one event (d) drawn from both technical and CRM tasks. Thus, event set 5 integrates technical and CRM tasks into a single graded event set.
7-12. SCRIPTING AND VALIDATING THE LOS SCENARIO.
a. Scripting. Experience has shown that the effectiveness of LOS relies on script detail and following the programmed script during the LOS session. To accomplish this, the LOS scenario should be carefully scripted, including ensuring that all ATC communications use correct terminology, timing, and routing. Attention to detail in communications scripting will enhance the realism of the LOS FSTD session. Below are some guidelines for developing effective scripts:
(1) Realism. Scripting should include events and conditions that will take place during the flight. The script should concentrate on realistic inputs that support the flight profile as well as variations determined during scenario development.
(2) Crew Autonomy. Scripting must allow for crew decisions other than the expected response. The crew must be given the flexibility to play out their hand to a logical conclusion.
These diversions can normally be identified as a result of a key event or within an event set
(e.g., scripting possible diversions and changes in routing). The script should include event set number, phase of flight, communications (including frequency and radio call), key events, and expected actions. With these categories, weather conditions and general environmental conditions should be presented as they would occur in an actual situation.
(3) Defined Criteria. If the script is being developed as an LOE, detailed success criteria must be established. Technical performance criteria are documented in applicable regulations or in company documentation, and only an overview is presented in the event set documentation. CRM performance criteria are presented for each event set and are divided by CRM behavior areas that have been integrated and validated by the design team. The desired behaviors are presented with a brief statement of what constitutes unsatisfactory behavior.
Examples of criteria for the sample LOS shown in Tables 7-2, 7-3, and 7-4 include:
(a) Technical Skills. The crew will be proficient in the knowledge and execution of all required takeoff data, analysis of terrain issues, winter operations, systems procedures, and performance limitations of the aircraft.
(b) Communication. The crew will accomplish a predeparture briefing, to include the entire crew (cabin and flightcrew). The briefing will establish the crew climate by emphasizing the importance of interactive decision making and participation of the entire crew.
The crew is encouraged to voice concerns they may have. Crewmembers will ask questions and seek information from each other about operational issues and decisions. Crewmembers will advocate issues until an acceptable solution is achieved. All problems should be recognized and decisions for their solutions made.