training in CRM and technical issues to elicit discussions of points of interest and operational relevance. Positive comments regarding crew performance should be emphasized in the debriefing, as well as comments regarding areas of crew performance that need improvement.
Crewmembers must be given the opportunity to critique and analyze their own performance and review key points.
602. SCENARIO DESIGN PROCESS.
a. The framework described here for developing LOS scenarios is based on the concept of an event set, a group of related tasks and conditions that are part of the scenario and are integrated into the LOS session with specific CRM and technical training objectives. Included in the framework is a new method for identifying specific CRM skills appropriate for the event sets and a tool for developing the CRM category profile of LOS scenarios. The result is an approach that makes LOS sessions more manageable and easier to assess by allowing the instructor or evaluator to concentrate on a few CRM categories within any given event set.
b. The primary unit of both LOS design and CRM assessment is the event set. The event set is made up of one or more events, including an event trigger, distracters, and supporting events. The event trigger is the condition or conditions under which the event is fully activated.
The distracters are conditions inserted within the event set timeframe that are designed to divert the crew’s attention from other events that are occurring or are about to occur. Finally, supporting events are other events taking place within the event set designed to further CRM and technical training objectives.
c. In LOS scenario design, the CRM and technical training objectives should be integrated into the event sets. This event set framework allows the design team to present the appropriate degree of realism in the LOS. Instead of focusing on a single technical issue, the event set integrates the entire complex line environment (e.g., terrain, air traffic control (ATC), weather issues, etc.) to facilitate and maximize the crew’s performance in response to specified CRM and technical issues. With the LOS scenario now defined by event sets, scenario validation is performed at the event set level rather than limiting validation to the overall LOS.
d. The event set framework supports the development of LOS scenarios based on complex events rather than the simple events. Simple events have no further consequences on the conduct of the flight once they have been diagnosed and corrected. Overuse of simple problems or events detracts from LOS realism. Routine prestart problems, followed by a start problem, followed by a taxi problem, intrude on the crew’s perception that the LOS is an actual flight.
However, one or two of these events can be useful for setting a proper environment to facilitate a CRM LOS when the objectives are stated properly within the event set framework.
e. Complex events have ongoing consequences that must be dealt within flight and cannot be solved by simply selecting and executing an abnormal checklist. Event set-based scenarios require the coordinated actions of all crewmembers for successful completion. Complex event set problems tend to be ambiguous, with no simple corrective checklist solution. The properly Page 26 Par 601
designed event set does not necessarily have a single solution. Rather, it may have a number of possible and reasonable solutions. Thus, the well-designed event promotes the management of a complex situation.
f. Table 6-1 describes an LOS development process. The overall purpose of the LOS development methodology is to build event sets that allow for the examination of the crew’s CRM and technical skills.
g. This design methodology provides a rigorous validation process to assure training and evaluation of all critical technical and CRM tasks identified by a training program. In addition, it allows for adaptation based on the operational environment of a particular organization. The remainder of this chapter analyzes each component of the design methodology in detail.
Par 602 Page 27
TABLE 6-1 LOS Design Methodology
1) Identification of primary Crew Resource Management (CRM)/technical training objectives.
1.1 Identify the primary CRM categories (e.g., decisionmaking, communication, workload management) and integrate with the primary technical training objectives.
1.2 Identify the related skills for the CRM categories identified in 1.1.
1.3 Identify the primary technical training objectives.
2) Identification of possible incidents that will produce the training objectives.
2.1 Identify incidents through a search of the Aviation Safety Reporting System (ASRS) database, own-carrier incident reporting, and own-carrier flight safety programs, such as Aviation Safety/Accident Prevention (ASAP) system and flight operational quality assurance (FOQA).
2.2 Develop a preliminary list of relevant incidents and events.
2.3 Refine the listing of incidents and events, and correlate with the CRM categories and observable behaviors.
3) Specification and development of line operational simulation (LOS) scenario event sets.
3.1 Specify LOS scenario objectives, related proficiency objectives, primary and secondary CRM categories, and observable crew behaviors for each scenario event set.
3.2 Translate incidents and situations into scenario event sets by identifying the event trigger, distracters, and supporting events, and specify the phase of flight.
3.3 Integrate the individual scenario event sets into the overall scenario.
3.4 Administer the LOS validation instrument to ensure event sets are specified and organized consistent with the CRM and technical training objectives or proficiency objectives.
Page 28 Par 602
TABLE 6-1 LOS Design Methodology (Continued)
4) Evaluation, modification and, if appropriate, FAA approval of the LOS scenario.
4.1 Represent the LOS scenario showing the event sets, event trigger, ATC communications, and the related CRM categories (e.g., use the CRM category profile and matrix methods to represent the frequency of CRM categories over the entire scenario).
4.2 Fly the LOS scenario using at least two different crews. Invite the principal operations inspector (POI)/automated performance measurement system (APM) to participate. Consider taping some of these sessions for use in developing instructor training materials.
4.3 Administer the LOS validation instrument form to crews and instructors that fly the scenario.
4.4 Make required modifications to the revised LOS scenario.
4.5 Submit LOFT or LOE scenario to the POI for approval.
5) Instructor training implementation and evaluation of LOS scenarios.
5.1 Develop the final representation of LOS for instructors with the emphasis on event sets.
5.2 Develop the training plan and materials for recurrent training instructors and train the instructors/evaluators.
5.3 Implement the LOS scenario at the fleet level, and evaluate using actual instructor and crew feedback.
Par 602 Page 29
603. IDENTIFICATION OF PRIMARY CRM TRAINING OBJECTIVES.
a. Before an operator can develop a meaningful LOS, it must identify the CRM concepts and definitions that are meaningful within its own culture (see AC 120-51, Crew Resource Management Training, current edition). Several different organizations of CRM concepts can be used but all of them follow a similar structure:
(1) High-level categories or elements, such as situational awareness and workload management.
(2) Supporting each category are knowledge and skills that should be trained. For example, in workload management, the knowledge would include how to prioritize tasks, while the skill would be the clear assignment of tasks that are understood by all crewmembers.
b. To measure proficiency in the LOS, there should be a set of observable behaviors to look for. For example, in workload management, communicating task priorities is an observable behavior.
604. IDENTIFICATION OF PRIMARY TECHNICAL OBJECTIVES. Each carrier’s flight operations department develops technical training objectives. Identification of these objectives, in the form of proficiency objectives, is a specific requirement of Advanced Qualification Program (AQP). A subset of these objectives can be selected as the technical objectives for each LOS. These objectives will serve an important role in selecting the event sets that will comprise the LOS scenario. Technical objectives might include:
a. Origin, routing, and destination.
b. Revised departure or arrival procedures.
c. Alternate operation of flight management systems.
d. Partial or full loss of integrated flight management systems.
e. Abnormal and emergency events.
f. Adverse weather and environmental conditions.
605. DEVELOPING LOS SCENARIOS WITH AUTOMATION THEMES.
a. Automation is the replacement of a human function, either manual or cognitive, with a machine function. This definition applies to all levels of automation in all aircraft. Effective use of automation means using that level most appropriate to support the priorities of safety, economy, and stated flight operations policies of the individual air carrier.
b. Pilots must be proficient in operating their aircraft in all levels of automation. They must Page 30 Par 603
be knowledgeable in the selection of the appropriate degree of automation, and must have the skills needed to move from one level of automation to another.
c. When developing LOS scenarios with an automation theme, the following items should be considered:
(1) The unique workload distribution between pilot flying and pilot not flying the aircraft.
(2) The effects of varying levels of automation on situational awareness and workload distribution.
(3) Pilots’ proficiency at dealing with ATC communications, clearance and weather changes in the automated cockpit.
(4) Company policy and guidelines on high technology procedures.
(5) The effects of lowering levels of automation with decreased levels of crew situational awareness.
d. Scenario designs should be guided by the skills necessary for the individual pilot as well as the skills necessary for the fully integrated crew. Scenarios should attempt to engage all crewmembers in CRM activities and should be based on specific training and performance objectives.
606. IDENTIFICATION OF POSSIBLE INCIDENTS THAT WILL PRODUCE THE TRAINING OBJECTIVES.
a. Candidate incidents can be identified through a search of the ASRS database, company incident reports, or company ASAP reports and FOQA events. Categories and primary issues identified by the airline for the new LOS scenario. Examples of primary issues include:
Rerouting/amended clearance incidents, low fuel during excessive vectoring, and airborne conflicts attributed to flightcrew workload (e.g., delayed approaches, similar call signs, autoflight incidents).
b. Some other excellent sources for candidate incidents are:
(1) Frequently misused or misunderstood sections of the flight manuals.
(2) Incident reports from other databases, including the International Civil Aviation Organization (ICAO) incident database.
(3) Maintenance-difficulty areas identified in line and simulator proficiency checks and training.
Par 605 Page 31
(4) Poor performance areas identified in line and simulator proficiency checks and training.
c. These sources focus primarily on operational abnormalities and emergencies. However, CRM skills are required in all aspects of flight operations, including normal operations. By focusing on a specific set of event triggers and associated crew behaviors, a more useful breakdown of these behaviors is obtained. Each type of trigger, whether occurring during normal or abnormal operations, has its own unique CRM requirements:
(1) Normal Operations. CRM behaviors should appear during briefings, crew formation/team building, communications (e.g., inquiry and advocacy), contingency planning, and workload distribution.
(2) Abnormal Operations. Once operations become abnormal or excessively demanding, the required CRM skills will be altered. Some examples include:
(a) Detection of an Abnormal Event.
1 To detect abnormalities, the crew must maintain workload and situational awareness at acceptable levels.
2 Knowledge of checklists, systems and procedures is required.
(b) Diagnosis and Assessment of an Abnormal Event.
1 Once detected, the abnormal event must be correctly diagnosed and appraised.
2 Appropriate information must be integrated.
3 Essential and non-essential information must be recognized requiring ongoing situational awareness.
4 The assessment must be communicated to, and acknowledged by, other crewmembers. Challenges should be made when appropriate and all information should be shared.
607. EVENT SET SOLUTION VERSUS MANAGEMENT AND EVENT TRIGGER SELECTION.
a. Selection of event sets should take into account the types of problems they raise. A mix of simple and complex problems increases the benefits offered by an LOS.
(1) Simple Problems:
Page 32 Par 606
(a) Have no further consequences on the conduct of the flight once they have been diagnosed and corrected.
(b) If overused, will detract from realism. Use of one or two of these events can set a proper stage for CRM LOS, but including a number of these events, without logical connection or reason, detracts from the training. If the LOS training objectives are stated properly, they will help to preclude excessive use of nuisance events in the scenario.
(2) Complex Problems:
(a) Have ongoing consequences that must be dealt with in flight, but cannot be fixed.
(b) Add sufficient complexity to the scenario to require the coordinated action of all crewmembers for successful completion, but not to the extent that they induce complete crew failure such as a crash.
(c) Can be compounded by other events such as weather or ATC induced complications.
b. The impact of an event set is also influenced by the extent to which the solution to the problem can be solved by means of established standard operating procedures (SOP). The combination of simple versus complex, and proceduralized versus non-proceduralized problem characteristics produces three basic categories likely to be found in realistic event sets.
(1) SOP Event Trigger Solution.
(a) These event problems have specific rules or procedures for resolution.
(b) The book procedure for problem solution resolves or reverses the abnormal condition.
(c) After diagnosis, this situation requires no crew decision. The crew selects and uses the appropriate rule (i.e., manual gear extension).
(d) The nature of this type of problem is unlikely to require a high level of CRM skill, unless the event set is already particularly demanding or time compressed, requiring multi-task prioritization. In this case, the events are interrelated in the same time frame.
(2) SOP Event Trigger Management.
(a) This type of event often entails continuous monitoring or system compensation.
(b) The defining characteristic is the corrective procedure does not solve the problem.
Par 607 Page 33
(c) In the case requiring continuous monitoring tasks, the crew will need to prioritize tasks and reduce the effects of distraction (e.g., monitoring a CSD outlet temperature in the caution-zone).
(d) Problem management should require more inherent CRM skills than SOP problem solution.
(e) On a continuum of difficulty, these types of problems lie between simple and complex problems.
(3) Knowledge-Based Solution/Management.
(a) A book procedure or solution is not available to the crew.
(b) Crews are required to brainstorm a solution or management strategy.
(c) Knowledge-based solutions and management strategies require a decision making process that often engages multiple crewmembers.
(d) An example of a knowledge-based solution is the selection of an alternate airport when weather or other conditions prohibit landing at the planned destination. There may be no SOP for alternate selection.
c. Awareness of the types of problems raised by each event set within a LOS scenario will help to ensure that scenario objectives are met and selected CRM skills are used.
Page 34 Par 607
TABLE 6-2 Selected Scenario Event Set Index With Phases of Flight and Proficiency Objectives
SCENARIO EVENT
SET NUMBER
PHASES OF FLIGHT TERMINAL PROFICIENCY
OBJECTIVES Scenario Event Set One Pre Departure, and Push Back, Dispatch - Winter Preflight - with Malfunctions Start and Pre-Taxi - Hung Start Scenario Event Set Two Taxi Taxi – Low Vis Taxi – Winter conditions De-icing Scenario Event Set Three Takeoff Takeoff - Winter Conditions Climb to Cruise Altitude - Winter Conditions
Scenario Event Set Four Climb Climb to Cruise Altitude - Winter Conditions
Scenario Event Set Five Cruise En route Cruise - Winter Conditions, with Malfunctions – Severe Compressor Stall
Scenario Event Set Six
Descent Descent from Cruise - Winter Conditions, with Malfunction Engine Out Driftdown - Winter Conditions Scenario Event Set Seven Approach and Landing Engine Out ILS - Winter Conditions Engine Out Landing - Winter Conditions Taxi In - Winter Conditions
Scenario Event Set Eight
Taxi/Parking Parking - Winter Conditions Shutdown - with APU Post Shutdown
608. THE EVENT SET MATRIX.
a. An event set matrix will provide a quick reference source for specific items to be accomplished during the LOS, and will help to ensure that all proficiency objectives identified in the training program are accomplished. In addition, the matrix can be used to categorize the problems as simple to complex in order to identify demands that will be placed on the crew.
Par 607 Page 35
b. CRM performance indicators for each event set must also be developed. This will require the integration and validation of the CRM skills to produce a similar matrix. The validation process is discussed in the next section.
609. INTEGRATION AND VALIDATION OF CRM SKILLS TO EVENT SETS.
a. The matrix in Table 6-3 shows how to link observable CRM behaviors to each of the scenario event sets based upon the defined CRM objectives and crew tasks of the event set. A subset of these linked observable behaviors will be selected for validation based upon agreement between the experts.
b. Management and fleet subject matter experts should be used to validate the links between observable crew behaviors and scenario event sets. The participants can include instructors, check airmen, FAA inspectors, and managers from CRM departments. This relatively small group of participants should be familiar with the CRM process and flight training program.
c. Validation data should be collected using a rating system. The form should be a standalone form used by pilots who have flown at least one of the series of event sets and, therefore, have some experience with the event sets. The form should present sufficient background information to explain the scenario event set approach to assessment and should be limited to a set of observable behaviors that can be rated in about 1 hour. The ratings may use a five-point scale where “1” signifies that there was a very low probability that the observable crew behavior was important in the assessment of tasks being performed, and the number “5” signifies that there was a very high probability that it was important.
d. The validation process begins with the presentation of the selected scenario event sets.
The selected event sets should be representative of the range of CRM assessment categories and should provide sufficient material to develop a number of scenarios. As modules, scenario event sets can be thought of as building blocks in scenario development, and the group of selected scenario event sets defines the range and boundaries. The validation results for the CRM assessment categories and observable crew behaviors are then presented, together with their links to the scenario event sets.
e. These two sources of data, the ratings of the CRM categories, and the ratings of the observable crew behaviors, can be used to generate complementary representations or profiles of a group of event sets, individual scenarios, or individual event sets. These ratings results also demonstrate that there can be agreement about the primary observable behaviors among those making CRM assessments. When scenario event sets are specified and listed with likely crew behaviors, experienced evaluators can show substantial agreement on the primary observable behaviors to properly assess the related tasks. Therefore, it is likely that making CRM assessments based on observable behaviors will produce reliable assessments.
Page 36 Par 608
f. The end result of the validation phase is an event set matrix that lists the events, technical requirements, and CRM behaviors for each event set (see Table 6-4). With the event sets defined, the proficiency objectives assigned, and the CRM objectives validated, the design team is now ready to develop the scripts and fly the scenarios. Flying the scenarios is a critical step in the final determination that the training objectives are being met. From this final step, instructor training and the development of supporting documentation for the LOS can be developed.
Par 609 Page 37