Par 2-9 Page 18
(2) SPOs. SPOs are statements of performance, conditions, and standards established at the subtask level. SPOs are used to develop training and evaluation curriculum lessons, modules, and segments. SPOs include the range of flight training equipment and the abnormal and emergency contingencies to be considered for training and evaluation.
(3) Enabling (Learning) Objectives (EO). EOs are used to prepare individuals and crews for subsequent training in an operational cockpit environment. An applicant may identify a certain knowledge factor, cognitive skill, motor skill, or CRM factor as an enabling proficiency objective. These are not normally carried forward in the supporting performance objective statement and, therefore, are not normally found on the qualifications standards document.
However, performance of an SPO would depend on a student first acquiring the particular knowledge, skill, attitude, or CRM factor covered by an enabling objective.
NOTE: A learning objective (usually an enabling objective (EO)), which does not require operational conditions, can be demonstrated in a classroom or academic type setting. A performance objective (usually a TPO or SPO), which requires operational conditions, must be demonstrated in an environment equivalent to the operational environment.
(4) Task Factors Analysis. This process rates TPOs and SPOs using the following factors: criticality, currency, need for training, applicable conditions, and applicable standards.
The determination of criticality and currency guides when and how the objective is trained, validated, or evaluated (see Figure 2-3). To make this determination, the analyst answers a series of questions about each TPO and SPO to describe its performance requirements, both on the line and in the training setting. Criticality is a determination of the relative impact of substandard task performance on overall safety. It indicates an increased need for awareness, care, exactness, accuracy, or correctness during task performance. Critical tasks are proficiency objectives that are trained, validated, or evaluated more frequently during an AQP evaluation period. A currency task is a proficiency objective for which individuals and/or crews maintain proficiency by repeated performance of the item in normal line, duty or work operations. Most currency items are validated during line checks and may be sampled in the Continuing Qualification Cycle.
Tasks that are determined to be critical and not current are trained, validated, or evaluated each evaluation period. Tasks that are determined to be neither critical nor current are trained, validated, or evaluated each continuing qualification cycle.
NOTE: Many flightcrew job task SPOs do not fit the classic definition of a subtask; a specific separate step or activity required in the accomplishment of a task. In recognition, non-critical/non-current SPOs under a common TPO that differ only in knowledge requirements may be trained, validated, or evaluated in a Sim/flight training device (FTD), during line checks, using orals, written or electronic exams, classroom briefings or distributed material. However, it is recommended that these SPOs, as appropriate, be demonstrated in a Sim/FTD on a recurring cycle approved by the ERT.
Par 2-9 Page 19 The FAA recommends that the applicant examine each task, subtask, and element, as appropriate, for the following factors:
(a) Primary factors to be considered:
1. Statement of performance.
2. Environmental conditions affecting difficulty/success.
3. Performance standards (parameters with tolerances).
4. Abnormal and emergency procedure contingencies.
5. Document references (title and section) governing or specifying the operation.
6. Consequence of error to safety.
7. Relative difficulty.
(b) Additional factors:
1. Equipment and system operation dependencies (if used for establishing learning sequences for curriculum development).
2. Criterion for success upon which performance standards are based. If new performance standards are created, this criterion should be established for each task and subtasks
(e.g., the tracking standards for very high frequency (VHF) omni-directional radio (VOR) approaches are based on navigation requirements). The navigation requirements are the criteria for success. Success criteria are developed in those cases where current standards are missing or thought to be inadequate.
FIGURE 2-3.
AQP CONTINUING QUALIFICATION CRITICAL/CURRENCY CHART Training Priority Critical (Y/N) Currency (Y/N) Terminal Proficiency Objective/Supporting Proficiency Objective 1 YES NO Train, validate, or evaluate each evaluation period.
2 NO NO Train, validate, or evaluate each Continuing Qualification Cycle.
3 YES YES Sample at first look/MV/LOE and/or line checks each evaluation period.
4 NO YES Sample at first look/MV/LOE and/or line checks each Continuing Qualification Cycle
Par 2-9 Page 20 FIGURE 2-3.
AQP CONTINUING QUALIFICATION CRITICAL/CURRENCY CHART (CONTINUED) Training Priority Critical Current Month Sample 1 Yes No 12 No 2 No No 24 No 3 Yes Yes 12 Yes 4 No Yes 24 Yes NOTE: The sample size must be large enough to provide reasonable assurance that the population is remaining proficient.
NOTE: The qualification standards document for instructors/evaluators does not need to include conditions or a criticality/currency analysis.
b. Evaluation Strategy. The qualification standards document will identify the curriculum (indoctrination, qualification, and/or continuing qualification) in which specific proficiency objectives will be met. The applicant should consider student entry level in determining this allocation. All terminal proficiency objectives must be included in a qualification curriculum regardless of entry-level analysis. For SPOs, an entry-level analysis determines what objectives will be taught under each curriculum (see Section 3, subparagraph 2-11a of this chapter). All objectives should also be covered in continuing qualification test and evaluation strategies.
(1) Consolidation of Objectives. In the qualification standards document, qualification standards are developed at the task and subtask level only and at no lower level. Tasks become TPOs and subtasks become SPOs by combining performance statements, conditions and standards. TPOs and SPOs having common knowledge, skill, attitude, and/or CRM factors may be consolidated to avoid duplication. The consolidated tasks are translated into TPOs, and a terminal level qualification standard is developed for each one. The consolidated subtasks are translated into supporting proficiency objectives, and a supporting level qualification standard is developed for each one of those as well. An example of consolidation would be non-precision approaches. VOR and Non-Directional Beacon (NDB) approaches may be consolidated as a single proficiency objective if the performance statement and standards are the same.
(2) Conditions, Contingencies, and Media. The qualification standards will include a listing of relevant operational and environmental factors along with equipment contingencies to be addressed in training. They will also identify the media that will be employed to either test, validate, or evaluate the specific training objective. There is a difference between the qualification standard for a qualification curriculum and one for a continuing qualification curriculum. The TPOs and SPOs in a qualification standard for a qualification curriculum must identify the specific set of conditions and contingencies to be employed in training and evaluating a task (usually with an asterisk). In addition, the media description will describe the specific media in which the task will receive final evaluation. In contrast, in a continuing qualification curriculum, the TPOs and SPOs may identify a selectable menu of conditions and contingencies to be employed in training or evaluation, and a media range that will specify the minimum media level in which the task may be trained, validated, or evaluated.
Par 2-9 Page 21
(3) Qualification and Continuing Qualification Curriculums. The qualification standards for both qualification and continuing qualification curricula can be combined into one document as long as the differences in conditions and media are addressed, and the tasks are annotated for applicable curriculums. However, if applicants find it more convenient to submit separate qualification standards documents for these curriculums (e.g., because of differences in conditions, contingencies, and media between the qualification and continuing qualification curricula), they may do so. In addition, some qualification standards may be “aircraft generic” in that they may apply to more than one aircraft type. For ease of use and definition of “fleet common” curricula, applicants may also find it convenient to specify these qualification standards separately from those qualification standards that apply to only one aircraft fleet type.
c. Qualification Standards Document. The qualification standards document is the third of the six documents unique to AQP that is maintained in a current status throughout the life of the program and must have a revision control methodology. It is the central AQP document because it provides the regulatory basis for all deviations from current regulations, and identifies the basic training and evaluation requirements. The qualification standards document has four parts: the prologue, a regulatory comparison, the evaluation/remediation methodology, and the specific TPO or SPO qualification standard.
(1) Prologue. The qualification standards document requires an introductory section that explains the methodology, format, and terminology of the standards to the broad range of company and FAA personnel who will need an in-depth understanding of the document to perform their job functions.
(2) Regulatory Requirements Comparison. The qualification standards document must also include regulatory comparison information. The comparison must meet the requirement of AQP regulatory guidance, which states that the AQP program “...must indicate specifically the requirements of part 61, 63, 65, 121, or 135, as applicable, that would be replaced by an AQP curriculum.” (see Appendix 3 for an example of a regulatory comparison chart).
NOTE: The purpose of these charts is not to justify differences from current traditional practices, but merely to document them once they have been justified. It is the quality control processes of AQP that form the basis for establishing an equivalent level of safety.
(3) Testing/Validation/Evaluation and Remediation Methodology. Before implementing an AQP curriculum, the applicant must decide how, when, where, and who will assess a student’s proficiency on each terminal and supporting objective. The guidelines in Figure 2-3 may be used to translate TPOs’ and SPOs’ criticality and currency ratings into testing strategies for the continuing qualification cycle. This testing strategy may include a discussion describing how similar SPOs will be addressed. For example, the TPO non-precision approach may have many SPOs, such as VOR, NDB, Localizer (LOC), etc. Depending on the criticality/currency rating, all of these approaches do not have to be evaluated each evaluation period. This section of the document is where the applicant can describe how these approaches will be alternated/sampled over multiple evaluation periods or continuing qualification cycles.
The testing/validation/evaluation methodology also addresses the applicant’s approach to
Par 2-9 Page 22 documenting the different qualifications of the instructor/evaluator, in terms of who will conduct the testing, validation, LOE, and line checks. In this section, the applicant can describe the level of device appropriate for conducting the evaluation. It also identifies the point in the curriculum when the testing/validation/evaluation methodology will be applied, and identifies what constitutes a failure and/or unsatisfactory performance. In addition, the applicant must specify the strategy for remedying unsatisfactory performance.
(a) General. For each terminal and supporting proficiency objective, the applicant will designate the testing/validation/evaluation strategy. These strategies could include, and are not limited to:
1. Train to proficiency.
2. Systems/knowledge validation.
3. Procedures validation.
4. Maneuvers validation.
5. LOE.
6. Operating experience.
7. Line check.
(b) Training Media. The applicant also will designate the level of training devices, simulators, or aircraft to be used to evaluate the proficiency objective.
(c) Rating Scale/Scoring. The applicant will identify the rating methodology that will be used to grade the performance of the proficiency objectives against the qualification standards. Typically, the measurement codes associated with performance events are ratings and reason codes or skill categories. Ratings are used to define different quality levels of performance. Rating codes usually are carrier specific and the FAA requires that something more sensitive to performance differences than a binary code is used (i.e., some rating method that provides more performance differentiation than pass/fail for individual items being evaluated.
Results of an evaluation event, such as an LOE, are still recorded on a pass/fail basis). Each carrier should ensure that the grades established on the rating scale are clearly defined, meaningful to the instructor/evaluator (I/E), and easily used for performance assessment.
Although consistency among fleets and across different types of evaluations (line check, maneuvers validation, and LOE) is important and generally desirable, rating scales may be slightly different when used for different purposes such as training vs. evaluation. Figure 2-4 provides an example of a rating scale that discriminates among performance levels:
Par 2-9 Page 23 FIGURE 2-4.
RATING SCALE FOR PROCEDURES VALIDATION, FIRST-LOOK, MANEUVERS
VALIDATION, LOE, OR LINE CHECK
GRADE CRITERIA 1 Unsatisfactory Major deviations from the prescribed qualification standards occur that are not recognized or corrected. Individual or crew performance could result in hull loss or loss of life.
CRM/DRM skills are not effective.
2 Below Standard Deviations from the prescribed qualification standards occur that are not recognized or corrected. Individual or crew performance is safe but would be unsatisfactory if diminished by any amount. CRM/DRM skills are not completely effective.
3 Standard with Debrief Deviations occur from the prescribed qualification standards that are recognized and most corrected. Individual or crew performance meets expectations. CRM/DRM skills are effective.
4 Standard Minor deviations occur from the prescribed qualification standards that are recognized and corrected in a timely manner. Individual or crew performance meets expectations.
CRM/DRM skills are clearly effective.
5 Excellent Performance remains well within the prescribed qualification standards. Individual or crew performance, management and CRM/DRM skills are exemplary.
NOTE: This example should not be taken as limiting possible intervals to a five-point scale. With appropriate scale construction and instructor/evaluator training, carriers may elect to define other scales that maximize the quality (sensitivity, reliability, validity) of the collected data.
(d) Remediation Strategy. This section of the qualification standards document should describe the methodology that will be used to remediate unsuccessful testing, validation, or evaluation sessions. This remediation strategy must detail when and what may be repeated and whether or not additional training is warranted. Remediation strategy must also specify when no more training will be offered to the individual and the resulting actions such as “Referred to Committee,” returned to previous position, etc. Chapter 3, subparagraph 3-4c has expanded discussions on validation, evaluation, and remediation in the qualification and continuing qualification curriculums. This strategy may be presented in narrative text or flowchart format.
(e) Special Tracking. Special tracking is the assignment of an individual trainee to an augmented training or evaluation schedule, or both. It should be applied to individuals who have failed to demonstrate proficiency during an evaluation event (e.g., LOE). There may be other criteria that the certificate holder may use to place an individual on special tracking.
Special tracking triggers could include failure to maintain proficiency, an extended absence from duty, for new PICs, or at the request of the individual trainee. However, the AQP participant may implement an AQP strategy such as reduced training intervals that would offset the need for a special tracking strategy. This section should discuss the following:
Par 2-9 Page 24
1. The situation(s) that requires an individual to be placed in special tracking.
2. The strategy to be used.
3. When special tracking is no longer required.
(4) The Qualification Standard. The information contained in the specific qualification standard is the basis for determining proficiency and evaluation criteria. The qualification standard is constructed by applying a performance statement, conditions, and standards to a task or subtask, thereby creating a TPO or an SPO. Although each operator will determine the format and content of its qualification standards, component fields have developed out of practice and are illustrated in Appendix 5. In this example, the phase of operations is
6.0 Approach Operations and Landing. The TPO is 6.1 Perform an Instrument Approach. The SPOs are 6.1.1 Perform a 2 Engine Precision Approach CAT I ILS (instrument landing system) Approach and 6.1.2 Perform a 1 Engine Inoperative CAT I ILS Approach.
NOTE: Variation in the format of a given air carrier’s qualification standards is permissible if all of the categories of information in the example are addressed.
(a) Header. A header identifies the airline and the document.
(b) Revision Control. Page revision control dates and revision numbers.
(c) Page Numbers. Consecutive page numbers.
(d) Phase of Operations. Number and title from task listing.
(e) Qualification Standard Title. Either TPO(s) or SPO(s).
(f) Hierarchical Task or Subtask. Identifier and title from task listing.
(g) Crew Duty Position(s). This identifies which crewmember(s) will be evaluated performing the task.
1. Pilot in Command = PIC.
2. Second in Command = SIC.
3. Flight Engineer = FE.
4. Captain, First Officer, Second Officer.
(h) Criticality/Currency Rating. This may be the first place that the task factors analysis is tied to the tasks.
NOTE: The qualification standards document for instructor/evaluators does not need to include conditions or a criticality/currency analysis.
Par 2-9 Page 25
(i) Curriculum. This field identifies the curriculum(s) in which the task will be trained and evaluated.
(j) Evaluation Strategy. The evaluation point for a particular qualification standard (e.g., (1) train to proficiency; (2) systems validation; (3) procedures validation; (4) maneuvers validation; (5) LOE; or (6) line check).
(k) Media. The specific media in which training and/or evaluation will be conducted. For qualification, the media is the lowest media used for final evaluation. For continuing qualification, the media includes the range of media used for training, validation, and evaluation.
(l) Performance Statement. An expanded statement of expected behavior that, when executed, will complete the work required for a specific portion of a job. A performance statement specifies precisely what behavior must be exhibited, and may include the knowledge and skill issues that comprise the enabling objective supporting that performance.
(m) Operational and Environmental Conditions. Conditions describe the range of circumstances under which student performance will be measured and evaluated. Conditions include the operational environment (Navigational Aid (NAVAID) inoperable, different aircraft weight, passengers not seated, aircraft configuration, etc.), and natural environment (ceiling, visibility, wind, turbulence, etc.). The qualification standard should: (1) indicate those specific conditions to be trained and tested as part of the qualification curriculum; and (2) provide a more exhaustive listing of conditions over which crewmembers will be trained and tested during the course of successive continuing qualification cycles.
(n) Contingencies. Contingencies include abnormal situations, minimum equipment list (MEL)/Configuration Deviation List (CDL), and emergencies. The qualification standard should (1) indicate those specific contingencies to be trained and tested as part of the qualification curriculum; and (2) provide a more exhaustive listing of contingencies over which crewmembers will be trained and tested during the course of successive continuing qualification cycles.
(o) Maneuver Standards. Observable, measurable parameters of performance with tolerances (e.g., course deviation degrees (+ or -)). Standards include maneuvers, procedures, and CRM considerations.
(p) References. Identify the primary references from which performance statements and associated standards were derived. Cite documents by title and where applicable, chapter or section. Page numbers are not required.
2-10. INSTRUCTIONAL SYSTEMS DEVELOPMENT METHODOLOGY DOCUMENT. This is another of the six documents unique to AQP and must have an acceptable revision control methodology. Applicants with established curriculum development guidelines may submit these for consideration. Others should describe a systematic approach for developing a proficiency-based training system that is organized around the teaching and testing of terminal, supporting, and enabling proficiency objectives. The methodology identifies the rationale, justification, and subsequent documentation to be used in the applicant’s curriculum
Par 2-10 Page 26 development process. The instructional systems development methodology document describes the approach to be used by applicant airlines to develop and maintain all AQP curriculums. The Instructional Systems Development Methodology document should be finalized before constructing curriculums for each duty position. It applies to pilot, instructor, and evaluator programs, and may be expanded to include other personnel, such as F/As and dispatchers. This document is divided into two sections. The first section, Curriculum Development Procedures, describes the applicant’s approach for using the JTAs and qualification standards as baseline documents to construct their general training curriculums across all AQP courses. The second section, Line Operational Simulation (LOS) Methodology, describes the approach for developing LOS scenarios.
a. Curriculum Development Process. Applicants should describe the process they will use to build their curriculums based on the JTA, qualification standards, and proficiency objectives they develop for each duty position. This document should discuss how:
• TPOs and SPOs are allocated to curriculums.
• Learning and evaluation activities are developed to support these objectives.
• Instructional media and methods are assigned to objectives.
• Objectives are clustered and sequenced into lessons, modules, segments, and curriculums (see Figure 2-5 for an example).
• An audit trail (hierarchical numbering or a matrix) will be maintained to link job tasks, proficiency objectives, lesson activities/content, and test items.
NOTE: The resulting curriculums are translated into a course footprint and are documented in the curriculum outline. These curriculums are expanded in more detail in the student and instructor syllabi and in individual lessons and tests.
FIGURE 2-5.
CURRICULUM DEVELOPMENT
Par 2-10 Page 27
b. LOS Methodology. While individual LOE scenario events are approved by the POI, the ERT must first approve the methodology that generates LOE scenarios. Unless the carrier has an alternative proposal, this methodology should follow the general guidelines for LOS development produced by the Airlines for America (A4A) AQP Working Group. This approach divides the typical scenario into a series of relatively independent segments, called event sets. A typical scenario might have six or eight event sets relating to the phases of operations (ground operations, takeoff, climb, cruise, descent, approach, landing, and after landing). Each event set consists of a series of training or evaluation events, which include both technical and CRM activities. This technique enables scenarios to be constructed in a building block approach, ensuring that each event set is carefully sequenced and considered in relation to the other event sets in the scenario.
2-11. CURRICULUM OUTLINE DOCUMENT. This is the fifth of the six documents unique to AQP that must have an acceptable revision control methodology. The curriculum outline is a listing of course material divided into segments, segments into modules, modules into lessons, and lessons into elements. Curriculum outlines are developed and submitted with the understanding that application of the course material may require some flexibility regarding the actual day on which each activity is accomplished. Each part of the curriculum outline must clearly indicate the subject matter to be taught and correspond directly to the hierarchical system of the task analysis. While the curriculum outline document need only go down to the level of the element under each lesson title, the applicant will be required to show the objectives (terminal, supporting and enabling) associated with each lesson. See Figure 2-5 for an illustration of this association. This is part of the necessary audit trail that links the job requirements (JTA) to the training requirements (qualification standards) to the training activities (curriculum outline). A curriculum outline provides the basis for the curriculum footprint, which is a high level graphical overview of the curriculum content depicting the training and evaluation activities and the planned hours for each day of the curriculum (see Figure 3-4). The curriculum outline document should reference the results of the student entry analysis, if one was conducted, and will include a curriculum footprint. Figure 2-7 provides an example of a curriculum outline showing portions of ground training and flight training segments.
a. Entry-Level Analysis. The applicant should develop and document a student entry-level performance analysis for TPOs and SPOs. This analysis compares the KSAs of the student population against the TPOs and SPOs in the JTA in order to fit the instruction to the student. A four-point performance difference rating scale is suggested (see Figure 2-6). Highly skilled instructors familiar with the experience and background of the student population and knowledgeable with the terminal and supporting proficiency objectives should make the rating.
This analysis provides guidance to determine efficient teaching strategies for the indoctrination and qualification curriculums. This analysis can also identify where training is not needed, where basic “enabling” skills must be taught, and what number of trials is needed for an applicant to reach TPO standards. More than one population group may be used in conducting the student entry analysis for a single duty position. The results of such an analysis may be used, for example, to justify alternative curriculum tracks or modules targeted at expected differences in entry background, and in order to achieve the most efficient use of training resources. The initial justification for alternative curriculum tracks must be validated though subsequent data analysis.
Par 2-11 Page 28 FIGURE 2-6.
PERFORMANCE DIFFERENCE RATING SCALE Performance Difference Code Performance Difference Description 4 Meets or exceeds the required performance.
3 Can accomplish tasks with minor errors or omissions. May take longer than expected or allowed.
2 Cannot accomplish tasks. Does demonstrate basic background skills and knowledge.
1 Does not demonstrate basic background experience, skills, or knowledge. Unfamiliar with simplest elements of a task.
b. Trainee and Instructor/Evaluator’s Curriculums. The instructor and evaluator curriculums associated with each duty position must be developed in the same systematic manner as those developed for the duty positions themselves. This requires the development of separate JTA, qualification standards, curriculum outlines and other documents for these positions although they may share some common modules or lessons. Instructors and evaluators require an indoctrination curriculum that is separate from the duty position indoctrination curriculum, as well as a separate qualification and continuing qualification curriculum.
c. Link Qualification Standards to Curriculums. Both the qualification and continuing qualification curriculums for a given duty position are derived from the same set of qualification standards. The link between qualification standards and a curriculum is the proficiency objective.
As a rule of thumb, qualification standards developed from TPOs focus more on the higher-level evaluation activities than qualification standards developed from SPOs. Qualification standards developed from SPOs focus more on the lower level learning activities that may be validated and represent the components of those higher level standards. Therefore, a qualification curriculum will focus equally on qualification standards developed from TPOs and SPOs, while the continuing qualification curriculum will focus more heavily on the qualification standards developed from TPOs. Indoctrination curriculums may or may not have qualification standards of their own. In either case, they provide training on common knowledge items (enabling objectives) that will support the mastery of qualification standards in the qualification and continuing qualification curriculums.
Par 2-11 Page 29 FIGURE 2-7.
SAMPLE AQP CURRICULUM OUTLINE B737 Qualification Curriculum Outline Segment: Ground Training Module: Introduction to Aircraft Lesson #: Aircraft Overview 9.1.4 Element: Fuselage 9.1.4.1 Element: Wings 9.1.4.2 Element: Flight Controls 9.1.4.4, 9.1.4.5, 9.1.4.6 Element: Landing Gear 9.1.4.7 Element: Powerplant 9.1.4.3 Element: Fuel System 9.1.4.8 Element: Hydraulic System 9.1.4.9 Element: Electrical System 9.1.4.10, 9.1.4.10.1, 9.1.4.10.2, 9.1.4.10.3 Lesson #: Aircraft Lighting 9.1.9 Element: Exterior Lighting 9.1.9.1 Element: Cockpit Lighting 9.1.9.2 Element: Cabin Signs and Lights 9.1.9.9, 9.1.9.4, 9.1.9.6 Element: Lighting Power Sources 9.1.9.5 Element: Emergency Lighting 9.1.9.5, 9.1.9.5.1, 9.1.4.10.2 B737 Qualification Curriculum Outline Segment: Flight Training (FT) Module: Flight Training Device (FTD) Lesson: FTD #1 Pre-Flight Normal Checklists Element: Flight Compartment Safety Inspection 1.2.5.1 Element: Flows 1.2.7.1 Element: Acceptance Checklist 1.2.8.1 Element: Before Starting Engines Checklist 1.2.11.1 Element: Normal APU First Engine Start 1.9.1.1 Element: Normal APU Second Engine Start 1.9.1.4 Element: Pushback 1.9.6.1 Element: Before Taxi Checklist 1.4.1.1 Element: Normal Taxi 1.4.2.1 Element: Before Takeoff Checklist 1.4.9.1 Element: Line-Up Checklist 1.4.4.1