Page 34 Par 2-17
d. The applicant and the FAA to support analysis for special subjects, such as CRM performance factors.
e. The FAA to analyze and validate curriculum performance.
f. The FAA to analyze and validate program development, implementation, and maintenance procedures.
2-18. REVISIONS. The focus of this phase is the validation of the AQP curriculum by acquiring performance/proficiency data and lessons learned in the course of conducting the curriculum. Applicants will summarize the lessons learned and adjustments made to the curriculums in an annual report. In addition, adjustments made to the AQP will be reflected in revisions to the approved AQP documents. The successful completion of this phase, and approval of the commensurate revisions to the approved AQP documents and reports, will qualify an applicant for entry into the final phase of the AQP process: Phase V, Continuing Operations.
2-19. APPROVAL. Approval of the continuing operations marks the end of phase IV and the entrance into phase V.
SECTION 6. PHASE V: CONTINUING OPERATIONS
2-20. OVERVIEW. In this phase, the applicant continues operation of the AQP unless the FAA withdraws approval or unless the applicant withdraws or modifies the AQP. This phase requires the maintenance of the AQP-approved documentation and updated documentation of the data requirements for all curriculums.
2-21. DATA’S IMPACT ON CONTINUING OPERATIONS. Data will continue to be collected and analyzed by the applicant and the FAA as it was during phase IV activities.
2-22. QUALITY ASSURANCE. Applicants must pay particular attention to overall program quality. The FAA expects an AQP quality assurance program to identify needed changes in curriculum, courseware, and equipment, and to make these changes before an unwanted trend in reduced proficiency manifests itself. Continued validation of performance/proficiency data as it pertains to individual and crew proficiency, as achieved and maintained by all personnel, is particularly important.
SECTION 7. DOCUMENT AND REPORTING REQUIREMENTS
2-23. OVERVIEW. The approved AQP documentation establishes the applicant’s regulatory requirements for the particular training program. This fact makes it imperative that the applicant develops a document structure that is organized and standardized. The structure must ensure that the information contained within the AQP documents will be used and translated down to the lowest level training guidance. It must allow the applicant’s entire organization to easily access
Par 2-23 Page 35 and use the contents. It is important to draw a distinction between the AQP process and the required FAA documentation each applicant must provide to ensure regulatory compliance. The FAA has established a minimum listing of AQP documentation. The applicant will want to develop other more specific materials that ensure the approved AQP information distills to all training and evaluation guides.
a. FAA-Required Documentation. Each document contains unique information integral to the AQP that will be used when developing subsequent curriculums. The documents, once developed, will require periodic updates and, therefore, are subject to the revision control process discussed in Chapter 4. These documents should be submitted to the FAA in electronic format:
(1) The application.
(2) The task analysis.
(3) The qualification standards.
(4) The instructional systems development methodology.
(5) The curriculum outlines.
(6) The I&O Plan.
b. Document Structures. Figure 2-8 provides an example of an AQP document structure.
It is not required that an applicant follow this structure. However, the document structure that is adopted must easily identify the location of the AQP documents and more specifically, the required AQP information. If the AQP documentation is part of a subset of other manuals, a method should be developed to identify the manual and the specific AQP document that it contains.
c. Annual AQP Report. AQP requires that each AQP certificate holder prepare an annual report for the FAA (see Appendix C for specific requirements). This report is based on the certificate holder’s analysis of the data that is collected during training and at strategic points (validation/evaluation gates) in each curriculum and maintained in the PPDB. AQP requires data collection and analysis in order to establish and maintain quality control of curriculums for crewmembers, instructors, and evaluators. The annual AQP report should summarize the lessons learned and adjustments made to the curriculum(s) during the reporting period. The report should also include projected or proposed changes to the curriculum(s) based on the certificate holder’s current analysis. The actual adjustments made to the AQP are reflected in revisions to the approved AQP documents. The report should be submitted to the FAA no later than 60 days past the end of the report period. The reporting period is usually based on the approval date for a particular curriculum in either phase IV or V. During AQP development, particularly for multiple fleet operators, with different approval dates for multiple curriculums, the reporting period may be modified (as agreed upon by the FAA and the certificate holder). Once the certificate holder has all its fleets and curriculums into phase V, the reporting period can be fixed
Page 36 Par 2-23 into a particular cycle. Copies of the report should be distributed to the POI and AFS-230 at least 2 weeks prior to the annual AQP review meeting.
d. Annual AQP Review - Phases IV and V. An annual AQP review meeting between the FAA and the certificate holder should be held to coincide with the submission of the annual AQP report in phases IV and V. The purpose of the meeting is to formally discuss the results of the certificate holder’s data analysis, program revisions, future revisions and the analysis of the data that is submitted to the FAA. Below is a listing of suggested agenda items for a Phase IV and V review:
(1) Data Management.
(a) Collection problems and fixes.
(b) Analysis - data reliability/validity/sensitivity.
(c) Data usefulness.
(d) Problem areas investigated.
(2) Data Collection Method/Tools.
(3) Data Analysis Methods/Tools.
(a) Review of annual report.
1 Qualification.
2 Continuing qualification.
3 Line check.
(b) Identified trends (positive and negative).
(c) Corrective measures.
(4) Program Critique Summaries.
(5) Recordkeeping. Do records consistently demonstrate qualification of crews and instructor/evaluators?
(6) I&O Plan Adherence.
(7) Modifications to the Program.
(a) Due to PPDB input.
Par 2-23 Page 37
(b) Due to other input.
1 Demographics.
2 Operational.
(8) Validity and Usefulness of Qualification Standards.
(9) AQP Maintenance Strategy - Is the Described Process Working?
(a) Any changes to the maintenance strategy.
(b) Currency of PPDB.
(10) Instructor Evaluator Programs.
(a) Inter/Referent rater reliability data.
(b) Question and answer observations summarized.
(c) Methods to maintain I/E Standardization.
(11) FAA Surveillance Findings.
(12) AQP Challenges and Difficulties.
(a) Progress towards phase III, IV, and V in other fleets.
(b) Special tracking.
(c) Seat substitution.
(d) Line oriented flight training (LOFT)/LOS scenarios for crewmembers.
(13) Use of Information from Related Programs, If Any (e.g., Flight Operational Quality Assurance (FOQA), Aviation Safety Action Prevention (ASAP), etc).
Page 38 Par 2-23 FIGURE 2-8.
DOCUMENT STRUCTURES CHART
Title List of Effective Pages
Section I - All Organizational Standardized Area’s Application Instructional Systems Development Methodology Implementation and Operations Plan
Section II - Indoctrination Training Curriculums Curriculum Outline
Section III - Qualification Training Curriculums Task Analysis Qualification Standards Curriculum Outline Data Acquisition/Grading Forms
Section IV - Continuing Qualification Training Curriculums Curriculum Outline Data Acquisition/Grading Forms
Par 3-1 Page 39 CHAPTER 3. AQP CURRICULUMS AND CERTIFICATION
SECTION 1. CURRICULUMS
3-1. BACKGROUND. The Advanced Qualification Program (AQP) requires three primary curriculums for each make, model, and series of aircraft (or variant), and each duty position.
These curriculums are indoctrination, qualification, and continuing qualification. Figure 3-5 illustrates how traditional curriculums correlate to the AQP primary curriculums. In addition to primary curriculums, operational necessities may require secondary curriculums to fulfill particular needs. Secondary curriculums are transition, upgrade, and requalification (see Figure 3-6).
3-2. INDOCTRINATION CURRICULUM. An indoctrination curriculum consists of all training elements that will be learned and evaluated before an individual may begin a qualification curriculum. Indoctrination curriculum segments typically consist of ground training and evaluation. Two distinct areas of indoctrination ground training are:
a. Certificate Holder-Specific Training. This training acquaints crewmembers, dispatchers, instructors, evaluators, and other operations personnel with company policies, practices, and general operational knowledge. The subject matter of indoctrination includes elements that pertain to the certificate holder’s methods of compliance with regulations and safe operating practices.
b. Duty Position-Specific Training. This training provides the basic aeronautical knowledge needed to enter the subsequent qualification curriculum. Weather, the regulations, security, some emergency, and hazardous material training (usually part of indoctrination) is partially certificate holder-specific and partially duty position-specific.
3-3. QUALIFICATION CURRICULUM. AQP requires a qualification curriculum for each duty position in each make, model, and series aircraft (or variant). Each Qualification Curriculum will include training, validation, and evaluation. The training activities include ground and flight training, operational experience, and may include special qualification.
Figure 3-1 illustrates the relationship between training activities and validation points. If the training is to result in airman certification or the addition of category, class, instrument, or type ratings, curriculum segments must explicitly identify the training and evaluation strategy to be used in place of the prescribed practical test requirements of Title 14 of the Code of Federal Regulations (14 CFR) parts 61, 63, or 65. The applicant must show, to the FAA’s satisfaction, that the proposed AQP training and evaluation strategy will ensure individual competency that equals or exceeds the practical test standards (PTS) and that each person certificated through an AQP has demonstrated satisfactory proficiency in the integration of technical and crew resource management (CRM) skills.
a. Training Activities.
(1) Ground Qualification Training Activities. To be qualified for a particular duty position, a person will receive job-specific ground training. This training typically includes
Page 40 Par 3-3 general operational subjects, technical systems, system and procedures integration, and emergency-type training. Ground qualification culminates in a systems knowledge validation session that may take the form of a traditional oral examination or may incorporate another means of systems knowledge validation (e.g., computer-based, written, etc), as approved by the FAA.
(2) Simulation/Flight Qualification Training Activities. Each AQP includes curriculum segments for training, validation, or evaluations, as appropriate in Flight Training Devices (FTD) and simulators. The use of FTDs for training and evaluation is encouraged.
However, the maneuvers validation must be accomplished in a full flight simulator that has been approved for its intended use. Training and evaluation in an aircraft is discouraged, but may be approved by the FAA on a case-by-case basis.
(3) Special Qualification Training. Curriculum segments may include special purpose training. These are portions of ground or flight training that has specific application to crewmembers who are in international operations or for introduction of new flight operations, such as Category III (CAT III) approaches. Special purpose training and validation may initially be a separate curriculum segment that is later integrated into ground and flight training segments.
b. Validation/Evaluation/Remediation. In AQP, validation is a determination that the training produces the required results as identified in the qualification standards and that the individual has met the performance objectives of the training module. Except for the Qualification Curriculum Maneuvers Validation session, additional training may occur during a validation session to ensure achieving the training-to-proficiency objectives. An evaluation is an appraisal of an individual to ascertain whether the standards required for a specified level of proficiency has been successfully demonstrated. Interrupting the evaluation session for training is not permitted. Both validation and evaluation are assessments that the proficiency objectives of the training module have been met and the individual can proceed to the next level of training or line operations. Figure 3-2 contains a table that summarizes the validation/evaluation gates and the associated remediation.
(1) Systems Knowledge Validation. This is an assessment of an individual’s technical systems knowledge. The intent of the systems knowledge validation session is to ensure an individual’s systems knowledge is at an appropriate level before progressing into the next training phase. Systems knowledge validation may be accomplished by a written, electronic or oral exam. If a written or electronic test system is used, an overall score of 80 percent or better, corrected to 100 percent, would be an acceptable validation. An overall score of less than 80 percent will require retraining and another complete test. A failure of an individual test module or sub-section, with an 80 percent or better overall score, only requires retraining and retesting of the specific module. Consideration should be given to establishing a maximum number of modules or sub-sections that if failed constitute an overall failure of the validation.
(2) Procedures Validation. This is an assessment of an individual’s systems integration knowledge and skill. This validation addresses the individual’s ability to assimilate system and procedural knowledge into the appropriate execution of procedures. This validation session should take place in an FTD or a simulator. The intent of the procedures validation session is to
Par 3-3 Page 41 ensure an individual’s systems and procedural knowledge and skill is at an appropriate level before progressing into the flight simulator training phase. There is no established requirement that the systems and procedures validation be accomplished sequentially. It is the prerogative of each certificate holder to develop an evaluation system that works for them. For example, the systems validation could take place in two parts. The first part may be a written exam administered at the conclusion of ground school. The second stage could be an oral conducted at the conclusion of procedures training, before the maneuvers validation or before the line operational evaluation (LOE). Validation is accomplished when it is verified that the individual is trained to proficiency.
(3) Maneuvers Validation. This validation addresses the individual’s proficiency in the execution of maneuvers. It must take place in a simulator. For a qualification curriculum, crewmembers are expected to have reached a satisfactory level of proficiency in the maneuvers prior to the validation event. Qualification maneuvers validation should not allow more than two repeats of any one maneuver or one repeat of any two maneuvers. A debriefing of why the maneuver(s) was unsatisfactory is allowed, but the repeats must occur with no training, practice, or coaching. If the crewmember fails to demonstrate proficiency in the time constraints of the simulator session, an additional training session is required. After additional training, the individual need only repeat the maneuvers that were unsatisfactory. Failure of the maneuvers validation or failure to complete the maneuvers validation during the scheduled simulator period does not result in the issuance of FAA Form 8060-5, Notice of Disapproval of Application.
(4) LOE. This evaluation addresses the individual’s ability to demonstrate technical and CRM skills appropriate to fulfilling job requirements in a full mission scenario environment. The intent of an LOE is to evaluate and verify that an individual’s job knowledge, technical skills, and CRM skills are commensurate with AQP qualification standards and that the individual is qualified to begin the Operating Experience (OE) portion of the qualification curriculum. LOE is considered a jeopardy event and a failure is reported to the FAA. If the LOE is to result in certification, a failure will result in the issuance of FAA Form 8060-5. Grading of an LOE is at the event set level. LOE success criteria is listed below:
(a) An LOE with more than 25 percent of the event sets graded unsuccessful would constitute a failure of the LOE and will require remedial training and another LOE.
(b) For LOEs with 25 percent of the event sets or fewer graded unsuccessful, repeats may be permitted at the end of the LOE session, if time permits and if it is possible to recreate the conditions similar to the original event set for the repeat. For example, if an LOE has five, six, or seven event sets, only one set may be repeated. If an LOE has between 8 and 11 event sets, two may be repeated. No event set can be repeated more than once. A debriefing of why the event set(s) is unsatisfactory is allowed, but the repeat must occur with no training, practice, or coaching. If any repeated event is unsatisfactory, remedial training and another complete LOE evaluation is required.
NOTE: Regardless of number of events sets, unsafe individual or crew performance that would result in signific ant damage, hull loss, or loss of life
(e.g., crash) during an LOE constitutes a failure of the LOE.
Page 42 Par 3-3
NOTE: An aircrew program design ee (APD) or FAA inspector must administer all flight crewmember qualif ication curriculum LOEs. A check airman can administer continuing qualification LOEs.
(5) Operating Experience (OE). OE curriculum segments are integral to qualification curriculums. OE provides hands-on experience in performing the duties of a newly assigned position under the supervision of a current and qualified evaluator (check airman). OE is conducted during actual flight operations. Operating experience is complete when the individual is recommended for a line check.
(6) Line Check. Captains receiving this evaluation are assessed for their proficiency in the duty position. Successful completion of the line check verifies that the individual is adequately trained and is capable of performing the duties and responsibilities of pilot-incommand (PIC). If any task is unsatisfactory, the individual must receive remedial training on that task, additional operating experience if necessary, and possibly another line check. If a pilot receives an unsatisfactory overall performance rating on a line check, the pilot must be removed from continued line operations until the approved remediation has been successfully completed.
c. Planned Hours. All curriculums will include planned hours for ground training, flight training, evaluation, and operating experience. Planned hours represent the estimated amount of time (as specified in a curriculum outline) that it takes an average student to complete a segment of instruction (to include all instruction, demonstration, practice, and evaluation, as appropriate, to reach proficiency). Planned hours replace the programmed hours associated with traditional programs. Planned hours enable the FAA and the applicant to schedule their personnel resources more efficiently and provide a baseline for curriculum adjustments. The sum of planned hours is not used by the FAA as a basis for program approval, review, or compliance assessment.
Planned hours can be shown on the course footprint that is part of the curriculum outline.
Figure 3-3 provides an example of a course footprint with planned hours.
Par 3-3 Page 43
FIGURE 3-1.
FLIGHT CREWMEMBER AQP TRAINING, VALIDATION, AND EVALUATION
Activity
Device
Purpose
Can it be Interrupted?
Training/ Validation or Evaluation
Sequencing of Events
Ground Training
Classroom or Computer Based Instruction, and Training Devices
Indoctrination and Systems Training
Yes
Training & Validation
Syllabus Maneuvers and Procedures Training
Flight Training Device & Simulator
Aircraft Maneuvers and Operational Procedures
Yes
Training & Validation
Isolated Maneuvers and Procedures.
Logical Specific Sequencing of Events Special Purpose Operational Training (SPOT)
Flight Training Device & Simulator Focuses on CRM Skills, Differences Tng., Windshear Tng., Special Qualification
Yes
Training
Isolated Maneuvers and Procedures.
Logical Specific Sequencing of Events Line Oriented Flight Training (LOFT)
Flight Training Device & Simulator
Crew Oriented Training (CRM) in Preparation for LOE No
Except To Begin Different Scenarios
Training
Logical Sequencing of Events Within Varying Length Scenarios Line Operational Evaluation (LOE)
Flight Training Device & Simulator
Evaluate the training and qualifications of the crewmembers May Be Segmented to Condense Distances as in International Scenarios
Evaluation
Specific Flight Scenario From Takeoff to Landing
Operating Experience
Aircraft Consolidation of knowledge & skills in a operational environment
Yes
Experience
Routine Flight Operations
Initial Line Check
Aircraft To verify a crewmember’s ability to satisfactorily perform duties & responsibilities
No
Evaluation
Routine Flight Operations
CQ Line Check
Aircraft To evaluate crew proficiency and their knowledge, skills, and ability to operate effectively as part of a crew.
No
Evaluation
Routine Flight Operations
Page 44 Par 3-3 FIGURE 3-2.
AQP VALIDATION/EVALUATION TABLE
Gate Indoctrination Curriculum Qualification Curriculum
Continuing Qualification Curriculum Evaluation Media
System Knowledge Validation
80% or Greater corrected to 100%,
< 80% retest
80% or Greater corrected to 100%,
< 80% retest
80% or Greater corrected to 100%,
< 80% retest
Written, Oral, or Electronic Testing System (ETS)
Procedures Validation
Train to Proficiency
As Approved
Maneuvers Validation
2 Repeats of One Maneuver, or 1 Repeat Of Any 2 Maneuvers*
Repeats Allowed Within Training Period
Approved Simulation Device
LOE
Grading By Event Set 5, 6 Or 7 Event Sets - Repeat 1 Event Set
–8 or more Event Sets Repeat - 2 Event Sets **
Grading By Event Set 5, 6 or 7 Event Sets - Repeat 1 Event Set
8 –or more Event Sets Repeat - 2 Event Sets **
Approved Simulation Device
Operating Experience
Meets Approved Number of Supervised Cycles or Hours and Recommended for Initial Line Check
Aircraft (Actual Line Operations)
Line Check
If Any Task Unsatisfactory, Remedial Training Needed, Additional Operating Experience if Needed, and Another Line Check as Recommended by the Evaluator
If Any Task Unsatisfactory, Remedial Training Needed, Additional Operating Experience if Needed, and Another Line Check as Recommenced By the Evaluator
Aircraft (Actual Line Operations) * For any repeat in a qualification maneuvers validation, qualification LOE or CQ LOE, no training, practice, or coaching is permitted. Training repeats are allowed in a continuing qualification maneuvers validation and are not counted as an evaluation repeat.
** Failure to meet the required level of proficiency during an LOE or line check will result in retraining, reevaluation, and will require special tracking. Regardless of number of events sets, unsafe individual or crew performance that would result in significant damage, hull loss or loss of life (e.g., crash) during an LOE constitutes a failure of the LOE.
Par 3-3 Page 45 FIGURE 3-3.
EXAMPLE OF AQP FLIGHTCREW QUALIFICATION CURRICULUM FOOTPRINT
Day 1 Day 2 Day 3 Day 4 Day 5 Welcome :30 Intro :30 CBT 5:00
TT 6:00 CBT 5:00 Review 1.00
TT 6.00 CBT 5:00 Review 1.00
TT 6.00 CBT 5:00 Performance 1.00
TT 6.00 Evacuation 3:00 Ditching 3:00
TT 6:00
Day Off
Day Off Day 6 Day 7 Day 8 Day 9 Day 10 CBT 5:00 Setup Lect. 1.00
TT 6.00 CBT 3:00 FTD Brief 1.00 FTD # 1 2.00
TT 6.00 CBT 5:00 Review 1.00
TT 6.00 CBT 3:00 FTD Brief 1.00 FTD # 2 2.00
TT 6.00
SYSTM VAL Testing 1:00 Flt Ops Brf. 4:00 Debrief :30
TT 6:30
Day Off
Day Off Day 11 Day 12 Day 13 Day 14 Day 15 Sys Rev 1:00 FTD Brief 1.00 FTD # 4 4.00
TT 6.00 FTD Brief 1:30 FTD # 5 4.00 Debrief 1:00
TT 6.30 FTD Brief 1:30 FTD # 6 4.00 Debrief 1:00
TT 6.30 FTD Brief 1:30 FTD # 7 4.00 Debrief 1:00
TT 6.30 Proc Val FTD Brief 1:30 FTD # 8 4.00 Debrief 1:00
TT 6.30
Day Off
Day Off Day 16 Day 17 Day 18 Day 19 Day 20 Sim Brief 1:30 Sim # 1 4.00 Debrief 1:00
TT 6.30 Sim Brief 1:30 Sim # 2 4.00 Debrief 1:00
TT 6.30 Sim Brief 1:30 Sim # 3 4.00 Debrief 1:00
TT 6.30 Sim Brief 1:30 Sim # 4 4.00 Debrief 1:00
TT 6.30
MAN VAL Sim Brief 1:30 Sim # 5 4.00 Debrief 1:00
TT 6.30
Day Off
Day Off Day 21 Day 22 Day 23 Day 24 Day 25 Loft Brief 1:30 Loft # 1/Spot 4.00 Debrief 1:00
TT 6.30 Loft Brief 1:30 Loft # 2/Spot 4.00 Debrief 1:00
TT 6.30 Loft Brief 1:30 Loft # 3/Spot 4.00 Debrief 1:00
TT 6.30 Loft Brief 1:30 Loft # 4/Spot 4.00 Debrief 1:00
TT 6.30 LOE Brief 1:30 LOE 4:00 Debrief 1:00
TT 6:30
Day Off
Day Off Day 26 Day 27 Day 28 Day 29 Day 30
IOE
IOE
IOE
IOE
IOE
Day 31
IOE Day 32
Line Check SYST VAL - Systems Validation PROC VAL - Procedures Validation MAN VAL - Maneuvers Validation LOE - Line Operational Evaluation Spot - Special Purpose Operations Training CBT – Computer-based training FTD - Flight training device Sim - Full flight simulator LOFT - Line oriented flight training IOE - Initial operations experience
3-4. CONTINUING QUALIFICATION CURRICULUM. A continuing qualification curriculum provides the means for fully qualified individuals to maintain their proficiency in their duty positions and aircraft assignments. Continuing qualification applies to all persons subject to an AQP, including instructors and evaluators. AQP requires a continuing qualification curriculum for each duty position in each make, model, and series aircraft (or variant).
a. First-Look Maneuvers. For AQP, first-look maneuvers are applicable only to the continuing qualification curriculum and the single visit training program (SVTP). First-look