Page 32 Par 2-12
1 The type of software data management system employed (e.g., relational database, spreadsheet etc.).
2 The organization of the information in the electronic medium (e.g., database definition, database table relationships, spreadsheet description, etc.).
3 A description of the user interface to this data management system.
(c) Data Analysis. Discuss the type of analysis that will be employed to facilitate the AQP performance information needs of the airline and the FAA. This discussion of the data analysis must address how each type of AQP data will be analyzed, including training and evaluation feedback as part of determining the effectiveness of the program. This discussion should be used as a prologue to the annual AQP report.
(d) Data Reporting. Discuss the FAA AQP data reporting requirements that must be met (to include format and frequency). In addition, discuss the internal quality assurance data reporting that will be employed, to include report types, frequency, and for which airline personnel the reports are intended.
2-13. APPROVAL. Approval of above documents marks the end of phase II and the beginning of phase III.
SECTION 4. PHASE III: SMALL GROUP TRYOUT (IMPLEMENTATION)
2-14. OVERVIEW. In phase III, the AQP applicant will acquire and test the resources essential to support one or more of the indoctrination, qualification, or continuing qualification curriculums. These activities include developing courseware to implement the curriculum(s), training instructors and evaluators, conducting small group tryouts, making program revisions, and submitting data.
a. Resources. During this phase, the applicant will secure the training facility and equip classrooms with training aids, courseware, and other features that contribute to creating and maintaining a positive learning environment.
b. Train Instructors and Evaluators. The applicant must train, evaluate, qualify and provide an opportunity for the FAA to observe, if necessary, their instructors and evaluators before and during the small group tryouts (refer to Chapter 5 for more information on the training, observation, and quality control of instructors and evaluators).
c. Conduct the Approved Curriculum. This training and evaluation will consist of small group tryouts of all lessons using actual students and instructor/evaluators. The evaluation may involve no-jeopardy credit for students, since its primary purpose is to determine lesson suitability and effectiveness. The applicant may choose, however, to give student credit for part or all training and qualification achieved in this phase. The decision to give credit must be
Par 2-14 Page 33 approved by the FAA before conducting the curriculum evaluation and the air carrier’s request to do so should be submitted by letter to the FAA VSP manager and the POI.
d. Performance/Proficiency Data Submission. The applicant will submit to the FAA VSP manager, on a monthly basis, the required performance and proficiency data. This data will be in electronic, digitized format. Any changes to format or procedures will be discussed with the FAA VSP manager prior to submission. The monthly submission of this performance/proficiency data will continue for the life of the curriculum’s lifetime (i.e., through phases III, IV, and V). See Chapter 8 for more information.
e. Program Revisions. Lessons learned as a result of this phase will be incorporated as changes to the approved AQP documents from phase I and II and in curriculum maintenance reports submitted to the FAA on an annual basis. Improvements will be made to the courseware, hardware, software, personnel, organization, and data collection and reporting system of the AQP, based on system performance and the examination of performance and proficiency data.
These enhancements will be implemented using the processes and the procedures described in the applicant’s approved AQP maintenance strategy. This plan is part of the I&O Plan.
2-15. APPROVAL. Approval of the initial operations marks the end of phase III and the beginning of phase IV.
SECTION 5. PHASE IV: INITIAL OPERATIONS
2-16. OVERVIEW. In this phase, the applicant implements the indoctrination, qualification, or continuing qualification curriculum in accordance with the approved AQP documents. This phase for the continuing qualification curriculum calls for initial operation of the curriculum for a minimum of 2 years to complete the AQP training cycle. The 2-year phase IV period may not necessarily be applicable to the indoctrination and qualification curriculums. Unlike the Continuing Qualification curriculums, these courses of training are not cycle-dependent.
Depending on the frequency that each curriculum is applied (new hires) and the successful results of each curriculum, phase IV may be longer or shorter than 24 months.
2-17. PHASE IV ACTIVITIES. During phase IV, the applicant will implement and complete a full evaluation, including collecting program audit data and individual performance/proficiency data, analyses, and reports. Collected data will be used by:
a. The applicant for its internal quality control program to maintain curriculum and courseware concurrence, suitability, and adequacy.
b. The applicant to analyze and validate crewmembers, dispatchers, and other operations personnel performance.
c. The applicant and the FAA to analyze and validate instructor/evaluators performance.
Page 34 Par 2-17
d. The applicant and the FAA to support analysis for special subjects, such as CRM performance factors.
e. The FAA to analyze and validate curriculum performance.
f. The FAA to analyze and validate program development, implementation, and maintenance procedures.
2-18. REVISIONS. The focus of this phase is the validation of the AQP curriculum by acquiring performance/proficiency data and lessons learned in the course of conducting the curriculum. Applicants will summarize the lessons learned and adjustments made to the curriculums in an annual report. In addition, adjustments made to the AQP will be reflected in revisions to the approved AQP documents. The successful completion of this phase, and approval of the commensurate revisions to the approved AQP documents and reports, will qualify an applicant for entry into the final phase of the AQP process: Phase V, Continuing Operations.
2-19. APPROVAL. Approval of the continuing operations marks the end of phase IV and the entrance into phase V.
SECTION 6. PHASE V: CONTINUING OPERATIONS
2-20. OVERVIEW. In this phase, the applicant continues operation of the AQP unless the FAA withdraws approval or unless the applicant withdraws or modifies the AQP. This phase requires the maintenance of the AQP-approved documentation and updated documentation of the data requirements for all curriculums.
2-21. DATA’S IMPACT ON CONTINUING OPERATIONS. Data will continue to be collected and analyzed by the applicant and the FAA as it was during phase IV activities.
2-22. QUALITY ASSURANCE. Applicants must pay particular attention to overall program quality. The FAA expects an AQP quality assurance program to identify needed changes in curriculum, courseware, and equipment, and to make these changes before an unwanted trend in reduced proficiency manifests itself. Continued validation of performance/proficiency data as it pertains to individual and crew proficiency, as achieved and maintained by all personnel, is particularly important.
SECTION 7. DOCUMENT AND REPORTING REQUIREMENTS
2-23. OVERVIEW. The approved AQP documentation establishes the applicant’s regulatory requirements for the particular training program. This fact makes it imperative that the applicant develops a document structure that is organized and standardized. The structure must ensure that the information contained within the AQP documents will be used and translated down to the lowest level training guidance. It must allow the applicant’s entire organization to easily access
Par 2-23 Page 35 and use the contents. It is important to draw a distinction between the AQP process and the required FAA documentation each applicant must provide to ensure regulatory compliance. The FAA has established a minimum listing of AQP documentation. The applicant will want to develop other more specific materials that ensure the approved AQP information distills to all training and evaluation guides.
a. FAA-Required Documentation. Each document contains unique information integral to the AQP that will be used when developing subsequent curriculums. The documents, once developed, will require periodic updates and, therefore, are subject to the revision control process discussed in Chapter 4. These documents should be submitted to the FAA in electronic format:
(1) The application.
(2) The task analysis.
(3) The qualification standards.
(4) The instructional systems development methodology.
(5) The curriculum outlines.
(6) The I&O Plan.
b. Document Structures. Figure 2-8 provides an example of an AQP document structure.
It is not required that an applicant follow this structure. However, the document structure that is adopted must easily identify the location of the AQP documents and more specifically, the required AQP information. If the AQP documentation is part of a subset of other manuals, a method should be developed to identify the manual and the specific AQP document that it contains.
c. Annual AQP Report. AQP requires that each AQP certificate holder prepare an annual report for the FAA (see Appendix C for specific requirements). This report is based on the certificate holder’s analysis of the data that is collected during training and at strategic points (validation/evaluation gates) in each curriculum and maintained in the PPDB. AQP requires data collection and analysis in order to establish and maintain quality control of curriculums for crewmembers, instructors, and evaluators. The annual AQP report should summarize the lessons learned and adjustments made to the curriculum(s) during the reporting period. The report should also include projected or proposed changes to the curriculum(s) based on the certificate holder’s current analysis. The actual adjustments made to the AQP are reflected in revisions to the approved AQP documents. The report should be submitted to the FAA no later than 60 days past the end of the report period. The reporting period is usually based on the approval date for a particular curriculum in either phase IV or V. During AQP development, particularly for multiple fleet operators, with different approval dates for multiple curriculums, the reporting period may be modified (as agreed upon by the FAA and the certificate holder). Once the certificate holder has all its fleets and curriculums into phase V, the reporting period can be fixed
Page 36 Par 2-23 into a particular cycle. Copies of the report should be distributed to the POI and AFS-230 at least 2 weeks prior to the annual AQP review meeting.
d. Annual AQP Review - Phases IV and V. An annual AQP review meeting between the FAA and the certificate holder should be held to coincide with the submission of the annual AQP report in phases IV and V. The purpose of the meeting is to formally discuss the results of the certificate holder’s data analysis, program revisions, future revisions and the analysis of the data that is submitted to the FAA. Below is a listing of suggested agenda items for a Phase IV and V review:
(1) Data Management.
(a) Collection problems and fixes.
(b) Analysis - data reliability/validity/sensitivity.
(c) Data usefulness.
(d) Problem areas investigated.
(2) Data Collection Method/Tools.
(3) Data Analysis Methods/Tools.
(a) Review of annual report.
1 Qualification.
2 Continuing qualification.
3 Line check.
(b) Identified trends (positive and negative).
(c) Corrective measures.
(4) Program Critique Summaries.
(5) Recordkeeping. Do records consistently demonstrate qualification of crews and instructor/evaluators?
(6) I&O Plan Adherence.
(7) Modifications to the Program.
(a) Due to PPDB input.
Par 2-23 Page 37
(b) Due to other input.
1 Demographics.
2 Operational.
(8) Validity and Usefulness of Qualification Standards.
(9) AQP Maintenance Strategy - Is the Described Process Working?
(a) Any changes to the maintenance strategy.
(b) Currency of PPDB.
(10) Instructor Evaluator Programs.
(a) Inter/Referent rater reliability data.
(b) Question and answer observations summarized.
(c) Methods to maintain I/E Standardization.
(11) FAA Surveillance Findings.
(12) AQP Challenges and Difficulties.
(a) Progress towards phase III, IV, and V in other fleets.
(b) Special tracking.
(c) Seat substitution.
(d) Line oriented flight training (LOFT)/LOS scenarios for crewmembers.
(13) Use of Information from Related Programs, If Any (e.g., Flight Operational Quality Assurance (FOQA), Aviation Safety Action Prevention (ASAP), etc).
Page 38 Par 2-23 FIGURE 2-8.
DOCUMENT STRUCTURES CHART
Title List of Effective Pages
Section I - All Organizational Standardized Area’s Application Instructional Systems Development Methodology Implementation and Operations Plan
Section II - Indoctrination Training Curriculums Curriculum Outline
Section III - Qualification Training Curriculums Task Analysis Qualification Standards Curriculum Outline Data Acquisition/Grading Forms
Section IV - Continuing Qualification Training Curriculums Curriculum Outline Data Acquisition/Grading Forms
Par 3-1 Page 39 CHAPTER 3. AQP CURRICULUMS AND CERTIFICATION
SECTION 1. CURRICULUMS
3-1. BACKGROUND. The Advanced Qualification Program (AQP) requires three primary curriculums for each make, model, and series of aircraft (or variant), and each duty position.
These curriculums are indoctrination, qualification, and continuing qualification. Figure 3-5 illustrates how traditional curriculums correlate to the AQP primary curriculums. In addition to primary curriculums, operational necessities may require secondary curriculums to fulfill particular needs. Secondary curriculums are transition, upgrade, and requalification (see Figure 3-6).
3-2. INDOCTRINATION CURRICULUM. An indoctrination curriculum consists of all training elements that will be learned and evaluated before an individual may begin a qualification curriculum. Indoctrination curriculum segments typically consist of ground training and evaluation. Two distinct areas of indoctrination ground training are:
a. Certificate Holder-Specific Training. This training acquaints crewmembers, dispatchers, instructors, evaluators, and other operations personnel with company policies, practices, and general operational knowledge. The subject matter of indoctrination includes elements that pertain to the certificate holder’s methods of compliance with regulations and safe operating practices.
b. Duty Position-Specific Training. This training provides the basic aeronautical knowledge needed to enter the subsequent qualification curriculum. Weather, the regulations, security, some emergency, and hazardous material training (usually part of indoctrination) is partially certificate holder-specific and partially duty position-specific.
3-3. QUALIFICATION CURRICULUM. AQP requires a qualification curriculum for each duty position in each make, model, and series aircraft (or variant). Each Qualification Curriculum will include training, validation, and evaluation. The training activities include ground and flight training, operational experience, and may include special qualification.
Figure 3-1 illustrates the relationship between training activities and validation points. If the training is to result in airman certification or the addition of category, class, instrument, or type ratings, curriculum segments must explicitly identify the training and evaluation strategy to be used in place of the prescribed practical test requirements of Title 14 of the Code of Federal Regulations (14 CFR) parts 61, 63, or 65. The applicant must show, to the FAA’s satisfaction, that the proposed AQP training and evaluation strategy will ensure individual competency that equals or exceeds the practical test standards (PTS) and that each person certificated through an AQP has demonstrated satisfactory proficiency in the integration of technical and crew resource management (CRM) skills.
a. Training Activities.
(1) Ground Qualification Training Activities. To be qualified for a particular duty position, a person will receive job-specific ground training. This training typically includes
Page 40 Par 3-3 general operational subjects, technical systems, system and procedures integration, and emergency-type training. Ground qualification culminates in a systems knowledge validation session that may take the form of a traditional oral examination or may incorporate another means of systems knowledge validation (e.g., computer-based, written, etc), as approved by the FAA.
(2) Simulation/Flight Qualification Training Activities. Each AQP includes curriculum segments for training, validation, or evaluations, as appropriate in Flight Training Devices (FTD) and simulators. The use of FTDs for training and evaluation is encouraged.
However, the maneuvers validation must be accomplished in a full flight simulator that has been approved for its intended use. Training and evaluation in an aircraft is discouraged, but may be approved by the FAA on a case-by-case basis.
(3) Special Qualification Training. Curriculum segments may include special purpose training. These are portions of ground or flight training that has specific application to crewmembers who are in international operations or for introduction of new flight operations, such as Category III (CAT III) approaches. Special purpose training and validation may initially be a separate curriculum segment that is later integrated into ground and flight training segments.
b. Validation/Evaluation/Remediation. In AQP, validation is a determination that the training produces the required results as identified in the qualification standards and that the individual has met the performance objectives of the training module. Except for the Qualification Curriculum Maneuvers Validation session, additional training may occur during a validation session to ensure achieving the training-to-proficiency objectives. An evaluation is an appraisal of an individual to ascertain whether the standards required for a specified level of proficiency has been successfully demonstrated. Interrupting the evaluation session for training is not permitted. Both validation and evaluation are assessments that the proficiency objectives of the training module have been met and the individual can proceed to the next level of training or line operations. Figure 3-2 contains a table that summarizes the validation/evaluation gates and the associated remediation.
(1) Systems Knowledge Validation. This is an assessment of an individual’s technical systems knowledge. The intent of the systems knowledge validation session is to ensure an individual’s systems knowledge is at an appropriate level before progressing into the next training phase. Systems knowledge validation may be accomplished by a written, electronic or oral exam. If a written or electronic test system is used, an overall score of 80 percent or better, corrected to 100 percent, would be an acceptable validation. An overall score of less than 80 percent will require retraining and another complete test. A failure of an individual test module or sub-section, with an 80 percent or better overall score, only requires retraining and retesting of the specific module. Consideration should be given to establishing a maximum number of modules or sub-sections that if failed constitute an overall failure of the validation.
(2) Procedures Validation. This is an assessment of an individual’s systems integration knowledge and skill. This validation addresses the individual’s ability to assimilate system and procedural knowledge into the appropriate execution of procedures. This validation session should take place in an FTD or a simulator. The intent of the procedures validation session is to
Par 3-3 Page 41 ensure an individual’s systems and procedural knowledge and skill is at an appropriate level before progressing into the flight simulator training phase. There is no established requirement that the systems and procedures validation be accomplished sequentially. It is the prerogative of each certificate holder to develop an evaluation system that works for them. For example, the systems validation could take place in two parts. The first part may be a written exam administered at the conclusion of ground school. The second stage could be an oral conducted at the conclusion of procedures training, before the maneuvers validation or before the line operational evaluation (LOE). Validation is accomplished when it is verified that the individual is trained to proficiency.
(3) Maneuvers Validation. This validation addresses the individual’s proficiency in the execution of maneuvers. It must take place in a simulator. For a qualification curriculum, crewmembers are expected to have reached a satisfactory level of proficiency in the maneuvers prior to the validation event. Qualification maneuvers validation should not allow more than two repeats of any one maneuver or one repeat of any two maneuvers. A debriefing of why the maneuver(s) was unsatisfactory is allowed, but the repeats must occur with no training, practice, or coaching. If the crewmember fails to demonstrate proficiency in the time constraints of the simulator session, an additional training session is required. After additional training, the individual need only repeat the maneuvers that were unsatisfactory. Failure of the maneuvers validation or failure to complete the maneuvers validation during the scheduled simulator period does not result in the issuance of FAA Form 8060-5, Notice of Disapproval of Application.
(4) LOE. This evaluation addresses the individual’s ability to demonstrate technical and CRM skills appropriate to fulfilling job requirements in a full mission scenario environment. The intent of an LOE is to evaluate and verify that an individual’s job knowledge, technical skills, and CRM skills are commensurate with AQP qualification standards and that the individual is qualified to begin the Operating Experience (OE) portion of the qualification curriculum. LOE is considered a jeopardy event and a failure is reported to the FAA. If the LOE is to result in certification, a failure will result in the issuance of FAA Form 8060-5. Grading of an LOE is at the event set level. LOE success criteria is listed below:
(a) An LOE with more than 25 percent of the event sets graded unsuccessful would constitute a failure of the LOE and will require remedial training and another LOE.
(b) For LOEs with 25 percent of the event sets or fewer graded unsuccessful, repeats may be permitted at the end of the LOE session, if time permits and if it is possible to recreate the conditions similar to the original event set for the repeat. For example, if an LOE has five, six, or seven event sets, only one set may be repeated. If an LOE has between 8 and 11 event sets, two may be repeated. No event set can be repeated more than once. A debriefing of why the event set(s) is unsatisfactory is allowed, but the repeat must occur with no training, practice, or coaching. If any repeated event is unsatisfactory, remedial training and another complete LOE evaluation is required.
NOTE: Regardless of number of events sets, unsafe individual or crew performance that would result in signific ant damage, hull loss, or loss of life
(e.g., crash) during an LOE constitutes a failure of the LOE.
Page 42 Par 3-3
NOTE: An aircrew program design ee (APD) or FAA inspector must administer all flight crewmember qualif ication curriculum LOEs. A check airman can administer continuing qualification LOEs.
(5) Operating Experience (OE). OE curriculum segments are integral to qualification curriculums. OE provides hands-on experience in performing the duties of a newly assigned position under the supervision of a current and qualified evaluator (check airman). OE is conducted during actual flight operations. Operating experience is complete when the individual is recommended for a line check.
(6) Line Check. Captains receiving this evaluation are assessed for their proficiency in the duty position. Successful completion of the line check verifies that the individual is adequately trained and is capable of performing the duties and responsibilities of pilot-incommand (PIC). If any task is unsatisfactory, the individual must receive remedial training on that task, additional operating experience if necessary, and possibly another line check. If a pilot receives an unsatisfactory overall performance rating on a line check, the pilot must be removed from continued line operations until the approved remediation has been successfully completed.
c. Planned Hours. All curriculums will include planned hours for ground training, flight training, evaluation, and operating experience. Planned hours represent the estimated amount of time (as specified in a curriculum outline) that it takes an average student to complete a segment of instruction (to include all instruction, demonstration, practice, and evaluation, as appropriate, to reach proficiency). Planned hours replace the programmed hours associated with traditional programs. Planned hours enable the FAA and the applicant to schedule their personnel resources more efficiently and provide a baseline for curriculum adjustments. The sum of planned hours is not used by the FAA as a basis for program approval, review, or compliance assessment.
Planned hours can be shown on the course footprint that is part of the curriculum outline.
Figure 3-3 provides an example of a course footprint with planned hours.
Par 3-3 Page 43
FIGURE 3-1.
FLIGHT CREWMEMBER AQP TRAINING, VALIDATION, AND EVALUATION
Activity
Device
Purpose
Can it be Interrupted?
Training/ Validation or Evaluation
Sequencing of Events
Ground Training
Classroom or Computer Based Instruction, and Training Devices
Indoctrination and Systems Training
Yes
Training & Validation
Syllabus Maneuvers and Procedures Training
Flight Training Device & Simulator
Aircraft Maneuvers and Operational Procedures
Yes
Training & Validation
Isolated Maneuvers and Procedures.
Logical Specific Sequencing of Events Special Purpose Operational Training (SPOT)
Flight Training Device & Simulator Focuses on CRM Skills, Differences Tng., Windshear Tng., Special Qualification
Yes
Training
Isolated Maneuvers and Procedures.
Logical Specific Sequencing of Events Line Oriented Flight Training (LOFT)
Flight Training Device & Simulator
Crew Oriented Training (CRM) in Preparation for LOE No
Except To Begin Different Scenarios
Training
Logical Sequencing of Events Within Varying Length Scenarios Line Operational Evaluation (LOE)
Flight Training Device & Simulator
Evaluate the training and qualifications of the crewmembers May Be Segmented to Condense Distances as in International Scenarios
Evaluation
Specific Flight Scenario From Takeoff to Landing
Operating Experience
Aircraft Consolidation of knowledge & skills in a operational environment
Yes
Experience
Routine Flight Operations
Initial Line Check
Aircraft To verify a crewmember’s ability to satisfactorily perform duties & responsibilities
No
Evaluation
Routine Flight Operations
CQ Line Check
Aircraft To evaluate crew proficiency and their knowledge, skills, and ability to operate effectively as part of a crew.
No
Evaluation
Routine Flight Operations