Appendix 4 Y N Comments
l. Does the applicant describe the level of training devices, simulators, or aircraft to be used to evaluate the proficiency objective at each point in the curriculum?
m. Does the remediation strategy specify when no more training will be offered to the individual and the resulting actions such as “Referred to Director of Training,” “returned to previous position,” etc.
n. Does the remediation strategy describe the criteria for placing an individual on special tracking?
o. Does the remediation strategy describe the strategy that will be used for an individual on special tracking?
p. Does the remediation strategy describe what must take place for an individual to be removed from special tracking?
4 Qualification Standards Y N Comments
a. Does each individual qualification standard contain the following:
A header identifying the airline and the document?
Page revision control dates and revision numbers?
Consecutive page numbers?
Phase of Operations: Number and title from task listing?
Qualification Standard Title: Either TPOs or SPOs?
Task or Subtask: Number and title from task listing?
Crew duty positions?
Criticality/Currency Rating: From the task factors analysis?
Curriculum: This field identifies the curriculum(s) in which the task will be trained and evaluated.
Evaluation Strategy: The evaluation point for this particular qualification standard:
e.g., train to proficiency, procedures validation, maneuvers validation, LOE, or line check?
Media: The specific media in which training and/or evaluation will be conducted. For qualification, the media is the lowest media used for final evaluation. For continuing qualification, the media is the range of media used for training.
Appendix 4 Y N Comments Performance Statement: An expanded statement of expected behavior, which, when executed, will complete the work required for a specific portion of a job?
Conditions operational and environmental?
Are the specific conditions to be used for the qualification curriculum specified?
Contingencies. Are the specific contingencies to be used for the qualification curriculum specified?
Maneuver Standards: Are they specific and do they correspond to the standards listed in the applicable Airman Certification Standards or practical test standards?
Procedural Standards: They may be specific or general. If they are specific, do they correspond to the standards listed in the manuals listed in the reference block? If general, do they reference information in a document or manual to chapter or section (page numbers are not required)?
References: Identify the primary references from which performance statements and associated standards were derived. Cite documents by title and where applicable, chapter or section. Page numbers are not required.
b. Are there any operations specifications requirements other than those listed above?
Appendix 4 The Instructional Systems Development Methodology document describes the approach to be used by applicant airlines to develop and maintain all Advanced Qualification Program (AQP) curriculums. This document is divided into two sections. The first section, Development Procedures, describes the applicant’s approach for using the Job Task Analyses and Qualification Standards as baseline documents to construct their general training curriculums across all AQP courses. The second section, Line Operational Simulation (LOS) Methodology, describes the approach for developing LOS scenarios.
1 Development Procedures Y N Comments
a. Is the procedure for allocating TPOs and SPOs to lessons, selecting media and methods, and developing the curriculums described?
b. Does the applicant describe how enabling objectives (EO) are developed to support their higher level objectives?
c. Does the applicant describe how learning and evaluation activities are developed to support these objectives?
d. Does the applicant describe how instructional media and methods to are assigned to objectives?
e. Does the applicant describe how objectives are clustered and sequenced into lessons, modules, segments, and curriculums?
f. Does the applicant describe how an audit trail will be maintained to link proficiency objectives, lesson activities/content, and test items?
2 Line Operational Simulation (LOS) Methodology Y N Comments
a. Does the applicant describe how the typical scenario is constructed?
b. Does the applicant describe how each event set relates to a phase of operation?
c. Does the applicant describe how each event set consists of a series of proficiency objectives that include both technical and CRM activities?
d. Does the applicant describe the use of event conditions, triggers, and distracters, as well as supporting events?
e. Does the applicant identify possible sources of incidents that will elicit the behavior required by the proficiency objectives selected for the i
f. Does the applicant define the basic success criteria for the LOS, and each event set within it?
g. Does the applicant describe the scenario development process?
Drafting – who will do the work?
Use of gradesheets?
Testing – who will be involved?
Training instructors/evaluators to administer LOS scenario?
Appendix 4 The curriculum outline is a listing of course content. It should be arranged from curriculum into segments, segments into modules, modules into lessons, and lessons into elements. Each part of the curriculum outline must clearly indicate the subject matter to be taught and correspond directly to the hierarchical system of the task analysis. A curriculum outline provides the basis for the curriculum footprint, which is a high level graphical overview of the curriculum content depicting the training and evaluation activities and the proposed hours for each day of the curriculum.
1 Curriculum Outline Y N Comments
a. Does the certificate holder have indoctrination, qualification, and continuing qualification curriculums for each duty position in each make, model, and series/variant aircraft?
b. Does the certificate holder have separate indoctrination, qualification, and continuing qualification curriculums for the instructors and evaluators?
c. Does the certificate holder have any special curriculums (transition, conversion, upgrade, requalification or refresher)?
d. Is each curriculum constructed in the following order: curriculum, segment, module, lesson, and lesson element?
e. Does the curriculum outline provide a level of detail that will allow the AQP applicant to make changes to the syllabus without submitting a new document for each syllabus change?
f. Does the curriculum outline include:
Operator’s Name?
Type of Aircraft?
Duty Position(s)?
Title of curriculum and/or curriculum segment?
A listing of numbered (coded) objectives organized into lessons, modules, and segments?
Numbers (codes) must allow the FAA to track objectives back to qualification standards and Job Task Analysis An outline of each training module within each curriculum segment?
Each module should contain sufficient detail to ensure that the main features of the principal elements or events will be addressed during instruction.
The checking and qualification modules of the qualification curriculum segment used to determine successful course completion, including any regulation qualification requirements for crewmembers to serve in part 121 or part 135 operations (such as operations familiarization (pilots), operating experience, line checks, operating familiarization (aircraft dispatchers))?
g. Does the curriculum outline indicate that it is part of the revision control system by page format?
Appendix 4 Y N Comments
h. Do the curriculum outlines provide a hierarchical link (proficiency objectives) between the qualification standards and a curriculum?
i. Does each part of the curriculum outline clearly indicate the subject matter to be taught and correspond directly to the hierarchical numbering system of the task analysis?
2 Curriculum Footprint Y N Comments
a. Does the curriculum footprint describe the training and evaluation activities conducted each day of the curriculum?
b. Does the curriculum footprint include the planned hours?
Appendix 4 Implementation and Operations Plan (I&O Plan). This document is a milestone schedule detailing the transition to an AQP for crewmembers, aircraft dispatchers, instructors, evaluators, and other operations personnel and a blueprint describing provisions for maintenance, administration, data management, and continuing quality control of curriculums. The I&O Plan can be sectioned into two parts. The first part spells out how the certificate holder proposes to implement the AQP. Included in this proposal is the schedule for the Phase III training evaluation to include instructor/evaluator training and small group tryouts. It should also include provisos for evaluating the effectiveness of performance measurement tools, and provisions for evaluating facilities, courseware, and equipment before starting the plans for the small group tryouts. The second part explains how the certificate holder intends to operate the AQP in Phases IV and V. Included in this section are strategies for maintaining the program, crew pairing policy, first-look administration, and instructor/evaluator requirements. The operations plan should also describe in detail the data management plan. This plan includes a statement of understanding addressing the collection and analysis of performance/proficiency data and a description of the performance proficiency database (PPDB), the data management collection process, and the FAA data submission, analysis, and reporting requirements.
1 Implementation – Phase III Y N Comments
a. Does this section include schedules for the implementation of the each of the AQP curriculums?
b. Do the schedules correlate to the MATS?
c. Do the schedules include dates for training the instructor/evaluators?
d. Does the instructor/evaluator training include difference training for those previously qualified (if applicable)?
e. Does this section include provisions for evaluating facilities, courseware, FSTDs, and training equipment before starting the Phase III
f. Does this section describe a plan for evaluating and observing the instructors and evaluators during the Phase III small group t t? g. Does the instructor/evaluator training include provisions for evaluating the effectiveness performance measurements such as the application of the rating scale, use of grade sheets and student feedback?
h. Does this section indicate that the applicant will request no-jeopardy credit for the small group tryout course graduates in a separate letter addressed to the manager of the appropriate S afety Standards office and the POI?
2 Operations Phases IV and V – AQP Maintenance Y N Comments
a. Does this section describe the methodology for maintaining control of the AQP approval documents?
b. Does this methodology include a procedure for providing document copies to the POI and t he manager of the appropriate Safety Standards office after the receiving the approval stamp f rom the POI?
Appendix 4 Y N Comments
c. Does this section describe the strategy to be employed for curriculum maintenance and update?
d. Does the strategy for curriculum maintenance and update include plans for acquiring and measuring data for tracking curricula?
e. Does the strategy for curriculum maintenance and update identify the person(s) responsible making changes in the AQP?
f. Does this section describe the strategy for monitoring and responding to demographic changes?
g. Does this section describe using training and evaluation feedback to maintain and improve the A QP?
h. Does this section describe how student and instructor feedback will be obtained?
i. Does this section describe any plans to upgrade training equipment?
3 Operations Phases IV and V – First-Look Maneuvers
Y N Comments
a. Does the applicant define first-look maneuvers, their purpose, and the strategy that will be used to administer them?
b. Does this strategy indicate who will administer the first-look maneuvers and at what point in the curriculum?
c. Does this strategy state that first-look will not be briefed prior to the first execution of these items?
d. Does this section describe how first-look maneuvers are selected?
e. Does this section describe how the first-look maneuvers would be updated?
f. Does the applicant describe how first-look maneuvers will be analyzed to determine trends of degraded proficiency?
4 Operations Phases IV and V – LOFT/LOE Crew Scheduling and Pairing Policy Y N Comments
a. Does this section describe the circumstances that would require a seat substitute?
b. Does this section describe the decision rules that would apply in obtaining a seat substitute?
c. Does the decision rules that would apply in obtaining a seat substitute ensure that in all cases, the seat substitute must be task familiar with the duty position?
Appendix 4 Y N Comments
d. Does this section describe at what point in the curriculum that a seat substitute would be used?
e. Does the applicant acknowledge that all occurrences of seat position substitution including the qualification of the seat substitute must be reported?
5 Operations Phases IV and V – Instructor/Evaluator Requirements Y N Comments
a. Does the applicant identify by title each instructor or evaluator position?
b. Does the applicant describe the job function(s) that each instructor or evaluator is authorized to perform?
6 Data Plan Y N Comments
a. Does the data plan have a prologue that establishes the intended purpose and methods for the collection, management, analysis, and r eporting of AQP training/evaluation data for each curriculum?
b. Does the prologue specify how the data plan will be maintained and updated?
c. Does the prologue acknowledge the airline’s responsibility to collect and analyze more data than is required to be submitted to the FAA in or der to adequately identify performance trends and make changes to factors that impact the crewmember or aircraft dispatcher performance?
d. Does the data plan address the methods
(e.g., gradesheets, computer-input screens, etc.)
u sed to collect performance/proficiency data for all curriculums?
e. Does the data plan address data input quality control, security, and usability?
f. Does the data plan address data management as the means and strategy the AQP airline intends to employ to store, access, and assimilate the AQP/SVTP performance/proficiency data that is collected?
g. Does the data plan address the type of software the data management system employs
(e.g., relational database, spreadsheet etc.), the o rganization of the information in the electronic medium (e.g., database definition, database table r elationships, spreadsheet description, etc.) and a d escription of the user interface to this data management system?
Appendix 4 Y N Comments
h. Does the data plan address the type of analysis it will employ to facilitate the AQP performance information needs of the airline and the FAA?
This discussion of the data analysis must address how each type of AQP data will be analyzed.
i. Does the data plan address the FAA data submission requirements including format and frequency?
j. Does the data plan address the type of data format it will employ for the reports, (e.g., tabular reports, graphs)?
k. Does the data plan address the frequency of the reports, both internal and FAA?
l. Does the data plan identify the internal airline personnel that will receive the reports?
m. Does the data plan include copies of all forms used for data acquisition and grading?
n. Does the data plan include a database description of data field types and a graphical depiction of the Database table relationships?
o. Does the data plan address a quality assurance strategy for ensuring data integrity? Does this strategy include instructor and evaluator grading calibration?
Appendix 4 Annual AQP Report – Phases IV & V. AQP requires that each AQP certificate prepare an annual AQP report for the FAA. This report is based on the certificate holder’s analysis of the data that is collected during training and at strategic points (validation/evaluation gates) in each curriculum and maintained in the proficiency/performance database (PPDB). AQP requires data collection and analysis in order to establish and maintain quality control of curricula for crewmembers, aircraft dispatchers, instructors, and evaluators. The annual AQP report should summarize the lessons learned and adjustments made to the curriculum(s) during the reporting period. The report should also include projected or proposed changes to the curriculum(s) based on the certificate holder’s current analysis. The actual adjustments made to the AQP are reflected in revisions to the approved AQP documents. The report should be submitted to the FAA no later than 60 days past the end of the report period. The reporting period is usually based on the approval date for a particular curriculum in either Phase IV or V. During AQP development, particularly for multiple fleet operators, with different approval dates for multiple curriculums, the reporting period may be modifiable as agreed upon by the FAA and the certificate holder. Copies of the report should be distributed the Principal Operations Inspector (POI) and the appropriate Safety Standards office at least 2 weeks prior to the annual AQP review meeting.
Annual AQP Report Y N Comments
a. Is there a prologue or introduction to the report that addresses how each type of AQP data w as analyzed? This prologue should corroborate the information in the I&O Plan.
b. Does the report discuss data reliability and consistency?
c. Does the report summarize the internal quality assurance reports as addressed in the I&O Plan?
d. Does the report validate the effectiveness of the AQP with supporting evidence of successful training and evaluation?
e. Is the report supported by the FAA analysis of the submitted data?
f. Does the report identify any trends, problem areas, and potential deficiencies that could result in decreased proficiency?
g. Does the report include a description of corrective measures taken and any resulting changes to curriculums?
h. Does the report include any projected corrective measures to be taken and provide rationale for these changes?
i. Does the report indicate a need for changes to the AQP maintenance strategy as described in the I&O Plan?
j. Does the report indicate a need for changes to the certificate holders data plan as described in the I&O Plan?
k. Does the report provide an analytical comparison of data between equivalent periods in preceding years?
l. Does the report identify any future operational changes that will affect the AQP (operational changes or trainee demographics)?
Appendix 4 Page 21 (and 22) Y N Comments
m. Does the report analyze training and evaluation feedback as part of the collected data t o determine the effectiveness of the training program?
n. Does the report analyze on-time completion rates for training curriculums and initial operating experience?
o. Does the report analyze special tracking rates?
p. Does the report analyze instructor/evaluator (I/E) rater reliability training results?
q. Does the report analyze instructor comments as part of the collected data to determine the effectiveness of the training program?
r. Does the report analyze first look data?
s. Does the report analyze maneuvers data?
t. Does the report analyze LOE data by technical topics, and CRM elements?
u. Does the report analyze Line check data (exclusive of initial OE)?
v. Does the report address progress towards Phases III, IV, & V in other fleets (as applicable)?
w. Does the report address seat substitution rates?
x. Does the report address recordkeeping?
y. Does the report address adherence to the I&O Plan?
z. Does the report address the validity and usefulness of the qualification standards?
aa. Does the report address internal audit or FAA surveillance findings?
bb. Does the report address related safety programs (i.e., FOQA/ASAP)?
Appendix 5 APPENDIX 5.
ADVANCED QUALIFICATION PROGRAM QUALIFICATION
STANDARDS DOCUMENTATION This is the section of the Job Task Analysis that serves as the foundation document for the sample qualification standards that follow. Task 6.1 serves as the basis for the Terminal Proficiency Objective (TPO)-level qualification standard while subtasks 6.1.1 and 6.1.2 serve as the foundation for the Supporting Proficiency Objective (SPO)-level qualification standards.
Generic Airlines Inc. Operations Manual Volume 9 Chapter 3: Qualification Standards, B757 Flightcrew Rev. # 5 Dated 10/01/07
6. Approach Operations and Landing
6.1 Perform an Instrument Approach*
6.1.1 Perform a 2 Engine Precision Approach Cat I ILS and Landing*
6.1.2 Perform a 1 Engine Inoperative Cat I ILS Approach*
6.1.2 Perform CAT II ILS
6.1.3 Perform CAT IIIb ILS
6.1.4 Perform Coupled Autopilot Approach and Autoland Procedures
6.1.5 Perform Non-Precision Approach Procedures (VOR, NDB, LOC, LOC/BC, LDA, SDF, ASR, RNav/FMS, GPS)
6.1.6 Perform an IMC One Engine Inoperative Missed Approach
6.2 Visual Approach
6.3.1 Perform Visual Approach and Landing
6.3.2 Perform Visual Approach and Rejected landing * Used as an example.