Par 8-1 Page 87 CHAPTER 8. AQP DATA MANAGEMENT SECTION 1. INTRODUCTION
8-1. OVERVIEW. This chapter provides general guidance for the management of performance/proficiency data within an Advanced Qualification Program (AQP) and a Single Visit Training Program (SVTP). It also defines the minimal requirements for the collection and submission to the FAA of de-identified curriculum data. Guidance for data management is found in the AQP Data Management Guide that is produced by the Airlines for America’s (A4A) Data Management Focus Group.
a. Regulatory Requirement. Data collection and analysis processes acceptable to the Administrator that will ensure that the certificate holder obtains performance information on its crewmembers, instructors, and evaluators that will enable the certificate holder and the FAA to determine whether the form and content of training and evaluation activities are satisfactorily accomplishing the overall objectives of the curriculum.
NOTE: The FAA uses de-identified data for program monitoring, not for individual crewmember, dispatcher, or other operations personnel monitoring.
b. Validation. The principal goal of the AQP is true proficiency-based training and qualification. This proficiency-based (expressed as performance objectives) is systematically developed and maintained, then continuously validated through the collection and analysis of performance/proficiency data. Data collection and analysis is a fundamental part of AQP and is also required for air carriers operating under an approved SVTP.
8-2. DEFINITIONS. In AQP, data is classified into the two broad categories of individual qualification records and performance/proficiency data.
a. Individual Qualification Records (Crew Records). These are identifiable records maintained in sufficient detail on each crewmember, instructor, and evaluator who is qualifying or has qualified under an AQP. These records show how and when the individual satisfied the requirements of each curriculum and therefore each assigned duty position. Air carriers may maintain a manual recordkeeping system, or a computerized recordkeeping system, based on the standard Title 14 of the Code of Federal Regulations (14 CFR) part 121 or 135 recordkeeping requirements. The recordkeeping process in AQP does not differ from traditional recordkeeping requirements.
b. Performance/Proficiency Data. This is de-identified information that represents the results of an individual’s ability to successfully demonstrate the performance objectives of each curriculum. This information is captured during validation and evaluation gates as a crewmember progresses through an AQP curriculum. This data is collected from each crewmember’s performance and is stored in aggregate in a Performance/Proficiency Database (PPDB). This data is used to analyze training programs and/or groups of participants, not for tracking individual accomplishment. Successful collection and analysis of this data will identify and correct problems, validate AQP curriculums, and identify developing trends.
Par 8-3 Page 88 SECTION 2. DATA MANAGEMENT
8-3. OVERVIEW. In AQP, data management is a continual process of data collection, entry, validation, and submission.
8-4. DATA COLLECTION—SVTP AND AQP.
a. SVTP. Collecting SVTP data essentially is a data collection process superimposed over the traditional recurrent training and line check program. SVTP data is collected for flight maneuvers performed during first-look sessions, proficiency, and line checks. This data consists of grades using a rating scale with associated reason codes as required in the certificate holder’s Single Visit Exemption (SVE) and contained in the individual carrier’s approved SVTP.
Figure 8-1 illustrates the single visit data collection fields required for submission to the FAA.
b. AQP. AQP data collection is required for all AQP curricula as defined by each carrier’s approved AQP data collection and analysis section of the certificate holder’s Implementation & Operations Plan (I&O Plan). AQP data is collected at each validation or evaluation gate. This data consists of graded proficiency objectives using a rating scale with associated reason codes (if applicable). Data collection requirements for the AQP will vary with the curriculum, the type of curriculum activity (training, validation, or evaluation), the type of participant (crewmember, instructor, or evaluator), and the overall management objectives for use of the data. Figure 8-1 illustrates the AQP data collection fields required for submission to the FAA. All performance data collected on each proficiency objective must be relative to the applicable AQP qualification standards defined for the training and evaluation activities.
NOTE: For each crewmember in a qualification or continuing qualification curriculum, the FAA must be able to associate the data records applicable for that crewmember in that curriculum through logical grouping of the records, or linkage by a common de-identified index number (but not by name).
8-5. DATA ENTRY. All performance data collected throughout the SVTP and AQP is entered into the carrier’s PPDB. Typically, this is an electronic database for ease in analysis, comparison, and reporting purposes. Considerations for data entry include the method, the hardware/software required for data input, and the hardware/software required for data storage and manipulation.
Database design is at the discretion of the carrier, providing that the design can generate the required report tables specified.
NOTE: The manager of the appropriate Safety Standards office requests copies of all grade sheets used in the evaluations or used through each gate.
8-6. DATA SUBMISSION. The FAA has established the minimal requirements for the submission of de-identified data by curriculum. Figures 8-2 through 8-4 summarize the submission requirements in three tables. The three tables are the Performance Data Report Table (PDRT), the Skill Reason Table (SklRsn) and the Training Objectives Report Table (TORT).
The information in these tables is downloaded from the carrier’s PPDB. The submissions are forwarded to the manager of the appropriate Safety Standards office electronically in 1 calendar-month blocks for both the SVTP and AQP using a Microsoft Access (.mdb) format
Par 8-6 Page 89 within 2 months of collection. The FAA will analyze the de-identified data using standard automated queries and reports to identify AQP performance trends. Figures 8-2, 8-3, and 8-4 contain specifications for each field in the three tables.
a. PDRT. The PDRT contains a listing of 23 fields that are reported for every measured item, providing a separate record for each. A measured item is a maneuver, task, procedure, or event set, and is the main component for data analysis. These fields provide a record of the results of the performance of each measured item along with supporting data for reporting and analysis. Certain supporting data fields (airline designator, curriculum, etc.) repeat for each record and can be automatically generated from a query/software routine. Each field in the PDRT must contain an entry, alpha/numeric, numeric, or text. Figure 8-2 contains the specifications for each field, defines the meaning of each field item, and provides examples of the field values.
b. SklRsn. An unsatisfactory rating (Mrate) of a measured item requires that a reason be entered to explain the rating. All reasons are entered in the SklRsn in field No. 24, Skill Reason Text. This table allows the carriers to report more than one Skill Reason Text for an unsatisfactory rating of a measured item without having to enter multiple records. There are only two fields in this table.
(1) The first field is Field No. 5 (SklRsn), which is the same as in the PDRT and provides the linkage between the two tables. For each record in the PDRT where one or more Skill Reasons need to be reported, an identifier is entered in the SklRsn field that is unique to the measured item. This identifier can be a concatenation of the fields that make it unique. For instance: Fleet+EvalType+CmID+Mitem. As long as the identifier is unique to the record, it is the carrier’s discretion to determine which fields are used.
NOTE: If this approach is used, care must be taken to ensure that none of the component field contains an NA value.
(2) The second field in the SklRsn Table is the Skill Reason Text, the field that will contain descriptive text for each reason relevant to a particular measured item. For example:
Systems Knowledge, Procedural, CRM, Technical Knowledge, etc.
c. TORT. This table is comprised of training objective(s) associated with each measured item. It provides a means of auditing the relationship between the measured item and the task analysis, qualification standards, and the curriculum. There are four fields in this table. The TORT links to the PDRT through the first field, Measured Item ID (MItemID). The second field, Objective ID (ObjID) is an identifier assigned to each Terminal Proficiency Objective (TPO) or Supporting Proficiency Objective (SPO). The Objective ID may be the same as the Measured ID if it is a single TPO or SPO. However, in the case of a measured item that is an event set, there may be several training objective identifiers associated to the Measured Item ID. The third field, Objective Title (ObjTitle) is a text description of the training objective. The last field in this table is the Objective Type (ObjType) that identifies the training objective as a TPO or SPO.
Par 8-6 Page 90
d. The File Naming Convention. Data submissions must use the file-naming format (xyzmmyya.mdb) as shown in the example below:
S ECTION 3. DATA ANALYSIS AND REPORTS
8-7. OVERVIEW. The primary users of data reports are internal carrier personnel and the FAA. Chapter 2, Section 7 discusses the specific reporting requirements to the FAA.
a. Air Carrier. AQP requires that the data collection conducted by the air carrier for its own use in monitoring curricula will support more analytical detail and diagnostic functions than the data collected for submission to the FAA. The FAA expects the certificate holder to do an in-depth analysis of the effectiveness of the training. Reporting of data is based on the analysis of the PPDB to provide information on the curriculum and participant groups (crewmembers, instructor/evaluators). Once the data is collected and entered into the PPDB, an analysis should be performed on the aggregate information. Statistical analysis of the proficiency data enables carriers to make an internal assessment of their performance. Carriers should tailor these processes and techniques to suit their own requirements. Each carrier’s data collection and performance assessment processes should be refined over time, based on their own practical experience. That is, the measures and processes should be optimized on an iterative basis to provide the degree of discrimination in crewmember performance needed to establish effective quality control over AQP curricula. See the A4A’s Data Management guide for a more extensive discussion on reports and their use.
b. FAA. The data submissions to the FAA are primarily ratings and reason codes associated with performance measures taken at validation and evaluation gates and supporting data. The data, presented to the FAA in the three tables previously discussed is analyzed using the Data Analysis Reporting Tool (DART). This tool allows Principal Operations Inspectors (POI) and the appropriate Safety Standards office personnel to conduct trend analysis to monitor overall program effectiveness.
Par 8-7 Page 91 FIGURE 8-1. DATA SUBMISSION REQUIREMENTS Training Program Single Visit Advanced Qualification Program Curriculum Recurrent Qualification Continuing Qualification Table # Field Short Name FL PC LC SV PV MV LOE LC FL MV LOE LC Performance Data Report Table 1 Measured Item MItem Req. Req Req Des Des Req Req Req Req Req Req Req 2 Measured Item ID MitemID N/A N/A N/A Des Des Req Req Req Req Req Req Req 3 Measured Item Rating Mrate Req. Req Req Des Des Req Req Req Req Req Req Req 4 Rating Meaning Rmean Des Des Des Des Des Req Req Req Req Req Req Req 5 Skill/Reason SklRsn Req. Req Req N/A N/A Req Req Req Req Req Req Req 6 PF/PNF PFPNF Req** Req Req N/A N/A DES Req Req Req** DES Req Req 7 Is Currency Item? Currcy N/A N/A N/A N/A N/A Req N/A N/A Req Req N/A N/A 8 Is Critical Item Crit N/A N/A N/A N/A N/A Req N/A N/A Req Req N/A N/A 9 Crew ID CrewID Req. Req Req Req*** Req.*** Req Req Req Req Req Req Req 10 Evaluation Date EvalDate Req. Req Req Req Req Req Req Req Req Req Req Req 11 Airline Designator AirDsgn Req Req Req Req Req Req Req Req Req Req Req Req 12 FAA Fleet Designator Fleet Req Req Req Req Req Req Req Req Req Req Req Req 13 Training Program TrPgm Req Req Req Req Req Req Req Req Req Req Req Req 14 Curriculum Curr N/A N/A N/A Req Req Req Req Req Req Req Req Req 15 Evaluation Type EvalType Req Req Req Req Req Req Req Req Req Req Req Req 16 FAA Simulator ID SimID DES DES N/A N/A N/A Req Req N/A Req Req Req N/A 17 Evaluator ID EvaltrID DES DES DES Req Req Req Req Req Req Req Req Req 18 FAA Inspector ID FAAID DES DES DES Req Req Req Req Req Req Req Req Req 19 Random LC Geo. Area GeoArea N/A N/A Req N/A N/A N/A N/A N/A N/A N/A N/A Req 20 Comments Comment Des Des Des Des Des Des Des Des Des Des Des Des 21 Crewmember ID CmID Req Req Req Req Req Req Req Req Req Req Req Req 22 Crew Position CrewPos Req Req Req Req Req Req Req Req Req Req Req Req 23 Evaluation Rating Type EvalRtg N/A Req Req Req Req Req Req Req N/A Req Req Req
Skill Reason Table 5 Skill/Reason SklRsn Req Req Req N/A N/A Req Req Req Req N/A Req Req 24 Skill Reason Text SklRsnText Req Req Req N/A N/A Req Req Req Req N/A Req Req
Par 8-7 Page 92 FIGURE 8-1. DATA SUBMISSION REQUIREMENTS (CONTINUED) Training Objectives Report Table 2 Measured Item ID MItemID N/A N/A N/A Des Des Req Req Req Req Req Req Req 25 Objective ID ObjID N/A N/A N/A Des Des Req Req Req Req Req Req Req 26 Objective Title ObjTitle N/A N/A N/A Des Des Req Req Req Req Req Req Req 27 Objective Type ObjType N/A N/A N/A Des Des Req Req Req Req Req Req Req Req - A required entry.
N/A - If no value is to be entered, use N/A. Leave no blank fields.
Des - Desired. This is optional information that is not required, but desired for submission. If not submitted, then N/A must be entered as the value.
** - See Notes in Data Reporting Format - Figure 8-2.
*** - If there is no crew pairing for Systems or Procedures Val, submit the single CmID (No. 21) for the crewmember.
SVT - Single Visit Training Qual - AQP Qualification Curriculum CQ - AQP Continuing Qualification Curriculum FL - First Look Maneuvers PC - Proficiency Check SV - Systems Validation PV - Procedures Validation MV - Maneuvers Validation LOE - Line Operational Evaluation LC - Line Check
Par 8-7 Page 93 FIGURE 8-2. PERFORMANCE DATA REPORT TABLE (PDRT) Table No. Field Description Short Name Type Field Size Example(s) Notes PDRT 1 Measured Item The task, maneuver, procedure, or event set statement.
Mitem Text 80 Perform Engine Failure Procedure
1) This field will be N/A for Qual. SV and PV because these validations are usually graded pass/fail and recorded in field No. 23.
2 Measured Item ID An alphanumeric used to identify the task, maneuver, procedure or event set being graded.
MitemID Text 12 1.2.1.3 or KK73456
1) This value is associated with a TPO, SPO, or event set. When it is a TPO or SPO, use the related number from the hierarchical numbering system. When it is an event set, enter the event set identifier.
2) This field will be N/A for Qual. SV and PV because these validations are normally graded in aggregate (pass/fail), and recorded in field No. 23.
3 Measured Item Rating The numeric rating assigned on the first attempt for each measured item.
Mrate Numeric 1 3 1) Values depend on rating scale.
2) Do not provide records that are missing MRate: Provide only performance related information. Do not provide “Incomplete”.
3) Reported rating must refer only to the first attempt of the measured Item, not the terminal performance.
4) * Use 9 instead of N/A for Qual. SV and PV. This is a numeric only field.
4 Rating Meaning A description of the Measured Item Rating.
Rmean Text 40 Satisfactory 1) The text meaning of each number in the rating scale.
5 Skill/Reason Link between the PDRT and SKLRSN tables.
SklRsn Text 90 B737-LOE- 5555-Taxi
1) This field is a unique record identifier that must be provided for Unsatisfactory Measured Item ratings (MRate) to provide a link to the Skill Reason table for single or multiple reasons for a failure.
2) This identifier can be a combination of the fields that make it unique. For example: Fleet+EvalType+CmID+MItem. As long as the identifier is unique to the record, it is the carrier’s discretion to determine which fields are used.
3) If Mitem is satisfactory, insert N/A in this field.
6 PF/PNF Indicates whether the pilot performing the Measured Item was flying or not flying.
PFPNF Text 3 PF or PNF 1) No entry requirement for FE, FA or DISP.
2) N/A for Qualification Curriculum Systems and Procedures Validation.
3) * For FL, it is desired and encouraged that PNF data be submitted along with PF data. However, if significant grade sheet changes are necessary to accommodate this, only the standard PF items need be identified. For these airlines, the value entered for all FL Measured Items will be “PF”.
4) This field is desired for Qual. and CQ Maneuvers Validation, however, if PNF is not submitted then the value will be “PF”.
Par 8-7 Page 94 Table No. Field Description Short Name Type Field Size Example(s) Notes PDRT 7 Is currency Item?
Indicates whether the Measured Item is a currency item.
Currcy Text 3 Yes, No or N/A 1) N/A is used when the Measured Item is an Event Set.
8 Is Critical Item Indicates whether the Measured Item is a critical item.
Crit Text 3 Yes, No or N/A 1) N/A is used when the Measured Item is an Event Set.
9 Crew ID The de-identified alphanumeric assigned to a specific pairing of crewmembers that will be maintained for the duration of a curriculum, unless the crew make-up changes or a seat substitute is present.
CrewID Text 40 P21234 SIC F12312
1) CrewID N/A for FA’s, IRO’s or dispatchers.
2) CrewID is the concatenation of the CmID’s (#21) of all crewmembers present. Order of ID’s: PIC SIC FE.
3) If two PICs or SICs are trained together use the structure PIC PIC FE, or SIC SIC FE, as appropriate, separated by spaces.
4) If there is no crew pairing for Systems or Procedures Val, submit the single CmID (#21) for the crewmember.
5) Maintain CrewID in exact form and order, except for seat substitutes. For seat subs, use Crew Position (No. 22) values relative to the seat position in the CrewID sequence. For example, P21234 SIC F12312 would indicate that the first office position was a seat substitute.
10 (MM/YYYY) The month and the year the Measured Item data is collected.
Date Date 7 10/2001 1) Set the date field format in Access to month/year; use full century: MM/YYYY.
2) The day value will default to 01.
11 Airline Designator The airline’s four character designator reporting the measured items.
AirDsgn Text 4 XYZA 12 FAA Fleet Designator The FAA designation of the fleet or equipment type relevant to the Measured Item.
Fleet Text 20 B-737 1) See Airline/Fleet Designator Menu.
13 Training Program The training program relevant to the Measured Item.
TrPgm Text 4 AQP or SVT 14 Curriculum The AQP curriculum in which the Measured Item is being validated or evaluated.
Curr Text 4 Qual, CQ or N/A
1) N/A for SVT.
15 Evaluation Type The evaluation type in which the Measured Item is accomplished.
EvalType Text 4 FL, PC, SV,
PV, MV, LOE, LC
Par 8-7 Page 95 PDRT 16 FAA Simulator ID The FAA simulator ID number of the simulator where the Measured Item is performed.
SimID Text 4 1234 or N/A 1) If a simulator is not used, enter N/A.
17 Evaluator ID The identification number of the evaluator who graded the Measured Item.
EvaltrID Text 15 123456 1) If an FAA inspector is the evaluator, use the FAAID in this field and in the FAA Inspector ID field, No. 18.
18 FAA Inspector ID The Form 110A number of the aviation safety inspector who observed the Measured Item.
FAAID Text 4 1234 1) Report the FAA ID Number (110A) of a FAA inspector who is there as an observer and/or evaluator (#17). If no FAA inspector is present, enter N/A.
19 Random Line Check Geographic Area The geographic area where the Measured Item is collected for Random Line Checks.
GeoArea Text 30 Pacific 1) Use geographic descriptors as define by the airline.
2) This field is only required for airlines with approved Random Line Check programs otherwise enter N/A.
20 Comments Additional comments submitted by evaluator.
Comment Memo 1) The comment field provides further explanation of unsatisfactory or outstanding measured Item Rating.
2) Evaluator comments are provided to the appropriate Safety Standards office at the discretion of the airline, unless otherwise require in accordance with the approved airline’s AQP.
3) If there are no comments, enter N/A.
21 Crewmember ID The de-identified alphanumeric assigned for the duration of a curriculum to the crewmember performing the measured item.
CmID Text 10 P123456 1) Begin all PIC ID’s with P, Sic ID’s with S, FE ID’s with F, FA ID’s with A, dispatcher ID’s with Des, and IRO ID’s with R.
2) The P, S, or F refer to the position for which the pilot in training is being qualified, not necessarily the seat occupied.
22 Crew Position The “seat” occupied by the person performing the Measured Item.
CrewPos Text 4 PIC PIC = Pilot in Command; FA = Flight Attendant;
SIC = Second in Command; DISP = Dispatcher;
FE = Flight Engineer; IRO = International Relief Officer E.g., a SIC in the left seat would be entered a PIC.
23 Evaluation Type Rating The overall rating the pilot performing the Measured Item received for the evaluation type.
EvalRtg Text 5 Sat or Unsat 1) Rating must refer to the first execution of the evaluation type.
Repeated sessions are not reported in this table.
Par 8-7 Page 96 FIGURE 8-3. SKILL/REASON TABLE (SKLRSN) Table # Field Description Short Name Type Field Size Example(s) Notes SklRsn 5 Skill/Reason Link between the PDRT and SKLRSN tables.
SklRsn Text 90 B-737-LOE- 5555-Taxi
1) This field is a unique record identifier that must be provided for Unsatisfactory Measured Item ratings (MRate) to provide a link to the Skill Reason table for single or multiple reasons for a failure.
2) This identifier can be a combination of the fields that make it unique. For example: Fleet+EvalType+CmID+ MItem.
3) As long as the identifier is unique to the record, it is the carrier’s discretion to determine which fields are used.
4) None of the component fields can contain a N/A value.
24 Skill Reason Text A description of the skill or reason for assigning an unsatisfactory Measured Item rating.
SklRsn Text Text 40 Technical CRM Procedural
1) Provided for Unsatisfactory Measured Item Ratings from the carrier’s Reason Codes.
2) If more than one skill or reason is relevant, allow one field for each skill or reason.
3) Enabling Objectives (EO’s) are acceptable as skill descriptors.
Par 8-7 Page 97 (and 98) FIGURE 8-4. TRAINING OBJECTIVES REPORT TABLE (TORT) Table # Field Description Short Name Type Field Size Example(s) Notes TORT 2 Measured Item ID An alphanumeric used to identify the task, maneuver, procedure or event set being graded.
MitemID Text 12 1.2.1.3 or KK73456
1) This value is associated with a TPO, SPO, or event set. When it is a TPO or SPO, use the related number from the hierarchical numbering system. When it is an event set, enter the event set identifier.
2) This field will be N/A for Qual. SV and PV because these validations are normally graded in aggregate (pass/fail), and recorded in field No. 23.
25 Objective ID An alphanumeric assigned to each terminal or supporting objective tested by the Measured Item.
ObjID Text 12 3.2 1) List all the TPO’s, SPO’s or other high level objective grouping that apply to the Measured Item.
26 Objective Title A description of the objective ID.
ObjTitle Text 80 Perform Engine Fire Procedures
1) A description of the objective ID in field No. 25.
27 Objective Type A description of the objective type.
ObjType Text 5 TPO or SPO or EO
1) Identify the training objective as a TPO, SPO, or EO.
Par 9-1 Page 99 CHAPTER 9. CREW RESOURCE MANAGEMENT SECTION 1. INTRODUCTION
9-1. OVERVIEW. Crew Resource Management (CRM) training develops skills that enhance flight safety through the effective use of all available resources: human, hardware and information. CRM training increases awareness of human and system error and provides techniques and skills that will minimize their effects. This is accomplished through awareness of crewmember attitudes and behavior as well as the use of practical flight management skills.
CRM training has achieved a greater degree of integration through the Advanced Qualification Program (AQP). This chapter describes the integration of CRM training into pilot training curriculums using the instructional systems development methodology that forms the foundations of the AQP process. The approach demonstrates how the analytical methodologies in AQP have the potential to produce more rigorous CRM training, including the appropriate integration of CRM and technical training.
SECTION 2. INTEGRATING CRM INTO AN AQP
9-2. SCOPE OF INTEGRATION. One of the major objectives of AQP is the seamless integration of CRM and technical training. Where appropriate, CRM procedures are identified, documented, integrated, and accorded the same weight as the technical procedures required for the execution of a given phase of flight and its associated flight tasks.
a. However, integration does not mean that only those aspects of CRM that can be proceduralized are addressed in flight operations technical training and evaluation. Some aspects of CRM are inherent to maneuver performance, and have already been proceduralized. For example, communication procedures for coordinating callouts during takeoffs, approaches, and other maneuvers are specified in terms of what should be said and when it should be said. These callouts take place during most flight phases and are performed at fixed points in the flight-phase sequence of task activities. Similarly, some aspects of communication during the management of non-normal conditions can be easily positioned within the sequence of activities performed to manage the non-normal condition.
b. Other equally important CRM activities, in contrast, are performed on an as-needed basis, in order to manage the flight, work well as a team, or respond to unique situations.
Recognizing the need for, and effectively executing these activities, is critical to coordinating the various duties the crew must perform during the flight. For example, certain communications should be performed in order to maintain crew awareness of flight status. Regardless of the phase of flight, it is critical that the crew recognize this communication requirement and effectively act upon it in a timely manner to maintain crew situational awareness.
c. These global activities do not fit neatly within a hierarchical list of technical activities, organized by phase of flight. Instead, they act as a “shell” or “template” that overlays and organizes the activities that may need to be performed during any phase of flight. These skills as a whole constitute a management strategy that represents a critically important part of the inventory of defenses against error that crews bring with them to the flightdeck. In particular, they can provide the crew with the tools needed to resolve problematic situations to which they