MINIMUM FLIGHT CREW
AC 23.1523
JANUARY 12, 2005
Initiated By: ACE-100
FOREWORD
The advisory circular (AC) system became effective in 1962. It provides a single, uniform, agency-wide system that the Federal Aviation Administration (FAA) uses to deliver advisory material to FAA customers, industry, the aviation community, and the public. This AC sets forth one method that may be used to show compliance to the requirements contained within 14 CFR, part 23, § 23.1523, which prescribes the certification requirements for minimum flight crew on part 23 airplanes.
S/
Michael K. Dahl Acting Manager, Small Airplane Directorate Aircraft Certification Service
i (and ii)
CONTENTS
1. PURP OSE1
2. BACK GROUND1
3. APPL ICABILITY1
4. RELATED REGULATIONS AND PUBLICATIONS2
a. Regul ations2
b. AC' s2
c. Policy Statements2
d. AC's for Sale2
5. GENERAL CERTIFICATION CONSIDERATIONS3
6. CERTIFIC ATION PROCEDURES3
a. Evaluation Test Plan3
b. Analytical Approach4
c. Testing4
d. Data Collection and Analysis6
e. Credit for Tests7
APPENDIX 1– PILOT BEHAVIORS ASSOCIATED WITH HIGH WORKLOADA1-1
APPENDIX 2 – MINIMUM FLIGHT CREW RULE AND WORKLOAD FACTORSA2-1
APPENDIX 3 – WORKLOAD EVALUATION ANNOTATED REFERENCESA3-1
1. PURP OSEA3-1
a. Definition of WorkloadA3-1
b. Perf ormance and WorkloadA3-1
c. Wor kload and ErrorA3-1
APPENDIX 4 – RECOMMENDED TEST PLAN GUIDANCE AND DATA COLLECTION
INFORMATIONA4-1
1. DAT A COLLECTION TECHNIQUESA4-1
a. Selecting Performance Measures, Defining CriteriaA4-1
b. Questi onnaire DevelopmentA4-1
iii
c. Types of Evaluations – Comparative Versus Absolute EvaluationsA4-1
2. Form sA4-1
3. DebriefA4-1
4. Data ReductionA4-1
5. Level of TestingA4-2
6. Questi onnaire DevelopmentA4-2
7. Rating ScalesA4-3
FIGURE 1 - FORM TO COLLECT PILOT VEHICLE INTERFACE AND WORKLOAD DATAA4-4
FIGURE 2 – COLLECT PILOT/USER COMMENTS AND FEEDBACKA4-5
FIGURE 3 – COLLECT PILOT/USER OPINIONS AND FEEDBACKA4-6
iv
1. PURPOSE.
a. This advisory circular (AC) sets forth one method that may be used to show compliance to the requirements contained within 14 CFR, part 23, § 23.1523, which prescribes the certification requirements for minimum flight crew on part 23 airplanes.
b. This AC is one method that can be used to determine workload factors and issues for normal, utility, acrobatic and commuter category airplanes. Material in this AC is neither mandatory nor regulatory in nature and does not constitute a regulation.
c. This material is a reference for part 23 airplane manufacturers, modifiers, Federal Aviation Administration (FAA) design evaluation engineers, flight test engineers, engineering flight test pilots [Aircraft Certification Office (ACO), and Flight Standards, and Manufacturers] as well as human factors engineering evaluators. This material may be used by FAA authorized designees to perform workload evaluations.
d. This AC encourages participation and coordination from all the test community participants described in c.
above. This AC is consistent with the flight test guidance and workload factors described in the minimum flight crew evaluation sections and workload factors described in AC 23-8B. This AC is an acceptable means of showing compliance for part 23 on flight tests and pilot judgments.
2. BACKGROUND.
a. In the early 1980s, there was a move to reduce the crew size of the new generation of commercial jet transport airplanes from three to two. This caused the Transport Airplane Directorate to develop more criteria and guidance for minimum crew determination for part 25 airplanes. AC 25.1523 was developed to provide manufacturers and certification personnel a means of showing compliance to 14 CFR, part 25, § 25.1523. Most part 23 airplanes are single pilot, none require a crew of three, and only a few require a crew of two. There was no need to address crew complement in these airplanes, and no parallel effort was started then for part 23 airplanes. Replacement of conventional electromechanical instruments with complex integrated electronic displays has made it necessary to more closely examine pilot workload for these installations.
b. For many years, part 23 airplane cockpits were rather simple in design and used instruments and systems that were also similar in operation. This made it rather easy for pilots to transition safely from one part 23 airplane to another. However, in recent years because of the growth of modern technology and the reduced cost of electronic components, novel and more complex integrated avionic systems are increasingly being installed in part 23 airplanes. These new systems have changed the appearance, operation, and usability of the pilot-vehicle interface.
There is also much variation between manufacturers in the design and operational characteristics of these systems.
So, there is a concern that pilot(s) familiar and proficient with one system may not be able to understand sufficiently and operate another system. Although many of these systems can improve pilot situational awareness and safety, poorly designed systems can increase pilot workload, and increase the potential for pilot error. Also, the lack of standardization in the design and operation of these systems can negatively affect pilot training and impact performance and safety. So, there is a need to more closely examine pilot workload and error potential in these complex, integrated cockpits.
c. In August 2002, the Small Airplane Directorate published Human Factors Policy for Part 23 Airplanes. The policy included discussion of the human factors considerations for § 23.1523. The information contained in this AC further describes the human factors considerations referring to that rule. In addition, airplane certification process guidance has also been published to address general certification considerations; these documents include:
(1) The FAA and Industry Guide to Product Certification, 1998
(2) The FAA and Industry Guide to Avionics Approvals, 2001
3. APPLICABILITY. This AC is applicable to all part 23 airplanes as a means of showing compliance to 14 CFR, part 23, § 23.1523, Minimum flight crew. When § 23.1523 was modified by amendment 23-34 in February 1987, additional workload determination criteria were added for commuter category airplanes. The guidance contained 1
this AC provides recommended evaluation criteria for commuter category airplanes. These considerations may also be used to evaluate workload and usability of the pilot vehicle interface on highly modified cockpits of normal, utility, and aerobatic category airplanes. Historically, the majority of part 23 airplanes have been certified for single pilot operation.
It is not expected that this situation will change, as most of the newly developed part 23 airplanes are being designed from the onset to be operated as a single pilot airplane. It is therefore unlikely that the FAA would require two or more crewmembers unless the design was considered to be unsafe for single pilot operation. Therefore, the major focus of this guidance is to provide assistance in examining the acceptability of the workload for single pilot operations.
Consequently, for those part 23 airplanes previously certified or designed from the onset for single pilot operation, this AC should be used to examine pilot workload for the purposes of evaluating the adequacy of the cockpit layout, display formats, information presentation, control operation, and system operation logic to support single pilot operation.
Generally, the results of these evaluations will be used to identify areas (i.e. as those mentioned in the previous sentence) that may need modification to allow for single pilot operation.
4. RELATED REGULATIONS AND PUBLICATIONS.
a. Regulations: The following related regulations are provided as a quick reference.
23.771 Pilot Compartment
23.773 Pilot Compartment View
23.777 Cockpit Controls
23.1301 Function and Installation
23.1303 Flight and Navigation Instruments
23.1305 Power Plant Instruments
23.1309 Equipment, systems, and installations
23.1311 Electronic display instruments systems
23.1321 Arrangement and visibility
23.1322 Warning, caution, and advisory lights
23.1367 Switches
23.1381 Instrument lights
23.1523 Minimum flight crew
23.1525 Kinds of operation
23.1541 Markings and placards: General
23.1543 Instrument markings: General
23.1545 Airspeed indicator
23.1549 Powerplant and auxiliary power unit instruments
23.1551 Oil quantity indicator
23.1553 Fuel quantity indicator
23.1555 Control markings
23.1583 Operating limitations
b. AC's. Copies of current publications of the AC's listed below can be obtained without charge from the U.S.
Department of Transportation, Subsequent Distribution Office, M-30, Ardmore East Business Center, 3341 Q 75th Avenue, Landover, MD 20785. The website where these advisory circulars can be found is http://www.airweb.faa.gov/AC.
AC 23.1311-1A Installation of Electronic Displays in Part 23 Airplanes
c. Policy Statements. You may obtain copies of the current policy statements listed below from the web at http://www.faa.gov/certification/aircraft/ by selecting “Regulations, Policy and Guidance” and then selecting “Policy”. You may request a copy from the Small Airplane Directorate, 901 Locust St., Kansas City, MO 64106.
Part 23 Human Factors Policy PS-ACE100-2001-004
d. AC's for Sale. You may obtain current copies of the following publications from the web at http://www.airweb.faa.gov/AC. If you cannot download them from the web, you may buy them from the 2
Superintendent of Documents, U.S. Government Printing Office, P. O. Box 371954, Pittsburgh, PA 15250–7954.
AC 23-8B Flight Test Guide AC 23.1309-1C Equipment Systems and Installations in Part 23 AirplanesDraft
5. GENERAL CERTIFICATION CONSIDERATIONS.
a. Discussions on crew complement and crew workload should take place between the FAA and the applicant early in the program and part of this discussion should address the required level of testing. Minor modifications to the cockpit may not warrant a workload analysis; however, major modifications may require an extensive evaluation. The level of testing should be consistent with the extent of the modifications. The level of required testing should be developed by the applicant and provided to the FAA for concurrence. These discussions should also focus on identification of design features that are likely to impact crew workload. Resulting analyses, demonstrations and tests should be structured to verify that these design features do not place excessive workload demands on any crewmember or increase error likelihood. Usually, the applicant assumes a given crew size at the beginning of the program and there is little likelihood that a different number will be required. In such cases, evaluations of the minimum crew complement must still be conducted to validate the applicant’s proposed minimum crew complement. Also, these evaluations are necessary to corroborate the applicant’s estimated crew workload by demonstration in a representative operational environment and, thus, support compliance with 14 CFR, part 23, § 23.1523.
b. Minimum crew complement may be established in the beginning of the program considering anticipated pilot-vehicle interface crew workload, individual performance and other suitable factors. Final determination however, will require an evaluation of the fully functioning and integrated system. Because of the direct impact on workload and crew performance, the pilot interface with the cockpit equipment should be evaluated using structured ground tests, mockups and simulators. Applicants are encouraged to include FAA pilots and human factors personnel in these evaluations. We recommend that several flights be conducted in an operationally representative environment. We also recommend the test team be comprised of three to five pilots of varying background and experience to address individual differences in the general pilot population and match the level expected in field service for the aircraft/system under test. A flight test team with diverse backgrounds and experience will expose the system to various piloting techniques and skills. Operationally representative flights can reveal workload excesses that may require workload redistribution or help identify necessary changes in design, operation, training or procedures.
c. Many of the issues related to the human factors design of the cockpit are identified and discussed in the human factors part 23 policy listed in the reference section. Also, previous research and experience have identified several human behaviors that are typically associated with high workload. A list of these behaviors is contained in Appendix 1. It is recommended the test conductor/observer be familiar with the issues discussed in the policy and the list contained in Appendix 1. The flight test evaluator should be vigilant in these areas during testing, noting when and under what conditions any of these behaviors are observed. If any of these behaviors are observed during the evaluations, more detailed examinations of workload may be necessary.
d. When performing analyses and demonstrating workload for minimum crew determinations, several workload factors have been identified as significant in performing this task. These workload factors are listed in Appendix 2 and need to be addressed as part of the evaluations. It is important to include all of the workload factors listed in the appendix when showing compliance to the minimum flight crew rule 14 CFR part 23, § 23.1523.
6. CERTIFICATION PROCEDURES.
a. Evaluation Test Plan. Any new or extensively modified airplane should be evaluated using an FAA approved test plan. Methods for substantiating compliance with § 23.1523 may include the use of analyses, simulation, demonstrations, and/or flight tests. The minimum crew complement workload should be examined through a logical process of analysis, measurement, and demonstration of the workload encountered in the new or modified cockpit.
b. Analytical Approach.
(1) General. The applicant should conduct analyses at the beginning of the design process. Methods of analysis should be selected on the basis of predictive validity, reliability, and applicability to the particular cockpit configuration. When possible, the analyses should include comparisons with other cockpits similar to the cockpit under consideration. The focus of any analysis should be on the modifications or new equipment. More significant modifications, which largely change the cockpit configuration, pilot-vehicle interface, pilot tasking or operating procedures, will require a more comprehensive evaluation. Applicants are encouraged to share the results of these analyses with the certifying office as soon as possible to identify and resolve potential issues.
(2) Examples. One acceptable analytical approach assesses workload as a percentage of the time available to perform tasks (Time-Line Analysis) (other acceptable analytical approaches are described in Appendix 3). This process should be applied to an appropriate set of flight segments in which operationally important time constraints can be identified. This method is satisfactory for evaluation of cockpit changes about overt pilot tasks such as control movements and data inputs. The accepted practice involves careful selection of a limited set of flight scenarios and time segments that represent the range of operational requirements (including the range of selected normal, nonnormal, and emergency procedures). Task time-line analysis yields useful data when tasks must be performed within significant time constraints. An accurate determination of the time available is critical if this method is to have any value. Although time measurements cannot be interpreted to be an absolute standard, such data can be used to identify increased workload demands, which may require a more thorough examination. The impact of cockpit changes on tasks involved with planning and performing emergency or nonnormal procedures should be closely examined. If preliminary analyses by the certification team identify any potential problem areas, a more extensive evaluation should be conducted.
c. Testing.
(1) Overview. Testing to substantiate crew complement and to evaluate crew workload should be an iterative process beginning early in the program. The process should include tests in mockups, bench setups, or simulators and should progress to flight tests in the target airplane with the final configuration. It is useful and/or necessary to incorporate simulator-based testing during the development process to obtain preliminary performance evaluations before flying the prototype system in the airplane. Although not required, it is worthwhile to conduct some evaluations using participants other than company or FAA test pilots (i.e., individuals who are not members of the manufacturer's or FAA certification team). If used, these individuals should be representative of the target population for experience and training, which would be appropriate and expected for the system/airplane. The final determination of the minimum crew complement is to be reserved until the airplane has been flown by pilots with experience appropriate to the aircraft being certified. Paragraph 4 below contains the criteria for determining the minimum flightcrew under § 23.1523.
(2) Scope. The test program should address all workload functions and factors listed in § 23.1523 and Appendix 2. For example, an evaluation of workload should include the communications tasks required to operate properly the airplane in the environment for which approval is sought. The goal is to evaluate workload with the proposed crew complement during realistic operating conditions, including representative air traffic, weather, airline operational duties, and cabin communications.
(3) Evaluation Methodology. There are mainly three workload methods for conducting these evaluations, direct comparison, indirect comparison and standalone evaluation. These are all described below. Comparative evaluations (direct or indirect) examine pilot-system performance and acceptance of the new design with a previously certified similar system that has a demonstrated safety and service history in operational service.
Typically, if the workload for the new system is rated equal to or less than that of the referenced system, then it can be assumed that workload associated with use of the new system is acceptable. Comparative evaluations may be done directly whereby two systems are evaluated against one another for a given set of tasks and flight segments.
Evaluations may also be done indirectly in which only the new system is evaluated but test participants are asked to compare its performance based on their recollection of another system’s performance. When conducting 4
comparative evaluations, caution should be exercised, as it is possible the previously certified system may have function/operation problems or other undesirable characteristics that would not serve as “good” comparative references. For any comparative evaluation, a thorough review of the safety and service history should be conducted to ensure there are no problems or issues with the comparison system. The other method of evaluation is to evaluate a system as a stand-alone component, where no comparison is made to any other system, but rather the system is evaluated based on its own merits. However, the new component must be evaluated as part of the overall cockpit system, since integrating this component may affect task performance and workload of other systems in the cockpit.
The stand-alone approach requires establishment of success criteria for each data collection tool that will be used for the evaluation before the start of testing. For example, if the Bedford Workload Scale is used to collect data on subjective workload, the evaluator should show the scale value at which the workload is considered unacceptable before testing begins. Although collected data may show the workload associated with a given task is unacceptably high, the final determination of whether workload and/or performance warrants changes in design, procedures, and/or training should made by the test team. For items in which only one or two test team members experienced a problem, the team should factor in the skill level, training, learning ability and experience of those individuals. This ensures the differences were not the result of deficiencies in one or more of these areas.
(a) Direct Comparisons. For this approach, the preferred method for conducting a comparative evaluation is to structure tests such that data are collected from evaluation participants performing comparable tasks with both the new design and the previously approved like system. Specific evaluations should be conducted using scenarios designed to exercise fully the new design features. It is important that tasks, to be performed with both systems, be structured and scripted to be as exact and repeatable as possible to promote direct comparisons of pilotsystem performance between the two systems. Also, the interval between the evaluations of the two systems should be kept to a minimum.
(b) Indirect Comparisons. Occasionally, it may not be possible to conduct the multiple evaluations necessary to compare directly one system against another. It is acceptable in such cases to “indirectly” compare the systems by requiring test participants to “mentally” (i.e., based on their memory of a reference system performance and workload) compare their performance with the new design/system against a previously approved like system.
Care should be taken when analyzing results from such comparisons, as the data may not be as accurate or reliable as direct comparison data. So, the conclusions drawn from such testing should be reviewed carefully, checking for reasonableness and consistency.
(c) Standalone Evaluations. This method may be used when a direct or indirect comparison with a reference (baseline) system may not be suitable, if the new design differs radically from the other currently certified systems. For example, if the new design represents a significant change in the level of automation or pilot duties, comparison with a reference design may have limited value. In this case, there is a lack of a baseline for comparison; therefore, the evaluation of the system must be based on its own merits and intended function that has been determined by the applicant and agreed to by the FAA. Evaluation tasks should be well described in adequate detail to exercise fully system functions, implied by their intended function, under all probable operational conditions.
(4) Scenarios. Evaluations should be structured to assure that new systems and cockpit configurations are evaluated using scenarios representative of the kinds of operation for which the airplane is authorized. Although the applicant may provide quantitative substantiating crew workload data, structured subjective evaluations are usually required to verify submitted data in operationally representative environments. These evaluations compare the ease of execution of crew tasks in the subject airplane under identical or substantively similar scenarios. A proposed flight test program for showing compliance with § 23.1523 should address the following factors:
(a) Routes. The test program routes should be modelled to provide a representative mix of navigation aids, airports, instrument approaches and Air Traffic Control (ATC) services.
(b) Weather. The routes should be selected to provide the likelihood of encountering types of adverse weather appropriate to the airplane's intended operation [Instrument Meteorological Conditions (IMC) conditions, night, turbulence, icing, etc.].
(c) Crew Work Schedule. The test crew should be assigned to a daily work schedule that is representative of the operations for which the airplane was developed. The program should include the duration of the workday and the maximum expected number of departures and arrivals, flights that begin at night, maximum allowable duty times, and minimum rest periods.
(d) Minimum Equipment List. The applicant should incorporate representative dispatch configurations in the proposed flight test program. Combinations of these representative dispatch configurations with probable resulting simulated malfunctions should form the basis of many of the evaluation scenarios.
(e) Intended Operations. The airplane should be operated on routes that would adequately sample high-density areas in both IMC and Visual Meteorological Conditions (VMC). These routes would include the types of approaches expected in the intended operations, such as precision/non-precision approaches, missed approaches, holding and diversions to alternate airports.
(f) Incapacitated Crewmember. Most part 23 airplanes are certified for single pilot operations, however there are some exceptions in which two crewmembers are required. In those cases, a workload evaluation for an incapacitated crewmember should be conducted. The certification program should include a demonstration of operations during the total incapacitation of a crewmember at any point in a given flight. It must be shown the airplane can safely operate and land with the remaining crew. In the event of an incapacitated crewmember, the pilot would notify ATC and special considerations would be given to aid the pilot in the approach and landing to a given airport. Therefore, it is not necessary to require performing this demonstration to a “high” density airport, as it would not accurately represent task requirements and demands. Incapacitated crewmember tests need not be additive to all other "dispatch plus subsequent failure" scenarios. Incapacitation should be viewed as another example of "subsequent failure" to be included within one or more scenarios beginning with a dispatch configuration, which includes selected items from the proposed minimum equipment list.
(g) System Failures. The workload outcomes of system failures and degraded modes of operation should be evaluated in the program. Failures of both primary and secondary systems and a representative combination of failures should be included. Data should be collected on pilot recognition time, interpretation accuracy, and the appropriateness of resulting actions to acknowledge, correct and/or compensate for the failure. A sampling of various emergencies and nonnormal conditions should be set up in the test program to show their effect on the crew workload. Data should be collected on the pilot's ability to recognize and take suitable action to emergencies and nonnormal situations. During these evaluations, it is a good time to verify the adequacy of associated checklists and procedures.
NOTE: Before selecting the system failures that will be evaluated in the flight test program, simulation or analytical studies should be conducted to determine which failures are more probable. System failures should be selected based on the likelihood of occurrence. It is not necessary to evaluate those failures that have a low chance of happening (i.e. 1 x 10-9). The crew workload distribution, during the executing of emergency or nonnormal situations, should be understood to assure selection of appropriate failure cases. Guidelines implementing a selected number of subjective, physiological, and performance workload measurement techniques, are contained in the FAA sponsored report "Assessment of Crew Workload Measurement Methods, Techniques, and Procedures" Vol. II (Report No. WRDC-TR-89-7006).
(h) Error Evaluation. The cockpit design should be evaluated to determine if workload is a significant contributor to pilot errors. Control and display design deficiencies could significantly contribute to pilot error. An example of evaluation of this type can be found in Appendix 3 under workload and error (reference Test Operation Procedure 1-2-610, test procedure-error likelihood analysis). An examination of errors should also be made to ensure that when they occur they do not create an unsafe or hazardous condition. Workload and error are discussed further in Appendix 3 with related references.
d. Data Collection and Analysis.
(1) Workload Data. Data may contain objective performance measurements, subjective (perceived) performance estimates collected through questionnaires and pilot comments, and observations recorded by the test conductor, flight test engineer or human factors engineer. Typically, objective data (time-to-perform a task, error 6