Appendix 3
APPENDIX 3. WORKLOAD EVALUATION ANNOTATED REFERENCES (CONTINUED)
Workload and Error
1. Test Operation Procedure (TOP) 1-2-610, May 1990, Section 5.12 Test Procedure-Error Likelihood Analysis..
This document contains an excellent discussion that describes a technique for analyzing the adequacy of the design of controls and displays in relationship to human error. It correlates error likelihood rates with the degree to which the controls and displays conform to human factors design criteria. A basic premise of this approach is that error rates and workload will increase with designs that fail to properly address existing human factors design criteria. A simplistic but effective approach is to measure error probability likelihood by evaluating human factors systems design criteria, developed in a checklist format, and then compare that against the actual cockpit/system design under evaluation. The Error Likelihood Analysis section, Design Checklists and Workload Assessment techniques has been made available through the human factors web site for the small airplane directorate at the following url
http://www.faa.gov/certification/aircraft/aceMeasurementTech.pdf
In addition, the entire report can be obtained from the DTIC web site at http://handle.dtic.mil/100.2/ADA226480.
2. GAMA Publication No. 10. Part 23 Best Practices for Cockpit/Flight Deck Design
This document contains several examples of good practices to employ in designing the user interface to reduce human error. It also identifies design practices to consider in order to mitigate the consequences of errors. The document includes discussion of how to use usability testing to assist in reducing error in the design of the system and reduce the seriousness of errors made in the cockpit. This document is also available from the GAMA web site, http://www.gama.aero/pubs/itemDisplay.php?catalogID=26
3. Part 23 Human Factors Policy
This document provides several discussions regarding what to consider when performing human factors cockpit evaluations to include examining human error and the consequences of those errors. This document is available from the Small Airplane Directorate web site under human factors.
4. DOT/FAA Human Factors design guidance DOT/FAA/CT-96/1, JAN 15, 1996
This design guide discusses human error in Chapter 5. The guidance recommends assessment of complex systems that address human reliability, failure and risk assessment. It recommends that systems analysis treat humans as major components, consider human interfaces as contributing elements and treat human performance. The guidance also addresses the importance of error tolerant designs and proposes evaluation of complex systems through rapid prototyping, analytic methods or modeling.
5. Rouse, W. and Rouse, S (1983). Analysis and classification of human error. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13(4), 539-549.
Abstract: The literature on human error is reviewed with emphasis on theories of error and classification schemes. A methodology for analysis and classification of human error is then proposed which includes a general approach to classification. Identification of possible causes and factors that contribute to the occurrence of errors is also considered. An application of the methodology to checklists in the aviation domain is presented for illustrative purposes. (60 References)
A3-7
Appendix 3
APPENDIX 3. WORKLOAD EVALUATION ANNOTATED REFERENCES (CONTINUED)
6. Swain, A. and Guttman, H. (1983). Handbook of human reliability analysis with emphasis on nuclear power plant applications (NUREG/CR-1278). Washington, D.C.: Nuclear Regulatory Commission.
This handbook describes methods, models, and estimated human error probabilities that allow an analyst to make quantitative or qualitative assessments of occurrences of human errors. These examples were developed for operations involving operation of nuclear power plants. This handbook also shows how to recognize error likely-equipment design, procedures, and other human factors problems so that improvements and interventions can be considered. Although written specifically for nuclear power plants much of the information is applicable to designs of other equipment including those developed for aviation.
A3-8
Appendix 4
APPENDIX 4. RECOMMENDED TEST PLAN GUIDANCE
AND DATA COLLECTION INFORMATION
1. DATA COLLECTION TECHNIQUES - Guidelines concerning the implementation of a selected number of subjective, physiological, and performance workload measurement techniques, are contained in the FAA sponsored report "Assessment of Crew Workload Measurement Methods, Techniques, and Procedures" Vol. II (Report No.
WRDC-TR-89-7006).
a. Selecting Performance Measures, Defining Criteria. The criteria to evaluate the system against are set either by the performance of the reference system or by the acknowledged intended function of the system.
Performance measures are developed directly from the criteria.
b. Questionnaire Development
(1) Construction/Selection of Workload Scales. Numerous workload rating scales have been developed over the years.
(2) Wording/Biasing Considerations and Capturing Key Aspects of System Workload Contributors.
The structure and manner in which questions are phrased can significantly affect responses.
c. Types of Evaluations – Comparative Versus Absolute Evaluations
(1) Obtaining baseline data and making comparisons using each participant as a comparative evaluator (testpilot evaluation)
(2) Obtaining individual performance data for a baseline and a new system
(3) Collection and Analysis of Error Data
2. FORMS. Data collection forms should be developed and provided to the FAA prior to the start of testing to demonstrate compliance to 14 CFR, part 23, § 23.1523. The applicant and the FAA should work together to ensure that data collection forms collect pertinent, meaningful data that can be used to draw conclusions about minimum crew, system workload and usability. It is generally recommended that forms be developed to query and record pilot ratings and observations, and to record in-flight observations, to include crew performance, crew errors, missed communications, and problems with checklists and flight management or flight guidance systems. When possible, the airplane should be configured to allow observers to view crew actions and hear all (i.e. internal and external) communications. It is preferable to collect data immediately after the performance of a task, as the greater the period of time between an event and the recording of data, the greater the likelihood information may be inaccurate or lost. Pilot comments are an excellent source of data; pilot comments should be encouraged and recorded throughout testing. Also, particularly during the performance of more complex tasks, pilots should be encouraged to speak their thoughts out loud as they step through the performance of a task. Members of the certification team who will serve as evaluation pilots and observers should be familiar with the data collection forms that will be used for the project prior to the start of testing. It is highly recommended that a pretest be conducted using proposed data collection forms to ensure that forms are usable and effective.
3. DEBRIEF. After the completion of each test flight, a post-flight debrief should be conducted. The test conductor should review the events of the flight and go over recorded comments with the pilot. The pilot should be given the opportunity to clarify comments and discuss salient issues with the design/system. If a post-flight questionnaire is to be given, it should be provided to the pilot prior to the post-flight debriefing.
4. DATA REDUCTION. The data collected during the course of testing will need to be reduced and summarized.
Typically, statistical analysis of data is not warranted due to small sample sizes and limited testing. It is preferable to group data by topic and summarize the results in tables, indicating, for example, the number of responses for each rating (e.g. 1-Much Less Than, 2-Somewhat Less Than, 2-Equal To, 1-Somewhat More Than, and 0-Much More Than). Likewise, comments and observations should be grouped by topic and differentiated based on positive
A4-1
Appendix 4
APPENDIX 4. RECOMMENDED TEST PLAN GUIDANCE
AND DATA COLLECTION INFORMATION (CONTINUED)
versus negative comments. All areas of “increased” or “unacceptably high” workload must be clearly identified and investigated. It should be noted that a finding of increased workload (over a reference system) does not necessarily constitute an unacceptable situation (certification issue), but rather indicates an area requiring further examination to ensure that workload is acceptable. The report should include a copy of all completed data collection forms and any recorded comments or observations. The applicant should provide a disposition in the report for each certification issue identified during testing. The applicant may choose to provide the test results only and permit the FAA draw the conclusions, or propose conclusions from the data collected. In the event the applicant provides conclusions in the test report, it needs to be clear that the FAA is the final authority in terms of the acceptability of the design and compliance findings.
5. LEVEL OF TESTING. The level and amount of testing should be commensurate with the level of changes and differences between the previous system (if there is one) and the new system/cockpit. In case where only a minor change has been made to a component or cockpit configuration, or where a single piece of equipment has been added to an existing approved configuration, workload analyses should be tailored to focus on the function and operation of the specific component and its interactions with other cockpit equipment. However, if there have been significant changes made to the cockpit configuration, then more extensive workload evaluations should be conducted.
6. QUESTIONNAIRE DEVELOPMENT
a. This section is not meant to be a tutorial on questionnaire construction, but rather to highlight some of the areas and issues that should be considered when developing a questionnaire for collecting data on crew workload and performance
b. A questionnaire is a subjective measurement tool for systematically eliciting perceptions or attitudes from selected personnel. Questionnaires often form an important part of the data collection effort. The major benefit of a questionnaire is that each test participant is asked the same questions in the same way, making it easier to compare results from different participants. It also forces the participants to focus on and critically evaluate the key areas and issues with a new design/system. When properly designed, the questionnaire also aids in the data tabulation and analysis. Questionnaires can be used to assess a wide variety of qualitative variables such as acceptance, ease of use, and preference. Unfortunately, some of the very characteristics that make questionnaire use so beneficial can also be detrimental.
c. Poorly researched and constructed questionnaires can cause evaluators to focus on less important (less critical) system attributes and therefore ignore or dismiss some of the key issues with the system. One of the more important areas, and also the most difficult, of questionnaire construction is wording the questions. Most authorities agree that faulty or inappropriate question wording is the greatest source of error in the use of questionnaires.
Misunderstanding and misinterpretation of questions by test participants often cause errors and distortions in the resulting data. Such misunderstanding is frequently due to improper vocabulary or ambiguous phrasing.
d. Wording must be carefully selected to stimulate thought and solicit answers relevant to the system under evaluation. It is usually best if questions are worded such that there is no hint or suggestion of a “desired” answer.
Questions should be neutrally stated, signifying that any answer is acceptable. For example, a question concerning the performance of a task could be stated as: Capability to insert user-defined waypoints using system “Y” was:
Much Easier, Somewhat Easier, About the Same, Somewhat More Difficult, Much More Difficult than performing this task with system “X”. The evaluator would be asked to circle the applicable phrase. A second method for constructing questions is to provide a mix (usually randomly distributed) of positive and negative statements and request the evaluator to rate their level of agreement or disagreement with the statement. For example, It was much easier to insert a user-defined waypoint in system “Y” than it was with system “X”. The evaluator is asked to rate their level of agreement or disagreement with that statement using a scale, which may be worded: Strongly Agree, Somewhat Agree, Neither Agree or Disagree, Somewhat Disagree and Strongly Disagree.
A4-2
Appendix 4
APPENDIX 4. RECOMMENDED TEST PLAN GUIDANCE
AND DATA COLLECTION INFORMATION (CONTINUED)
e. Even with the most carefully developed questionnaires, there are bound to be some areas that are missed or not adequately covered. It is, therefore, recommended that every questionnaire include a comment form, to allow the evaluator to freely express his/her opinion on system attributes. Also, another technique for collecting feedback from participating personnel is to provide a “Recommended Improvements” form. This approach may aid in identifying areas where improvements may be made, but do not constitute enough of an issue to warrant mentioning elsewhere.
7. RATING SCALES
a. Rating scales are an effective method to quantify evaluator responses. Many different types scales have been developed over the years for a variety of purposes. Typically, descriptor adjectives are used to define scale endpoints and intermediate graduations. Descriptor adjectives may characterize the quality (good to bad) of a system attribute; provide a measure of equivalency to another system (better than, about the same, worse than) or distinguish levels of adequacy or acceptability (e.g. Very Acceptable, Very Unacceptable). It is not the FAA's goal to precisely quantify crew workload or performance, but rather to ensure that the system provides for “acceptable” levels of crew workload and performance, and that no unsafe conditions exist. Consequently, rating scales can be less sophisticated. Some examples of acceptable rating scales are:
Completely Acceptable Completely Adequate Reasonably Acceptable Reasonably Adequate Borderline Bo rderline Extremely Unacceptable Completely Inadequate
Considerably Better Much Better Somewhat Better Somewhat Better About the Same No Better – No Worse Somewhat Worse Somewhat Worse Considerably Worse Much Worse
Much Less Very Easy Somewhat Less Somewhat Easy Same Borderline Somewhat More Somewhat Difficult Much More Very Difficult
Very Good Strongly Agree Good Lar gely Agree Fair (or So-So) Neutral Poor Som ewhat Disagree Very Poor Strongly Disagree
b. A significant number of workload rating scales have been developed and used by government and industry alike. Many of these scales have been well tested and validated to be effective measure of workload, being very sensitive to slight differences in perceived workload. Any of the scales discussed in the “Assessment of crew workload measurement methods, techniques and procedures,” (Volumes 1 and 2), may be used to collect workload data. Additionally, rating scale descriptors discussed previously may be used as well providing that questions are appropriately phrased for the selected scale descriptors. Figure 1 below is an example of a data collection form used to collect pilot-vehicle interface and workload data. Figures 2 and 3 provide examples of forms that may be used to collect pilot/user opinions and feedback.
A4-3
Appendix 4 APPENDIX 4. RECOMMENDED TEST PLAN GUIDANCE
AND DATA COLLECTION INFORMATION (CONTINUED) Pilot comments:
FIGURE 1 - FORM TO COLLECT PILOT VEHICLE INTERFACE AND WORKLOAD DATA Test Objective: Unusual Attitude Recovery Over Water - Evaluate the capability of the pilot to recognize current aircraft situation and take timely appropriate corrective action to recovery from an unusual attitude. Test Procedure: While over water with only water in view on the Vision 1 system, the safety pilot will put the aircraft in an unusual attitude (nose down) while the test pilot looks at his/her lap. Test pilot to recover aircraft to Other Conditions: Rating Values VP Much Less More Same More Rating scale values are VG- Very Good, G- Good, F- Fair, P- Test Point No.: 1 level flight with minimum altitude loss Airspeed: Altitude: As ≥3000ft required agl Question VG 1. Capability to recognize relative aircraft attitude and situation at start of test: 2. Capability to recognize, at a glance, the horizon: 3. Performance of display for unusual attitude recovery: Much 4. Workload Rating (Check appropriate box) Less Poor, and VP- Very Poor.
Appendix 4
APPENDIX 4. RECOMMENDED TEST PLAN GUIDANCE
AND DATA COLLECTION INFORMATION (CONTINUED)
FIGURE 2 – COLLECT PILOT/USER COMMENTS AND FEEDBACK
COMMENTS
Pilot’s Name: _______________________ Date: _________ Phone: ____________________
General Complexity and Usability:
_______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ Option/Function Selection Procedures:
_______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ Data Entry Procedures:
_______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ System Feedback:
_______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ Display Characteristics:
_______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ Warnings, Cautions and Alerts:
_______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ Other (Specify) ____________:
_______________________________________________________________________________ _______________________________________________________________________________ A4-5
Appendix 4
APPENDIX 4. RECOMMENDED TEST PLAN GUIDANCE
AND DATA COLLECTION INFORMATION (CONTINUED)
FIGURE 3 - COLLECT PILOT/USER OPINIONS AND FEEDBACK
Recommended Improvements
Pilot’s Name: _______________________ Date: _________ Phone: ____________________
Design and Operation of Controls:
_______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ Option/Function Selection Procedures:
_______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ Display Characteristics:
_______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ Warnings, Cautions and Alerts:
_______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ Other (Specify) ____________:
_______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________ A4-6