U.S. Department of Transportation Federal Aviation Administration Advisory Circular
Subject: Guidance for the Validation of Software Tools Used in the Development of Instrument Flight Procedures by Third Party Service Providers Date: 06/22/2011 Initiated by: AFS-460 AC No: 90-111 Change:
1. Purpose. This advisory circular (AC) provides guidance material for third party Instrument Flight Procedure (IFP) developers, hereafter referred to as “Service Providers,” who use software based tools to develop Performance Based Navigation (PBN) Title 14 of the Code of Federal Regulations (14 CFR) part 97 Standard Instrument Procedures, to have these software tools validated by the Federal Aviation Administration (FAA).
Any service provider who is designated by the FAA as an Other Transactional Authority (OTA) for developing IFPs is required to use validated software tools when developing such procedures.
As an alternative, any IFPs produced using software tools pending validation by the FAA will be subject to additional scrutiny, leading to delays in implementing those IFPs. Thus, it is important for all service providers to have their software tools validated, and to maintain the validation as new software tool versions and new criteria are introduced.
2. Applicability.
a. Service Providers may elect to use the guidance in this advisory circular or follow an alternative method, provided that the method is approved by the FAA.
b. Mandatory terms used in this AC such as “must” are used to denote that the particular means of compliance described herein are the only means authorized when applying this AC.
This AC does not change, add, or delete regulatory requirements or authorize deviations from regulatory requirements.
c. The software validation process is applicable to software systems that are intended to aid in the production of performance-based IFPs for which the criteria requires automated computations. These IFPs include, but are not limited to, Area Navigation Approach (both fixedand rotary-wing) and Required Navigation Performance (RNP) Special Aircraft and Aircrew Authorization Required (SAAAR)/Authorization Required (AR) Approach procedures.
3. Audience. The primary audience for this advisory circular is prospective Third Party Service Providers who have been authorized by the FAA to develop 14 CFR part 97 Standard Instrument Procedures. The secondary audience for this advisory circular is FAA Flight Standards Service (AFS) personnel, who are charged with the responsibility to qualify and provide oversight of non-governmental Service Providers.
4. Background. Historically the FAA has relied on internal resources to design and develop public IFPs. Recently, guidance has been promulgated that enables Service Providers to gain FAA authorization to design and develop IFPs. The ability to design IFPs relies on the availability of procedure design software and software based aiding tools that correctly implements FAA IFP design criteria. This advisory circular describes the process by which Service Providers must demonstrate that their software tools follow FAA intent in the application of IFP design criteria.
5. Definitions.
a. Instrument Flight Procedure (IFP) is a charted flight path defined by a series of navigation fixes, altitudes and courses provided with lateral and vertical protection from obstacles from the beginning of the path to a point from which a landing can be completed, and if a landing cannot be completed, then continuing on to a position and altitude at which either holding or en route flight can be continued.
b. Service Provider , for the purpose of this AC, is an individual or organization who develops or intends to develop instrument flight procedures.
c. Software System is a computer program that carries out computations or displays aviation-related information to assist the user in developing instrument flight procedures.
d. Aiding Tool is a software system that implements some aspect of instrument flight procedure design, but does not integrate all of the procedure design functions in a single system.
An example of an aiding tool would be a spreadsheet calculator that calculates one or more design formulas.
e. Expert Tool is a software system that provides a highly integrated instrument procedure design environment in which many or all criteria-specific computations are carried out automatically.
f. Companion Software Requirements Specification (CSRS) is a document that describes how FAA criteria from Orders, Notices, and Policy Memoranda are to be expressed in software tools.
6. Related Publications.
a. Regulations. Title 14 Code of Federal Regulations (14 CFR) part 97.
b. FAA Orders (current editions). Copies of the following Orders may be obtained from the U.S. Department of Transportation, Publications Department, Ardmore East Business Center, 3341 Q 75th Avenue, Landover, MD, 20785.
(1) FAA Order 8260.42, United States St andard for Helicopter Area Navigation (RNAV).
(2) FAA Order 8260.52, United States Sta ndard for Required Navigation Performance (RNP) Approach Procedures with Special Aircraft and Aircrew Authorization Required (SAAAR).
(3) FAA Order 8260.54, United States Standard for Area Navigation (RNAV).
c. Reading Material (current editions). Copies of the following documents may be obtained directly from the FAA Flight Technologies and Procedures Division, Flight Procedure Standards Branch (AFS-420) or the United States Instrument Flight Procedures Panel (US-IFPP), Automation Working Group (AWG) websites. These documents address performance-based Instrument Flight Procedures (IFPs) implementation through automation and software quality assurance requirements.
(1) International Civil Aviati on Organization (ICAO) Doc 9906-AN/472, The Quality Assurance Manual for Flight Procedure Design Volume 3—Flight Procedure Design Software Validation, 29 November 2007.
(2) FAA CSRS to FAA Order 8260.52
(3) FAA CSRS to FAA Order 8260.54A
(4) FAA CSRS General Requirements
(5) US-IFPP, AWG Terms of Reference
7. Scope. The purpose of the FAA software validation process is to ensure that software tools for instrument flight procedure development correctly implement FAA criteria, as embodied in FAA Orders, Notices, Policy Memoranda, and Companion Software Requirements Specifications.
a. The software capabilities that will be tested include the following:
(1) Formulas.
(2) Geodetic Constructions.
(3) Obstacle Identification and Mitigation.
(4) Minima Computations.
b. Capabilities and aspects that are outside the scope of the software validation and will not be tested include the following:
(1) User Interface.
(2) Data Sources.
(3) Robustness.
(4) Usability.
8. FAA Responsibilities.
a. Instrument Flight Procedure Criteria. The FAA will publish IFP criteria in the form of orders, notices, and policy memoranda. These documents will prescribe the requirements that must be followed to develop compliant instrument flight procedures.
b. Instrument Flight Procedure Compan ion Software Requirement Specifications. The FAA will provide software requirements documents which accompany the instrument flight procedure criteria. The documents will prescribe how the criteria must be implemented by software tools. The software requirements specifications will consist of the following:
(1) Input data requirements that list all da ta elements that are needed to describe and evaluate an instrument flight procedure in accordance with the software validation process.
(2) Software requirements that state wh ich calculations must be performed by the software tool.
(3) Example algorithms that illustrate how certain calculations should be performed.
(4) Output data requirements that list all data elements that must be reported by the software tool to illustrate compliance with the software requirements.
c. Instrument Flight Procedure Knowledge Repository. The FAA will provide a webbased interface for obtaining clarification of criteria and software requirements. The knowledge repository will be accessible from the AFS-420 website.
d. Software Validation Process Website. The FAA will provide a website, accessible from the FAA Flight Technologies and Procedures Division, Flight Procedure Implementation and Oversight Branch (AFS-460) website, which will facilitate the software validation process. This website will enable Service Providers to apply for and track the validation status of a software tool. The website will also be the means of distribution of validation test case inputs and reference test results.
e. Oversight. AFS-460 is responsible for administering the software validation process, maintaining the software validation website, and determining the validation status of each software tool. AFS-460 may confer, suspend, or revoke validation status.
f. United States-Instrument Flight Pr ocedure Panel (IFPP), Automation Working Group (AWG). The US-IFPP AWG is a panel composed of representatives from the FAA, Service Provider organizations, and other invited parties that will advise the IFPP on software validation issues.
9. Service Provider Responsibilities.
a. All software tools. Regardless of the type of software tool, the Service Provider must provide certain documentation to the FAA via the software validation website to demonstrate that the software tool was developed in accordance with sound software engineering principles.
This documentation must include the following items:
(1) The software tool version number and creation date.
(2) Test reports generated by the Service Provider and/or software developer that prove that the software tool was tested in accordance with ICAO software quality assurance manual recommendations.
(3) A configuration management plan that demonstrates how new versions of the software tool will be introduced.
b. Aiding Tools. For an aiding tool to be considered for validation, the Service Provider must provide an operational copy of the aiding tool, documentation of any tests conducted and necessary usage manuals to the FAA to facilitate independent testing.
c. Expert Tools. A Service Provider should ensure that its expert tool is capable of loading test procedure data sets, evaluating the procedures described within, and outputting defined test result data in the appropriate formats. The input and output formats are described in the Companion Software Requirements Specifications. Expert tools not capable of importing test procedure data sets and exporting the evaluation results will have to be manually validated. The Service Provider will be responsible for executing the FAA-provided validation tests using the candidate software tool.
d. Appeal. A Service Provider may appeal a denial or revocation of validation status by notifying AFS-460 in writing.
e. Configuration management. Service providers must ensure that the version of software tool in use is identical to the version that was approved for validation. When new software tool releases are made, the new version of the software tool must be submitted for validation.
Similarly, when a new version of criteria is published superseding or canceling existing criteria, any software tool that was validated against the existing criteria must be revalidated before generating IFPs under the new criteria.
Jolen M. Allen John M. Allen Flight Standards Service 5
THIS PAGE IS INTENTIONALLY LEFT BLANK
Appendix A A-1 Appendix A. Software Validation Process Description
1. General. This appendix provides a description of the software validation process to guide Service Providers who want to participate. It describes the steps that must be followed to obtain a software validation website account, enter a software tool for consideration, and manage validation status of existing software tools. Also, see figures 1 and 2.
2. Software Validation Website. The software validation process is managed using the AFS Software Validation Website (hereafter referred to as “the website”). The website is accessible from a link on the AFS-460 website.
3. Software Development. FAA will provide Companion Software Requirements Specifications (CSRS) that describe how criteria shall be implemented in aiding or expert tools.
Service Providers should ensure that their software tools are developed in accordance with the CSRS and other criteria documents. This will enable the most efficient validation testing process.
4. Account Application. The website is username/password protected. In order to gain access, one representative from the Service Provider must apply for an account. Each software tool can be linked to only one account, so multiple accounts per Service Provider are not necessary and will not be granted. Once the application for an account is made, AFS-460 will verify that the applicant is a valid Service Provider, and will approve the account, if appropriate.
Note that the website uses encrypted usernames and passwords and communicates via an encrypted connection. Therefore, all information transmitted to or from the website is protected, in accordance with standard web practices. However, absolute security cannot be guaranteed.
Therefore, each Service Provider must accept the risk that the information they provide via the website could be accessed by unauthorized persons.
5. Enter Software Tool Details for Validation Consideration. Once access to the website is granted, the Service Provider can enter specific details on the software tool intended for validation consideration. Details for multiple software tools may be entered. These software tools may include any combination of aiding tools and expert tools. Each software tool entered in the website for validation consideration must be labeled as either an aiding tool or expert tool.
The validation process differs for the two types of software tools, so the function of the website depends on how each software tool in is labeled. The version of the software tool must also be provided. If the aiding tool or expert tool version changes, the website will automatically flag the system for revalidation.
6. Supply Supporting Documentation. Regardless of the software tool type, supporting documentation must be provided that demonstrates that the Service Provider and/or software developer has conducted tests of the software tool.
7. Aiding Tools. For an aiding tool to be considered for validation, an operational copy of the aiding tool must be provided to FAA. This may be uploaded to the website or sent to FAA by some other means (e.g., e-mail, physical media, etc.). Documentation that explains how to use the tool must also be provided to ensure that independent tests are executed correctly.
Appendix A A-2
a. AFS Validation Testing. Once entered into the validation process, aiding tools will be tested by AFS-460 to verify correct operation according to FAA criteria.
b. Validation Test Reports. For each aiding tool, results of the AFS-460 validation tests will be saved to the website in the form of a validation test report. These reports may be reviewed by the Service Provider to determine why validated status was or was not granted.
8. Expert Tools. Due to their expected complexity, expert tools will not be provided to the FAA. Rather, the FAA will provide test inputs, either automated or manual, that the Service Provider must evaluate using the expert tool. Results of the test evaluations will then be uploaded to the website for comparison with reference test results.
a. Specify Procedure Type. The procedure type must be specified for each expert tool. The available procedure types are listed on the website.
b. Specify Data Transfer Capabilities. Each expert tool will be classified as either having been developed in accordance with current criteria and CSRS documents, to include a defined XML schema to automate data transfer, or systems that have no data transfer capability. The input and output data format is prescribed in the CSRS for the procedure type to be tested using automated test sets. Systems that cannot input automated test sets will require manual validation testing.
c. Download Reference Test Sets.
(1) Automated Evaluation: A set of reference test inputs for each procedure type shall be downloaded from the website.
(2) Manual Evaluation: A set of IFPs, with all associated data for each procedure type shall be downloaded from the website.
d. Evaluate Test Sets.
(1) Automated Evaluation: Th e expert tool must be used to evaluate the reference tests procedures and generate test output data for each test input set.
(2) Manual Evaluation: The expert tool must be used to evaluate the reference tests manually and a procedure report must be generated for each procedure.
e. Upload and Compare Test Results. Test results for either automated or manual test sets shall be uploaded to the website, where they will be compared to reference test results.
f. Validation Test Report. Differences between the expert tool and reference outputs will be summarized and displayed to the Service Provider in the form of a validation test report.
Appendix A A-3
9. Determine Approval. Items that will be evaluated during the approval process will be the extent of testing, and any associated documentation, conducted by the software developer and/or Service Provider and the results of the automated and/or manual validation tests. A validation test report for each software tool will be reviewed by a Flight Standards manager who will determine whether or not to approve the software tool. Approval may be denied, or three different approval levels may be granted, depending on the completeness of the Service Provider’s supporting documentation and the results of the validation tests. A signed letter will be sent to the Service Provider explaining the approval decision and the approval status of the software tool will be recorded on the website.
a. Approval Level 1 will be conferred upon software tools that minimally pass validation tests. IFPs developed using Level 1 software tools will require significant oversight prior to publication.
b. Approval Level 2 will be conferred upon software tools that pass all validation tests but also demonstrate some acceptable deviations from FAA criteria (for example, alternate application of tolerances). IFPs developed using Level 2 software tools will require less oversight prior to publication than those produced using Level 1 software tools.
c. Approval Level 3 will be conferred upon software tools that pass all validation tests and fully and correctly implement FAA criteria. IFPs developed using Level 3 software tools will require minimal oversight prior to publication.
10. Appeal of Validation Decision. If the Service Provider believes that the approval decision was reached in error, then the decision may be appealed in writing. Upon appeal, the validation test results will be reviewed and the reference test results may be checked for correctness. Your request for reconsideration must be made in writing within 30 calendar days from the date of the validation denial letter and must include all related information and arguments. Your request should be sent to the following address:
Manager, Flight Technologies and Procedures Division, AFS-400 470 L’Enfant Plaza, SW Suite #4102 Washington, DC 20024
11. Revocation. There are various reasons why software tool approval may be revoked. These reasons include the discovery of errors in criteria, discovery of errors in reference test results, or further investigation of software tool performance. In this case, the status of the software tool will be changed on the website and a notification will be sent to all affected Service Providers.
These Service Providers will be required to make any needed changes to their software tool and repeat the validation process.
12. Revalidation. If a new software tool version is released or if criteria documents are updated, previously approved software tools must be revalidated to ensure that they continue to correctly implement criteria.
Appendix A A-4
THIS PAGE IS INTENTIONALLY LEFT BLANK
Appendix A Figure 1: The AFS Validation Process - Aiding Tools FAA Orders, Memos and CSRS Develop Aiding Tool Step 1: → Perform Test Results Step 5:
Comparison Step 2:
Apply for Website Account and Start Process Modify Fail Validation Report Fail Aiding Tool _ _ Pass Step 7:
Step 3: Determine Provide Aiding Tool Validation Level For Validation Tests Step 8:
Issue Approval Step 4: Letter Execute Validation Tests Step 9:
Appeal Decision Legend Service Provider Action AFS Action Optional Step A-5
Appendix A Figure 2: The AFS Validation Process - Expert Tools FAA Orders, Memos and CSRS Submit Test Results Step 5:
Output XML or Step 1: Manual Test Results Develop Software Step 6:
Perform Test Step 2:
Results Apply for Website Comparison Account and Start Process Modify Fail Validation Fail Software Report Pass Step 3:
Download Tests Step 7:
Determine (Procedure Specific) Validation Level Input XML or Step 8:
Manual Tests Issue Approval Letter Step 4:
Execute Validation Step 9:
Tests Appeal Decision Legend Service Provider Action AFS Action Optional Step A-6