C Advisory U.S. Department of Transportation Federal Aviation Circular Administration Subject: Describing Flight Safety Analysis Date: 09-20-2024 AC No: 450.115-2 Methods Initiated By: AST-1 This Advisory Circular (AC) provides guidance for documenting the methods used in a flight safety analysis in accordance with title 14 of the Code of Federal Regulations (14 CFR) § 450.115(c). This guidance is not legally binding in its own right and will not be relied upon by the Federal Aviation Administration (FAA) as a separate basis for affirmative enforcement action or other administrative penalty. Conformity with the guidance is voluntary only and nonconformity will not affect rights and obligations under existing statutes and regulations.
If you have suggestions for improving this AC, you may use the Advisory Circular Feedback form at the end of this AC.
Daniel Murray Executive Director, Office of Operational Safety Commercial Space Transportation
09-20-2024 AC 450.115-2 Contents Paragraph Page
1 Purpose1
2 Applicability2
3 Applicable Regulations And Related Documents3
4 Definition of Terms6
5 Acronyms7
6 Introduction8
6.1 Relationships between paragraphs in § 450.1158
7 Explanation of § 450.115(c)9
7.1 Scientific Principles and Statistical Methods Used9
7.1.1 Scientific Principles9
7.1.2 Statistical Methods10
7.2 Assumptions and Justifications10
7.2.1 Scope of the Method11
7.2.2 Physical Phenomena11
7.2.3 Data12
7.2.4 Human Operators12
7.3 Level of Fidelity12
7.3.1 Bias and Uncertainty13
7.3.2 Rationale13
7.4 Verification and Validation13
7.4.1 Verification14
7.4.2 Validation14
7.4.3 Standards14
7.4.4 Rigor Based on Level of Criticality15
7.4.5 Evidence15
ii
09-20-2024 AC 450.115-2
7.4.6 Off-the-shelf Items15
7.5 Benchmarks16
7.5.1 Choosing Benchmarks16
7.5.2 Comparison to Benchmarks16
7.6 Risk Mitigations17
8 Standard of Sufficiency18
8.1 Content18
8.1.1 Logic and Mathematics18
8.1.2 Topics18
8.2 Validity18
8.2.1 Accurate Data19
8.2.2 Accurate Scientific Principles19
8.2.3 Valid statistical methods19
8.2.4 Consistency with Past Results20
8.3 Editorial Requirements20
8.3.1 Glossary20
8.3.2 Reference List21
8.3.3 Derivations21
8.3.4 Notation21
8.3.5 Units21
8.3.6 Diagrams21
8.4 Depth of Detail22
8.5 Review and Revision Process22
8.5.1 Maturity Rating23
8.5.2 Response to Feedback24
8.5.3 Recommendations24
8.6 Example24
8.6.1 Insufficient Description25
iii
09-20-2024 AC 450.115-2
8.6.2 Sufficient Description25
Tables Number Page
Table 1 - Methodology Maturity Levels23
Figures Number Page
Figure 1 - Method Description with Insufficient Detail25
Figure 2 – Method Description with Sufficient Detail26
iv
1 PURPOSE.
This Advisory Circular (AC) provides guidance for documenting and submitting a description of the methods used in a Flight Safety Analysis (FSA) in accordance with title 14 of the Code of Federal Regulations (14 CFR) § 450.115(c). Engineers developing and documenting engineering methods should use this AC to understand the requirements in § 450.115(c). Regulatory compliance leads should use this AC to ensure submissions thoroughly respond to requirements. FAA evaluators should use this AC as a guide to evaluate submissions. All readers should have a general understanding of flight safety analysis and of standards for scientific and engineering documentation.
1.1 Level of Imperatives.
This AC presents one, but not the only, acceptable means of compliance with the associated regulatory requirements. The FAA will consider other means of compliance that an applicant may elect to present. In addition, an operator may tailor the provisions of this AC to meet its unique needs, provided the changes are accepted as a means of compliance by the FAA. Throughout this document, the word “must” characterize statements that directly follow from regulatory text and therefore reflect regulatory mandates. The word “should” describe a requirement if electing to use this means of compliance; variation from these requirements is possible but must satisfy the regulation to constitute an alternative means of compliance. The word “may” describe variations or alternatives allowed within the accepted means of compliance set forth in this AC.
2 APPLICABILITY.
2.1 The guidance in this AC is for launch and reentry vehicle operators required to comply with 14 CFR part 450. The guidance in this AC applies to operators seeking a launch or reentry vehicle operator license, a licensed operator seeking to renew or modify an existing vehicle operator license, and FAA commercial space transportation evaluators.
2.2 The material in this AC is advisory in nature and does not constitute a regulation. This guidance is not legally binding in its own right, and the FAA will not rely upon this guidance as a separate basis for affirmative enforcement action or other administrative penalty. Conformity with this guidance document (as distinct from existing statutes and regulations) is voluntary only, and nonconformity will not affect rights and obligations under existing statutes and regulations.
2.3 The material in this AC does not change or create any additional regulatory requirements, nor does it authorize changes to, or deviations from, existing regulatory requirements.
3 APPLICABLE REGULATIONS AND RELATED DOCUMENTS.
3.1 Applicable United States Code (U.S.C.) Statute.
• Title 51 U.S.C. Subtitle V, Chapter 509, Commercial Space Launch Activities.
3.2 Related Code of Federal Regulations.
The following Title 14 CFR regulations should be accounted for when showing compliance with 14 CFR § 450.115(c). The full text of these regulations can be downloaded from the U.S. Government Printing Office e-CFR. A paper copy can be ordered from the Government Printing Office, Superintendent of Documents, Attn: New Orders, P.O. Box 371954, Pittsburgh, PA, 15250-7954.
• Section 401.7, Definitions.
• Section 450.35, Means of Compliance.
• Section 450.45, Safety Review and Approval.
• Section 450.101, Safety Criteria.
• Section 450.108, Flight Abort.
• Section 450.113, Flight Safety Analysis Requirements—Scope.
• Section 450.117, Trajectory Analysis for Normal Flight.
• Section 450.119, Trajectory Analysis for Malfunction Flight.
• Section 450.121, Debris analysis.
• Section 450.123, Population Exposure Analysis.
• Section 450.131, Probability of Failure Analysis.
• Section 450.133, Flight Hazard Area Analysis.
• Section 450.135, Debris Risk Analysis.
• Section 450.137, Far-field Overpressure Blast Effects Analysis.
• Section 450.139, Toxic Hazards for Flight.
3.3 Related FAA Advisory Circulars.
FAA Advisory Circulars are available through the FAA website, Advisory Circulars (ACs) – AST. The following Advisory Circulars contain information about the specific content of a methods descriptions in flight safety analysis:
• AC 413.13-1, Guidance on Submitting a Complete Enough and Complete Application for a Vehicle Operator License, dated December 18, 2023.
• AC 450.101-1B, High Consequence Event Protection, dated May 3, 2024.
• AC 450.108-1, Using Flight Abort Rule as a Hazard Control Strategy, dated July 7, 2027.
• AC 450.115-1B, High Fidelity Flight Safety Analysis, dated August 2, 2024.
• AC 450.117-1, Normal Trajectory Analysis for Normal Flight, dated August 19, 2021.
• AC 450.123-1, Population Exposure Analysis, dated October 12, 2022.
• AC 450.141-1A, Computing System Safety, dated August 16, 2021.
Note: Additional ACs are planned for other flight safety analysis sections of the regulation.
3.4 Additional References Related to Rigor of Flight Safety Analysis Methodology.
1. National Institute of Science and Technology (NIST). NIST/SEMATECH e-Handbook of Statistical Methods, http://www.itl.nist.gov/div898/handbook/, April 2012.
2. Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B, Vehtari, A., and Rubin, D.B.
Bayesian Data Analysis (3rd ed.). Chapman and Hall/CRC, 2013, https://doi.org/10.1201/b16018.
3. Department of Defense. Department of Defense Standard Practice: Documentation of Verification, Validation, and Accreditation (VV&A) for Model and Simulations.
MIL-STD-3022 Change 1. 5 April 2012.
https://quicksearch.dla.mil/qsDocDetails.aspx?ident_number=275961.
4. Zang, Thomas A, Steve R Blattnig, Lawrence L Green, Michael J Hemsch, James M Luckring, Joseph H Morrison, and Ram K Tripathi. NASA Standard for Models and Simulations (M&S): Development Process and Rationale. NASA NTRS. July 2009 https://ntrs.nasa.gov/api/citations/20090028626/downloads/20090028626.pdf.
5. National Aeronautics and Space Administration (NASA). NASA Technical Standard, NASA-STD-7009B, Standard for Models and Simulations, March 2024 https://standards.nasa.gov/standard/NASA/NASA-STD-7009.
6. National Aeronautics and Space Administration (NASA). NASA Technical Standard, NASA-HDBK-7009A, NASA Handbook for Models and Simulations: An Implementation Guide For NASA-STD-7009, May 2019.
https://standards.nasa.gov/standard/NASA/NASA-HDBK-7009.
7. National Aeronautics and Space Administration (NASA). NASA Procedural Requirement, NPR 7150.2D, NASA Software Engineering Requirements, March 2022. https://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPR&c=7150&s=2B.
8. National Aeronautics and Space Administration (NASA). NASA Systems Engineering Handbook, February 2019. https://www.nasa.gov/reference/systemsengineering-handbook/.
9. National Aeronautics and Space Administration (NASA). NASA Procedural Requirement, NPR 7123.1D, “NASA Systems Engineering Processes and Requirements,” July 2023.
https://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPR&c=7123&s=1B
10. Aerospace Research Central. “Guide for Verification and Validation of Computational Fluid Dynamics Simulation.” AIAA G-077-1998, September1998.
https://arc.aiaa.org/doi/epdf/10.2514/4.472855.001.
11. The American Society of Mechanical Engineers (ASME). “Guide for Verification and Validation in Computational Solid Mechanics.” ASME V&V 10, ASME. New York, NY, August 2007. https://cstools.asme.org/.
12. Shackelford, J.F., Han, Y.-H., Kim, S., & Kwon, S.-H. (2015). CRC Materials Science and Engineering Handbook (4th ed.). CRC Press.
https://doi.org/10.1201/b18971
13. NIST Digital Library of Mathematical Functions. National Institute of Standards and Technology (NIST). https://dlmf.nist.gov/.
14. International Organization for Standardization (ISO). ISO 9001:2015, “Quality Management Systems Requirements,” September 2015.
https://www.iso.org/standard/62085.html#lifecycle.
15. Capability Maturity Model Integration (CMMI) Version 3.0.
https://cmmiinstitute.com/ 5
4 DEFINITION OF TERMS.
For this AC, the terms, and definitions from § 401.7 and this list apply.
4.1 Real-World System (RWS).
An actual physical system that has operated, is operating, or will operate which a simulation (e.g., a computer model) emulates.
4.2 Generally Accepted.
Described in standards published by Federal Government or recognized standards organizations, in textbooks that are widely used in educational settings, or in widely cited published documents (journal articles, books , etc.).
5 ACRONYMS.
• 3DOF – Three Degrees of Freedom
• AC – Advisory Circular
• APA –American Psychological Association
• AST – Office of Commercial Space Transportation
• CFR – Code of Federal Regulations
• CMMI –Capability Maturity Model Integration
• CSE – Council of Science Editors
• ECI – Earth Centered Inertial
• FAA – Federal Aviation Administration
• FSA – Flight Safety Analysis
• FHA – Functional Hazard Analysis
• IEEE – Institute of Electrical and Electronics Engineers
• ISBN – International Standard Book Number
• ISO – International Standards Organization
• MIL-STD – Military Standard
• MOC – Means of Compliance
• MPL – Maximum Probable Loss
• NASA – National Aeronautics and Space Administration
• NIST – National Institute of Science and Technology
• OMB – Office of Management and Budget
• RWS – Real World System
• URI – Uniform Resource Identifier
• URL – Uniform Resource Locator
• U.S.C. – United States Code
• U.S. – United States
• V&V – Verification and Validation 7
6 INTRODUCTION.
Section 450.115 provides requirements that apply to all flight safety analysis regulations, from § 450.117 through § 450.139. This AC primarily discusses paragraph
(c) of § 450.115, which contains requirements for how a flight safety method must be described. § 450.115(c) is referenced by § 450.108 and each regulation section from § 450.117 through § 450.135.
6.1 Relationships between paragraphs in § 450.115.
Paragraph (a) of § 450.115 identifies the scenarios that need to be covered in all FSA, so that the analysis comprehensively covers the hazards—and thus the methods need to describe all such scenarios. Paragraph (b) discusses the level of fidelity required – with the fundamental principle that the level of fidelity of a flight safety analysis need only be sufficient to demonstrate compliance with the safety criteria, accounting for uncertainty. Another AC is planned to further discuss level of fidelity and uncertainty.
However, importantly, a thorough description of methods required by paragraph (c) is fundamental to demonstrating compliance with paragraph (b). The level of fidelity of an analysis cannot be assessed without an understanding of the method used to perform the analysis. Therefore paragraph (c) requires an applicant to provide information that allows the FAA to assess the method an applicant used to perform the analysis, which thereby allows the FAA to confirm whether an applicant’s level of fidelity is sufficient to meet paragraph (b).