7 EXPLANATION OF § 450.115(c).
Section 450.115(c) requires that applications include a description of the flight safety analysis methodology, including identification of:
• The scientific principles and statistical methods used;
• All assumptions and their justifications;
• The rationale for the level of fidelity;
• The evidence for validation and verification required by § 450.101(g);
• The extent to which the benchmark conditions are comparable to the foreseeable.
conditions of the intended operations; and
• The extent to which risk mitigations were accounted for in the analyses.
Each of these is discussed in this chapter. However, it is generally helpful to begin a description with an overview of the basic structure of the method, including the conceptual models used and behavior being modeled. Developing a new method is often a major undertaking for applicants and requires commensurate significant review by the FAA. See AC 413.13-1 for other approaches to developing new methods.
7.1 Scientific Principles and Statistical Methods Used.
The first element of § 450.115(c) are the scientific principles and statistical methods used to perform the analysis; this is sometimes referred to as a technical description.
The technical description should cover the entire process from gathering input data through to the specific output products that result. The logic and flow of data through the approach should be clear to the reader. The description should not simply reference principles and methods, but needs to show how they are used, i.e. a description of how they are applied to the situation being analyzed. Additionally, it should describe how different elements of the method connect and how the data flows through the process and show iterative and parallel application of principles. Modularizing the description of methods also helps clarity, where the inputs and outputs of each module are explicit.
It is common to use flow charts at different levels of detail (e.g., how modules relate and steps within the modules) to illustrate the process. Specific and consistent notation (symbology and indices) should be used to ensure clarity. If a method includes an existing method documented elsewhere, the description should provide a clear mapping to the notation used in the external reference, if it is different.
7.1.1 Scientific Principles.
Scientific principles are based on the scientific method: hypotheses tested and demonstrated using repeatable experiments or other empirical data. The validity of a scientific principle is often limited to a set of conditions. For example, Newton’s laws of motion are valid scientific principles, which become inaccurate when an object 9
approaches the speed of light. There are many scientific principles established in the fields of physics, chemistry, and/or biology, which are generally implemented through the use of equations. A description may sometimes combine principles. Additional assumptions may be applied to allow simplification of existing equations. The technical description should show the derivation of any equations used, starting from established principles. Occasionally, a new principle may be developed from geometric arguments;
this should be carefully explained and justified. Considerations for assessing the validity of data and scientific methods are further discussed in paragraphs 8.2.1 and 8.2.2 of this document.
7.1.2 Statistical Methods.
Statistical methods are approaches to describing data and inferring conclusions. Data may be the product of observation/measurement or modeling/simulation. Descriptive statistics provide summaries of gathered data that aid in understanding and reduce the quantity of information to be analyzed. Statistical analysis results in identification of patterns: both the dependence of results on independent variables and the characterization and quantification of uncertainty due to unknown effects. There are variety of approaches to statistical analysis and data science. A structured framework should be followed, such as Exploratory Data Analysis (for an introduction, see Reference 1) or Bayesian analysis (reference 2). Applicants are encouraged to be cautious about developing novel approaches, instead focusing on applying generally accepted (see paragraph 4.2) approaches to the specific topic and referencing such approaches. Considerations of the validity of statistical methods are discussed in paragraph 8.2.3. The applicability of the conclusions and of the predictions to a specific scenario are dependent on the similarity of the scenario to the scenario(s) under which the data was obtained. These are important to characterize as part of the scope of applicability of the method (see paragraph 7.2.1). Statistical analysis should be used to characterize the uncertainties associated with inputs and output from the analysis in order to demonstrate compliance with § 450.115(b).
7.2 Assumptions and Justifications.
The description of methods must include the assumptions used and their justifications, per § 450.115(c)(2). An assumption is an axiom1 or postulate2 that is relevant to supporting the methods. Justifications provide reasons that support the use of a stated assumption. Assumptions related to methods used to perform analysis should be 1 A statement accepted as true as the basis for argument or inference. “Axiom.” Merriam-Webster.com Dictionary, Merriam-Webster, https://www.merriam-webster.com/dictionary/axiom. Accessed 26 Jul. 2024.
2 A hypothesis advanced as an essential presupposition, condition, or premise of a train of reasoning. “Postulate.”
Merriam-Webster.com Dictionary, Merriam-Webster, https://www.merriam-webster.com/dictionary/postulate.
Accessed 26 Jul. 2024.
considered in several categories. A first category is the range of applicability, or the scope, for which methods are intended to cover and not cover. A second category includes the assumptions about which physical phenomena are relevant to the modeling.
Thus, a technical description should identify the set of conditions or bounds that define when the scientific principles used are established to be valid. A third category covers the assumptions about data availability and uncertainty. A fourth category is about the capabilities of human operators. Careful consideration of assumptions and their justifications is fundamental to ensuring a method is appropriate for the scenarios that will be analyzed and has sufficient fidelity, in accordance with § 450.115(b). It is not possible to prove that every assumption has been identified nor is it appropriate to list the basic assumptions of the scientific method (e.g., consistency of physical laws). At a minimum, the technical description should identify those assumptions that could potentially be violated in the application of the method to a particular situation. These include both the environment (e.g. there are no nearby special events) and the vehicle/operation (e.g. effects of slosh on mass properties are negligible due to baffles).
Note: The FAA has found that overlooking important assumptions and insufficient justifications for assumptions are a primary reason that methods are deemed unacceptable.
7.2.1 Scope of the Method.
The description should identify intended uses and permissible use of the method (see Ref 3). This should be listed as a set of constraints or limitations, ideally expressed mathematically, e.g., only when parameter X is less than parameter Y . The scope of a method may be limited for many reasons. One common reason is that the range of data that was used to develop the method is limited, and it is inappropriate to extrapolate beyond the range of the data. Another common reason is that assumptions have been applied to simplify the modeling approach (see paragraph 0).
7.2.2 Physical Phenomena.
The description should identify the assumptions that are made about which physical phenomena are relevant to the models used in the method. These assumptions are often based on the time scale or length scale of the effects being studied. As a very simple example, when dealing with rocket flight, Newtonian physics are normally sufficient; it is unnecessary to consider relativistic or quantum effects. These kinds of assumptions generally make a problem more manageable and easier to model and are necessary to make an analysis practical. Some assumptions may be valid for some methods and not others. For example, for some flight simulation, it is essential to model using six degrees of freedom, but for other (e.g., some malfunctions), it is reasonable to assume fewer degrees of freedom (see the discussion in AC 450.115-1A regarding malfunction trajectory analysis).
7.2.3 Data.
Two key areas of assumptions are important regarding data. First, an analysis often requires data that changes with time. As such, a key assumption is that the data will be available - and available in the timeframe required. A second assumption is that the data accuracy and thoroughness achieve a certain level. The description should identify the assumptions/conditions used regarding the accuracy and timeframe for data input.
These two assumptions should normally lead to flight commit criteria to ensure the operation is within the scope of the analysis validity. A second area of assumption is the character of the uncertainty of the data. A statistical model involves terms related to random effects; assumptions are often made about the probability distribution of these errors. A few of the most common assumptions in statistics of data are normality, linearity, uniformity, and in certain cases, the equality of variance. The description should identify all assumptions in the method regarding the nature and extent of the uncertainties for data input.
7.2.4 Human Operators.
The description should identify any assumptions regarding the human operators involved in the method. A process of analysis involves humans as well as software.
Most analysis approaches require some level of skill from the analyst; it is an assumption that the analyst has such skills. These skills are an essential element in verifying the proper operation of software, both in terms of using proper inputs and validating outputs. Likewise, humans have practical limits; this is relevant for analysis in terms of the speed they can perform tasks and the ability to perceive distinctions. A specific example is the time it takes a mission flight control officer to identify a failure and initiate flight termination action.
7.3 Level of Fidelity.
Section 450.115(c)(3) requires a description of the methodology to include a rationale for the level of fidelity. Assessing fidelity is a comparison of an approach, process, model, or simulation to the real-world. One form of fidelity is accuracy which is the closeness of a parameter or variable (or a set of parameters or variables) within a model, simulation, or experiment to the true value.
Section 450.115(c)(3) is not referencing the justification for the choice of the level of fidelity, but rather the discussion of how the level of fidelity of the method was determined and characterized. Using qualitative terms (e.g., high-fidelity) to describe the fidelity of a method is sometimes misleading and should be avoided.3 Qualitative terms may be useful for discussion of the relative fidelity of different methods.
3 The FAA recognizes that AC 450.115-1 is titled "High-Fidelity Flight Safety Analysis" in contradiction to the guidance here. This is short-hand language for approaches that were accepted as the highest-fidelity approaches generally used (there exist higher-fidelity approaches).
7.3.1 Bias and Uncertainty.
Instead, a quantitative measure of fidelity should be determined: the bias and uncertainty. Bias is systematic tendency for a prediction to be skewed in "one direction" as compared to the actual value. This may be intentional, such as biasing toward more safety. This may be accomplished by choosing an upper bound instead of a mean value (especially when a "reasonable upper bound" might be easier to justify than the full distribution or as a simplifying approximation). Uncertainty is quantified by a probability distribution of the difference between predictions and the actual value, and both aleatory and epistemic uncertainties should be considered (see Reference 5). A description of uncertainty may utilize a functional form (e.g., Gaussian distribution with mean and standard deviation) or be specified by selected confidence levels. In some cases, assessing bias and uncertainty requires engineering judgment. Characterization of the bias and uncertainty of each method is a key input to demonstrating that the flight safety analysis method has sufficient fidelity to establish compliance with the safety criteria accounting for bias and uncertainty in accordance with § 450.115(b).
7.3.2 Rationale.
Rationale can be considered from two different perspectives. One is the rationale for the choice of the level of fidelity, which of course has implications to the operator.
Generally, a higher fidelity model is more costly to implement and operator, but often reduces the mitigations that are required to protect safety thus saving other costs.
However, the FAA does not assess these considerations, so they are not relevant and need not be discussed as part of the method. Instead, as noted above, the method needs to present the rationale for the determination and characterization of fidelity. The description should discuss the fidelity of each module (one of a set of separate parts that, when combined, form a complete whole) of the method and then aggregate into a summary description of the fidelity for each method, including characterization of the relative importance of each module to the overall fidelity. The fidelity may be different for different parallel parts of a method (e.g., different fidelity for different failure response modes in § 450.119); these should be characterized separately.
7.4 Verification and Validation.
Verification and Validation (V&V) refer to activities performed to determine that a product, service, and/or system meets requirements and specifications and that it fulfills its intended purpose. V&V activities can include testing, analysis, demonstration, and inspection. V&V evidence, as required by § 450.115(c)(4), refers to documentation showing that V&V activities have occurred. V&V evidence can include test procedures and reports, analyses, and demonstration and inspection records. A common misunderstanding involves the scope of V&V , where the scope is too limited. There are many ways an analysis can produce the wrong result: a model can be incorrect for the scenario being analyzed, software can incorrectly implement a model, a user can 13
incorrectly operate software, etc. Thus, the model, the software and/or hardware, and the processes to operate the model all need to have appropriate V&V .
7.4.1 Verification.
Verification is the evaluation that a product, service, or system complies with a regulation, requirement, specification, and/or imposed condition. In the specific context of § 450.115(c), verification is the evaluation that the implemented process matches the documented approach described in § 450.115(c)(1).
7.4.2 Validation.
Validation is the assurance that a product, service, or system meets the needs of the operation. For a simulation (model), validation demonstrates that it adequately reflects the Real-World System (RWS) (see paragraph 4.1) that it is intended to emulate.
Successful validation establishes that the method accomplishes the intended purpose in the intended environment. It often involves acceptance and suitability with operation control needs and natural phenomena. Validation ensures that accuracy, bias, assumptions, and uncertainty satisfy associated requirements. One aspect of validation of simulation tools is benchmarking against information that was not used in the development of the models; a discussion of the relevance of such benchmarks to the actual scenario is required in § 450.115(c)(5).
7.4.3 Standards.
Applicants should follow standard practices for performing V&V . The type of process used depends on the type of element undergoing V&V. Examples of standard practices include:
• For modeling and simulation, NASA-HDBK-7009A (Reference 6 in section 3.4) and MIL-STD-3022 (Reference 3 in section 3.4) each provide a comprehensive approach. However, the scope of these documents do not include V&V of procedures for operational use of resulting software.
• For computing systems, including software, AC 450.141-1 provides an overview and references to specific sources.
• For V&V of processes, procedures, and responsibilities, a quality management system is typically used, such as ISO 9001 (Reference 14 in section 3.4).
• For maintaining on-going V&V of software, the CMMI Model (Reference 15 in section 3.4) provides an approach.
For nearly all methods, all four of these aspects should be addressed in the application material, often even for the same element of a method.
7.4.4 Rigor Based on Level of Criticality.
The appropriate rigor of V&V depends on the level of criticality (see AC 450.141-1 for discussion of levels of criticality). A higher level of criticality should have more rigor in the V&V process. A V&V effort performed by an independent organization provides higher rigor than an internal V&V . As example, for software, the V&V activities for high criticality custom software should include testing by a test team independent of the development division or organization consistent with the intent of 450.141(b)(4). For CMMI, the maturity level indicates the level of rigor. For simple software or tools, the V&V can be quite straightforward, such as two staff independently implementing in a spreadsheet
7.4.5 Evidence.
To demonstrate compliance with the V&V requirement of § 450.115(c)(4), the application should:
• Define levels of criticality and the standards used for V&V for each level of criticality,
• Identify the system, process, and software (items) that are used to perform the analysis,
• Document the rationale for the level of criticality of each item,
• Identify the standards used to perform V&V for each item, and
• Provide sample artifacts of V&V for each item.
7.4.6 Off-the-shelf Items.
Many items that are used in a safety analysis are obtained from market sources and it is difficult, or perhaps impossible, for operators to provide the standards for V&V or artifacts. For generally used items that is not specially designed for launch and reentry analysis, the FAA does not expect operators to provide these. However, the way an operator uses the item is subject to V&V requirements. So, for example, there is no need to provide evidence of V&V for Microsoft Excel, but there is for the implementation in a spreadsheet. For tools that are from market sources that are more specially designed for launch and reentry analysis, the operator is encouraged to discuss with the FAA the extent to which the FAA already has such information. For applicant developed custom software leveraging off-the-shelf code libraries, the applicant is expected to document third party product usage policies that provide for applicationspecific V&V, see § 450.141(c)(8).
7.5 Benchmarks.
Section 450.115(c)(5) requires that the description include “the extent to which the benchmark conditions are comparable to the foreseeable conditions of the intended operations.” Benchmarks are data sets of input data and associated results to which an analysis (or part of an analysis) can be compared. Benchmarking is a key element of validation (see above). In some cases, actual events have shown that prior analysis methods were significantly incorrect, typically making incorrect assumptions.
Note: As time progresses, additional benchmarks become available. If new data becomes available that appears to have a material effect on the validity of a method, in accordance with § 450.101(g), the FAA may require consideration of such new information.
7.5.1 Choosing Benchmarks.
Appropriate benchmarks are sometimes difficult to obtain and rarely cover exactly the scenarios that will be modeled by the analysis. Of course, the closer a benchmark is to the actual scenario, the more confidence the benchmarking provides to the analysis— and thus the lower the uncertainty, which is relevant to § 450.115(b). The necessary comprehensiveness of benchmarking corresponds to the level of fidelity of the analysis.
A low fidelity method may be benchmarked at the top-level of FSA analysis, demonstrating that the products (hazard areas, risk metrics) are more conservative than a higher-fidelity method. However, for a method where the safety results are critically dependent on a particular sub-model benchmarking should be accomplished at the submodel level.
7.5.2 Comparison to Benchmarks.
The “extent to which the benchmark conditions are comparable” is an important topic.
The discussion should review each benchmark that was considered as part of the validation and compare the conditions to the conditions within the scope of the method identified in paragraph 7.2.1. This discussion also supports the assessment of the level of fidelity of a method (see paragraph 7.3). A method that has limited or no relevant benchmarks inherently has high uncertainty.
7.6 Risk Mitigations.
Section 450.115(c)(6) requires discussion of “which risk mitigations are accounted for in the analysis.” Mitigations must be described in the functional hazard analysis (FHA), per § 450.107(b). The FSA method should relate applicable FHA mitigations to a specific implementation in the FSA and thus quantified in the analysis. The FSA should account for all flight safety limits and other operational limits identified in the flight commit criteria. For example, a wind-weighted rocket FSA should account for any wind conditions or launcher orientation limits in the normal trajectory variability analysis (§ 450.117). Likewise, risk mitigations might include fault-tolerance that affects the FSA. For example, the FSA for a mission can be completed even if an engine is lost should incorporate this in the normal trajectory analysis (§ 450.117), malfunction trajectory analysis (§ 450.119), and probability of failure (§ 450.131). For each method, the description should identify which of the mitigations in the FHA the method accounts for and any caveats such as significant approximations (covering “the extent to which” from the requirement). An applicant should review the FHA to identify potential impacts of any mitigations on the FSA and discuss the extent to which they are implemented.
8 STANDARD OF SUFFICIENCY.
8.1 Content.
Each flight safety analysis regulation is primarily a set of technical performance requirements: the sections prior to the "Application Requirements" section. The application must show in the description that the method results in compliance with each of these requirements. The application should be explicit about presenting the case for why the method results in compliance. A more thorough and rigorous explanation will reduce iteration during the evaluation process. This could be considered like a "closing argument" in a trial, where the evidence has been presented, and now the logic connecting to the regulation is presented. Although a compliance table may be used, this has often not provided sufficient clarity as to how and why the method demonstrates compliance.
8.1.1 Logic and Mathematics.
The description should include careful exposition of the logic of methodology. This should clearly demonstrate the derivation of the specific method from empirical evidence or generally accepted (see paragraph 4.2) methods. The mathematics used in the analysis needs to be complete, accurate, and well-integrated into the narrative.
8.1.2 Topics.
Usually, many analysis steps are used to provide a comprehensive and valid method for a single regulation. The requirements in a regulation are not necessarily organized by the analysis flow and reflect the interdependence of the nature of the elements of an FSA. For example, § 450.123(b) includes four constraints that apply to the entire method and cannot be satisfied by separate sub-models. The FAA intends to provide checklists for typical elements of the methods within ACs for each regulatory requirement. Generally, the description of each element of each method should include responses to § 450.115(c)(1) and (2), whereas it may be acceptable for the responses to § 450.115(c)(3), (5), and (6) to address a regulation section as a whole. The V&V requirements of § 450.115(c)(4) usually are met on a process by process and software tool by software tool basis. The description of methods should clearly identify these elements.
8.2 Validity.
All analyses used to demonstrate compliance with § 450.101 must be valid in accordance with § 450.101(g). As evidenced by its placement in § 450.101 Safety Criteria, this is a fundamental requirement. Further, in accordance with § 450.37(b), all analyses must demonstrate compliance with § 450.101(g), no equivalent level of safety is allowed. There are four aspects to validity in the regulation, as described in the paragraphs below.
8.2.1 Accurate Data.
Accurate data encompasses both the data upon which scientific models are based and the input data specific to the analysis being performed. Scientific models are based on empirical evidence and experimental results. The data for a specific analysis is based on measurements that are needed as input to the models, including the vehicle properties and the environment. All measurement data has uncertainty; uncertainty must be accounted for when evaluating safety metrics, per § 450.115(b). Accurate data does not mean perfect precision, but instead that which is obtained via valid methods and is applied appropriately for the purpose. For example, inaccurate data would be that obtained using a broken scale or measurements that are inapplicable or outdated.
Inappropriate application of data includes extrapolation of measurements outside the domain in which they were measured, especially to different physical regimes or locations.
8.2.2 Accurate Scientific Principles.
Accurate scientific principles are those that are developed through the process of formulating hypotheses and then conducting experiments to test the hypothesis. To evaluate the validity of a method, the following4 should be considered:
• Whether the technique or theory in question can be, and has been tested;
• Whether it has been passed a peer review;
• Its known or potential error rate;
• The existence and maintenance of standards controlling its operation; and
• Whether it has attracted widespread acceptance within a relevant scientific community. See the definition of “Generally acceptable” in paragraph 4.2.
This standard allows an applicant to use a novel approach, provided it has been developed with the scientific method.
8.2.3 Valid statistical methods.
Valid statistical methods are approaches to analyzing data that are generally accepted (see paragraph 4.2). The starting point is to clearly define the goal of the analysis, and then a statistical analysis can be considered as four steps: collection, analysis, inference, and validation. A complete description of a valid statistical method should include documentation of each of these steps, usually following the structured approach of a standard statistical method (see paragraph 7.1.2). Since the safety criteria are fundamentally statistical measures, statistical approaches are discussed in every FSA AC.
4 This language is from the U.S. Supreme Court decision, Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 113 S. Ct. 2786, 125 L. Ed. 2d 469 (1993) 19
8.2.4 Consistency with Past Results.
The final sentence of § 450.101(g) reads:
The method must produce results consistent with or more conservative than the results available from previous mishaps, tests, or other valid benchmarks, such as higher-fidelity methods.
This connects to the requirements of § 450.115(c)(4) and (5), discussed in paragraphs
7.4 and 7.4 above. As discussed, the availability of benchmarks is limited, but where there are benchmarks, the results of the method must be compared to them. Operations by the applicant are clearly necessary to be considered. The "consistent with or more conservative than" means that any bias in the method should be toward increased public safety. This phase can be most easily understood in the context of the risk analysis products: larger individual risk values, more restrictive flight safety limits, higher predictions of collective risk, etc. For some aspects of flight safety analysis, more conservative is straightforward: the prediction of the consequences of a debris impact should be at least as large as the observed consequences. However, for other areas, especially impact dispersions, conservatism is less direct: dispersion size can be correlated or anti-correlated with the size of the hazard area.
8.3 Editorial Requirements.
All application materials must be written and in English per § 413.7. Materials should use correct grammar, syntax, usage, spelling, and punctuation, as well as being consistent in the handling of capitalization, hyphenation, abbreviations, and numbers.
As highly technical documents with safety implications, method descriptions should undergo internal substantive review and then thorough copyediting and proofreading before submission. A well-written and edited document is more likely to be technically accurate and is much easier for evaluators to follow. Section 450.45(e)(1) provides additional requirements for the submission of material for a safety approval. This section is listed as requirements for the entire application, but the FAA recognizes that these are more useful as applied to each document. The following paragraphs provide additional understanding of how these requirements apply to descriptions of methods.
The FAA does not require any particular style or formatting.
8.3.1 Glossary.
The material should contain a glossary of unique terms and acronyms used in alphabetical order, per 450.45(e)(1)(i). These are typically separate tables or lists in the front matter of a document. Unique terms are those that are not generally known (within the field of the subject of the document) or those that have a precise meaning within the document. The application should use terms as defined in § 401.7; any exceptions should be avoided, and, when unavoidable, specifically noted. Very common acronyms,
e.g., U.S. for United States, may be omitted, but all others must be included.