Section Narrative Objective Criteria to Determine if the Application is Complete Timeframe
• Is there a description for how values of critical parameters required in paragraphs (c)(3) and
(e) of § 450.108 were determined?
• Are there quantitative criteria for each critical parameter for a representative flight?
§ 450.108(g)(3) Graphical depictions
• Is there a graphical depiction of each flight safety limit for a representative flight?
• Does each graphical depiction include the nominal trajectory?
• Does each graphical depiction include the extents of normal flight?
• Does each graphical depiction include limits of a useful mission trajectories?
• Does each map include uncontrolled area boundaries together with the launch or landing point?
§ 450.108(g)(4) Available data
• Is there a clear and complete narrative description of vehicle data available to evaluate flight abort rules?
Section Narrative Objective Criteria to Determine if the Application is Complete Timeframe
10.3 § 450.109 Flight hazard analysis may be utilized as a This section only applies if flight hazard Phase 1;
Flight Hazard hazard control strategy but may also be analysis is identified as a hazard control Phase 2-3 for Analysis mandated by § 450.107(c) A flight hazard analysis identifies key system design and operation data, documents the overall system safety risk to the public, and determines the necessary hazard controls (mitigations) to ensure the residual risk meets strategy in § 450.107 OR if the exception per § 450.143(a)(2) is being applied. If it is not, nothing needs to be submitted.
• Is there a description of the flight hazard analysis methodology for the lifecycle?
completion of § 450.109(b)(5) validation effort acceptable criteria.
The flight hazard analysis should be performed early in system development and operation conceptualization to define the system safety risk to the public in order to positively influence design and operation decisions.
• Does the flight hazard analysis identify hazards?
• Is there a failure mode identified for each hazard?
• Does the analysis methodology consider hazards from:
o Vehicle operation, including staging and release?
AC 450.103-1, System Safety Program, provides guidance on severity categories and likelihood levels in Table 3 and Table 4 of Appendix A, including acceptable criteria for “extremely remote.”
o System, subsystem, and component failures or faults?
o Software operations?
o Environmental conditions?
o Human factors?
o Design inadequacies?
o Procedure deficiencies?
o Functional and physical interfaces between subsystems, including any vehicle payload?
o Reuse of components or systems?
Section Narrative Objective Criteria to Determine if the Application is Complete Timeframe o Interactions of the above?
• For each hazard, is there identification of:
o Likelihood?
o Severity?
• Is there a definition of what constitutes “extremely remote” likelihood?
• For every hazard with a severity of death or serious injury to the public, is the mitigated likelihood “extremely remote”?
• Are risk elimination and mitigation measures identified and described?
• Are verification methods for risk elimination and mitigation measures identified and described?
• Is there documentation of verification and validation achieving the risk levels above?
• Is traceability demonstrated in the hazard analysis data:
o For each hazard is there at least one failure?
o For each hazard, is there as least one mitigation?
o For each failure, is there at least one cause?
o For each cause, is there at least one mitigation?
Section Narrative Objective Criteria to Determine if the Application is Complete Timeframe o For each mitigation, is there at least one verification?
o For each mitigation, is there a risk assessment?
• Is there documentation of the techniques for identifying new hazards?
• Does the documentation include criteria for identification of new hazards?
10.4 § 450.110 Physical containment.
Physical containment is a hazard control strategy described in § 450.110. The focus in pre-app is on the methods used to demonstrate that the launch vehicle does not have sufficient energy for any hazards associated with its flight to reach outside the flight hazard area. FAA reviews the proposed means of compliance in pre-app to confirm that the methodologies will result in compliance with the regulation.
This section only applies if physical containment is identified as a hazard control strategy in § 450.107. If it is not, nothing needs to be submitted.
§ 450.110(c)(1) Extent of hazard
• Is the method for determining maximum range of the vehicle described?
• Is the size of the hazard area around an impact discussed?
§ 450.110(c)(2) Clearance
• Are the methods for clearing flight hazard areas of the public documented?
• Is the potential for critical assets within the flight hazard areas discussed?
Phase 2 28
Section Narrative Objective Criteria to Determine if the Application is Complete Timeframe
10.5§ 450.111 Wind weighting.
Wind weighting is a hazard control strategy described in § 450.111. The focus in pre-app is to identify a means of compliance for the wind weighting analysis methods, including the method and schedule of determining wind speed and wind direction for each altitude layer. FAA reviews the proposed means of compliance in pre-app to confirm that the methodologies will result in compliance with the regulation.
This section only applies if wind weighting is identified as a hazard control strategy in 450.107. If it is not, nothing needs to be submitted.
§ 450.110(e)(1) Methods
• Are the methods of performing wind weighting to determine launcher (e.g., launch rail) parameters documented?
• Is the method for determining wind as a function of altitude documented?
• Is there a schedule for making wind measurements?
§ 450.110(e)(2) System
• Is there a description of the wind weighting safety system?
• Does the description include equipment?
• Is there a wind weighting analysis for a representative flight?
• Does it include launch areas winds?
• Are there samples of wind weighting output?
Phase 2 29
11 FLIGHT SAFETY ANALYSIS.
11.1.1 Flight Safety Analysis (FSA) consists of sections §§ 450.113 through 450.139. Sections
450.113 and 450.115 provide general requirements that apply to each of the sub-analyses, which are discussed in §§ 450.117 through 450.139. Pursuant to § 450.35(a)(1), a means of compliance must be accepted by the Administrator for § 450.115(b)(1) prior to application acceptance.
11.1.2 Note that § 450.115(b) is a requirement on the level of fidelity of the applicant’s FSA method. The FSA method is the accumulation of the methods for each sub-analysis (§§ 450.117 through 137), as well as flight abort (§ 450.108). Thus, the FSA MOC needs to cover every sub-analysis. However, the FAA does not require that the same MOC type (see the types of acceptable MOC discussed in the Means of Compliance section of this document) be used for each sub-analysis. Instead, the applicant should specify which of the MOCs is being used for each of the requirements. For example, when operating from a Federal Range, it is common to use Federal Entity Services for most of the sub-analyses, and an applicant-specific method for the sub-analyses that the Federal Range does not perform.
11.1.3 The FSA method must apply for the entire scope of the analysis in accordance with § 450.115(a).1 Applicants should consider performing preliminary functional hazard analyses prior to submitting their FSA method for review since the system safety process identifies “reasonably foreseeable events and failures” that ultimately determine the proper scope of the FSA method under § 450.115(a). For this reason, applicants are encouraged to include any preliminary functional hazard analyses with their FSA methodology documentation to facilitate the FAA’s review of their FSA method.
11.2 FSA Application Requirements.
11.2.1 Each of the FSA sub-analysis sections contains application requirements. These application requirements may be considered as three categories: methodology, narratives, and data for a representative analysis, as discussed below.
11.2.2 The first type of application requirement includes those that begin with the words “a description of the methods” which refer to § 450.115(c). An applicant-specific MOC for FSA (type 5 in in the Means of Compliance section of this document) for a regulation section is an FAA-approved methodology. Thus, if an applicant is submitting a new methodology which will be used as a MOC for any of the flight safety analysis requirements, they will be focal points for pre-application conversations. See below for guidelines for a complete methodology description.
1 § 450.115(a) provides essential context for understanding § 450.115(b).
11.2.3 The second type of application requirements are narrative descriptions. There are only a few requirements of this type (e.g., § 450.119(c)(3)). Advisory Circulars provide guidance of satisfactory versions of these. It is important that the narrative cover all phases of flight.
11.2.4 The third type of application requirement is representative data, obtained by performing an FSA for a representative operation. A representative operation is one that is covered by the scope of the vehicle operator license (§ 450.3). It should be appropriate to all vehicle configurations (see § 450.45). For example, if solid rocket boosters are part of one configuration, the representative analysis should include them.
The analysis must cover all phases of flight as required by § 450.113. The representative analysis should result in satisfying the safety criteria in § 450.101, in order to demonstrate that the methods have sufficient fidelity to be appropriate for the license, per § 450.115(b)(1). This analysis process is then performed throughout the lifecycle of all proposed operations, for all phases of flight, to demonstrate compliance with the safety criteria for each operation are met at the time of the launch commit decision. The application materials should be of sufficient detail for evaluating the safety criteria for all proposed operations within the scope of license.
11.3 FSA Methodology Description.
11.3.1 There are three general aspects to determining completeness for a methodology:
content, rigor, and depth. The content must 1) address each element of the regulations for the subject sub-analysis (e.g., constraints, objectives, and application requirements) with a scope and level of fidelity in accordance with § 450.115(a) and (b), and 2) cover each element of § 450.115(c). The level of fidelity means the exactness to which the approach reflects the actual physical world‒a higher level of fidelity has less uncertainty and/or bias. The content should also describe the intended usage and limitations for each proposed method. The rigor of the description should ensure that the logic is clear, and discussions (especially of processes) are definitive; equations and flowcharts are often helpful to clarify. A knowledgeable reader should be able to read a methodology and produce nearly the same answer for the same vehicle, provided they had access to the raw data. current ACs and other previously approved methodologies provide specific guidelines for satisfactory content, rigor, and depth for a valid methodology description.
Note: Some sections of ACs implicitly rely on long-established Federal Range precedent as rationale, so therefore have less detailed derivation and/or justification).
11.3.2 Pre-application conversations on the proposed method for FSA, particularly for novel operations and new methodologies, are an efficient means to reach a mutual understanding of the appropriate content, rigor, and depth. Pre-application conversations on the “ground rules and assumptions” used in a proposed FSA methodology should address each of the requirements of § 450.115(c) over the lifecycle of the proposed license activities. For example, the input data and assumptions used for some FSA methods, such as the trajectory analysis for normal flight, naturally evolve given higher uncertainties prior to the first flight. If an applicant proposes multiple flights, the application, per § 450.103(e)(2), must include a summary of the processes 31
and products identified in the system safety program that enable FSA method updates to comply with the post-flight data requirements in § 450.103(d). Since the nature of the methodology updates often depend on the specifics of the data or anomalies identified during the post-flight data review, the applicant should consider defining a process that involves the FAA in evaluating proposed FSA methodology updates2 based on postflight data. For example, a complete submission may include a post-flight data review process with an objective to identify any element of the flight safety analysis content, rigor, or depth that should be updated given the post-flight evidence. Such a post-flight data review process may involve the FAA in discussions intended to gain mutual understanding of any updates to the ground rules and assumptions appropriate for future flights based on the specifics of the data or anomalies identified during the post-flight data review.
11.3.3 The following provides additional guidance regarding the requirements of § 450.115(c):
1. The scientific principles and statistical methods used;
This is the data, including reference data, and mathematics that are used to describe the physical phenomena and the associated uncertainties. Modeling should be based on established physics, standard statistical methods, and/or empirical data.
References to appropriate literature should be provided. Scientific principles refer to knowledge based on the scientific method, such as that established in the fields of physics, chemistry, and engineering. An analysis based on non-scientific principles, such as astrology, would not be consistent with this standard.
A statistically valid analysis is the result of a sound application of mathematics and accounts for the uncertainty in any statistical inference due to sample size limits, the degree of applicability of data to a particular system, and the degree of homogeneity of the data.
2. All assumptions and their justifications;
There are two types of assumptions in a methodology: 1) the scope of scenarios for which the methodology is intended to apply (and not apply), and 2) the physical phenomena that are relevant to the modeling. These assumptions should be stated clearly at relevant points within the narrative. Key assumptions should also be summarized separately, such as the assumptions on the scope of applicability of the method.
3. The rationale for the level of fidelity;
The applicant should discuss the fidelity of the approach and explain how this was determined. Fidelity means the degree of exactness of the approach relative to the actual physical world, so this usually involves a discussion of the biases and uncertainties associated with the method (ideally quantitative). A key aspect of part 450 is that the fidelity of the analysis only needs to be sufficient to demonstrate compliance with the safety criteria, accounting for uncertainty; see § 450.115(b).
2 An application for a license modification would be required if this is significant update (material change).
4. The evidence for validation and verification (V&V) required by § 450.101(g);
Department of Defense Modeling and Simulation (M&S) Verification, Validation, and Accreditation (VV&A) Instruction Number 5000.61 Change 1, dated October 15, 2018, provides a good overview of V&V for modeling and simulation.
The rigor of V&V depends on the level of criticality of the model (so independent V&V is not usually necessary); see also § 450.143 and associated guidance.
An appraisal of the software development process (e.g., comparison to the CMMI best practices), is an effective way of demonstrating ongoing V&V. The V&V discussion should also consider processes, particularly those that must happen in a short time frame, such as during the countdown to an operation.
5. The extent to which the benchmark conditions are comparable to the foreseeable conditions of the intended operations.
Benchmark conditions are situations where the modeling approach has been compared to empirical data and/or other modeling approaches. These should be compared to the intended scope of the methodology, discussing the regimes where the model is closer to and further from the benchmarks.
6. The extent to which risk mitigations were accounted for in the analyses.
This describes how mitigations (e.g., flight safety system, hazard areas, trajectory limits) are incorporated in the analysis process. Mitigations are those described in the functional hazard analysis.
11.3.4 It is helpful if a complete methodology description includes an overview discussion guiding the reader to where each required element may be found. It is usually convenient for sub-analysis descriptions to be separate documents, and often, even within a regulation section, there may be multiple methodology documents. For example, § 450.121 includes breakup and debris analysis; the methodologies for determining breakup criteria (structural limits), debris catalog, and debris propagation/dispersion are often separate documents. Many descriptions may reference a common document, such as a software development plan to discuss how software implementations undergo verification per § 450.115(c)(4).
11.3.5 In accordance with § 450.35(a)(1), the FAA will only accept an application that uses previously accepted flight safety analysis methodologies for all sub-analyses for all phases of flight. A compliance matrix is a helpful approach to demonstrate compliance with this requirement.
11.4 FSA MOC for a Single Planned Flight Profile.
11.4.1 Scoping the initial FSA MOC using a single planned flight profile may reduce the initial effort for an applicant for their first operations because the level of rigor of the methodology description may be significantly lower. This occurs because it significantly limits the scope of the FAA’s evaluation of compliance of the methodology description with § 450.115(b) and § 450.101(g). This approach may be useful to applicants who only need a single flight profile for first flights of a new vehicle in development, but it is not intended for long term operational needs. Each new mission profile requires a license modification and subsequent FAA review. This approach is only a viable path for situations where collective risk is significantly below the safety criteria and mitigations can include larger operational restrictions (e.g., larger hazard areas). It is generally not suitable to optimize operational constraints for a frequent operational cadence.
11.4.2 To utilize this approach, an applicant may choose to scope their methodology and data to a single planned flight profile instead of requesting that a methodology be approved for variety of flight profiles and operations. The applicant should discuss the suitability of this approach with the FAA during pre-application consultations. The application should state that:
• The data in application submissions are for the actual planned flight profile rather than just “representative” data.
• The risk analysis is applicable to all foreseeable conditions within the launch commit criteria, in accordance with § 450.135(a)(1) and § 450.137(a)(1).
Section Objective Criteria to Determine if the Application is Complete Timeframe
11.5 § 450.115 Flight safety analysis methods.
This requirement is referenced by each flight safety analysis section as part of the methodology descriptions, which are a Means of Compliance.
Prior to application submission
11.6 § 450.117 Trajectory analysis for normal flight.
§ 450.117(d)(1) Vehicle's flight behavior methods
• Does the normal trajectory analysis methodology reference an accepted means of compliance?
§ 450.117(d)(2) Quantitative input data
• Is there normal trajectory analysis input data for all phases of flight?
• Does the normal trajectory analysis input data include uncertainties?
§ 450.117(d)(3) Atmospheric conditions
• Does the normal trajectory analysis atmospheric data account for all phases of flight?
• Is there a definition of the worst atmospheric conditions under which flight will be attempted, such as a table of criteria?
• Is there a description of how atmospheric conditions will be evaluated prior to initiating the operation?
• Is there a description of how uncertainty in the atmospheric conditions will be evaluated prior to initiating the operation?
§ 450.117(d)(4) Nominal flight trajectory analysis outputs for a representative flight
• Does the normal trajectory data account for all phases of flight?
• Does the normal trajectory data include position, velocity, and orientation for each second of flight?
• Is there a nominal trajectory?
Phase 1 35
Section Objective Criteria to Determine if the Application is Complete Timeframe
• Is there a set of trajectories (normally at least four) that characterize variability in the intended trajectory based on conditions known prior to initiation?
• Is there a set of trajectories (normally hundreds) that characterize how the actual trajectory could differ from the intended trajectory due to random uncertainties?
11.7§ 450.119 § 450.119(c)(1) Vehicle's flight behavior methods Phase 1 Trajectory • Does the malfunction trajectory analysis methodology reference an analysis for accepted means of compliance?
malfunction flight. § 450.119(c)(2) Limits of useful mission methods
• D o the limits of a useful mission determination methodology reference an accepted means of compliance?
§ 450.119(c)(3) Malfunction flight behavior
• Does the malfunction trajectory analysis input data account for all phases of flight?
• Is there a list of each cause of malfunction flight considered?
• Is there a list of each type of malfunction flight for which malfunction flight behavior was characterized?
• Is there a quantitative description of the parameters with a significant influence on the vehicle's malfunction behavior?
• Does the quantitative description of the malfunction flight parameters include uncertainties?
• Does the quantitative description cover each type of malfunction flight characterized?
Note: A quantitative description of a parameter specifies the numerical values. A quantitative description of an uncertain parameter should be in the form of a distribution with its numerical parameters specified (e.g., mean and standard deviation for a normal distribution).