8.3.2 Reference List.
The description should contain a listing of all referenced material, per 450.45(e)(1)(ii).
No particular format of references is recommended by the FAA, but a standard format should be used, such as Chicago Manual of Style, Council of Science Editors (CSE) style, American Psychological Association (APA) style, or Institute of Electrical and Electronics Engineers (IEEE) style. The applicant should include the Uniform Resource Identifier (URI), International Standard Book Number (ISBN), or Uniform Resource Locator (URL) for references that are publicly available. For references that are not publicly available, the applicant should be prepared to provide access to them upon FAA request and should proactively provide those that contain significant aspects of the method. Wherever another document is referenced, the reference should indicate specific section, subsection, equation, or page number.
8.3.3 Derivations.
The descriptions must use equations and mathematical relationships derived from or referenced to a recognized standard or text, per 450.45(e)(1)(iii), i.e. one that is generally accepted (see paragraph 4.2). There should be no unsubstantiated claims or technical assumptions. There should be no "orphan" equations; all equations must be cited to an external source or derived within the description. Most equations are limited to certain regimes of applicability and/or include simplifying assumptions. A recognized standard or text should characterize these, and the applicant should be clear as to how these affect the derived results. Handbooks, such as References 12 and 13, provide straightforward sources that allow easy referencing.
8.3.4 Notation.
The descriptions should define all algebraic parameters, per 450.45(e)(1)(iii). These should be defined within the text when first used. They may also be provided as a list of notations used in the front matter.
8.3.5 Units.
The descriptions must include the units of all numerical values provided, per
450.45(e)(1)(iv). Units are often only relevant in the context of a specific coordinate system, especially a reference point, so that should be clear as well. Consistent unit systems should be used to the maximum extent possible.
8.3.6 Diagrams.
All schematic diagrams must include a legend or key that identifies all symbols used, per 450.45(e)(1)(v). All maps and charts should also include a legend or key so that the reader can easily interpret the information. Axes should be labeled clearly, including units where relevant. Applicants should avoid complex figures that require significant discussion to understand. Further, the application should make clear in the text the 21
conclusions that are drawn from each diagram; it should not be assumed that the reader has the same interpretation of the information.
8.4 Depth of Detail.
The FSA methodology should be verifiable, inspectable, and repeatable, which means being explicit about details. The description should be sufficient that it produces consistent results using the same set of input data. Later, with specialized tools, training on the use of those tools, and using the same inputs and following the same approach, one should be able to reproduce consistent results and derive the same conclusion as posted by the applicant's flight safety analysis. The flight safety analysis should be inspectable. Any statements made should be clearly supported by evidence. Two different engineers looking at the provided methodology description should not interpret them differently in a meaningful way. Thus, the descriptions should include equations and/or examples that derive the conceptual approach of the methodology. It is not necessary to provide an algorithmic implementation (e.g., pseudo-code) or software code. Use of standard mathematics such as linear algebra or calculus can be assumed.
8.5 Review and Revision Process.
An applicant must submit new and revised flight safety analysis methods to the FAA, and those methods must be accepted, prior to application submission in accordance with § 450.35(a)(1)5. The process from initial submission of methods to acceptance has often involved multiple iterations between the applicant and the FAA. Iterations can be reduced by a thorough internal process by the applicant of methods descriptions prior to submission. This vetting process should rigorously scrutinize the documentation:
challenging assumptions, identifying logical leaps, and ensuring that language is definitive.
The typical process for review of a description of methods involves the FAA providing feedback to the applicant with the applicant revising and then resubmitting. The FAA normally first performs a checklist review (see paragraph 8.1.2), which is only a screening. The screening does not aim to identify technical issues. At the conclusion of the screen, the FAA typically identifies to applicants of material that is expected but is not found, or that the material has passed the screening. The FAA will not proceed to further evaluation of a description of methods until this screening finds the material sufficient. The further evaluation is typically performed by highly knowledgeable reviewers in the specific analysis area. Feedback from this review typically comes as specific comments to the text along with a summary of the feedback.
5 See AC 413.13-1 for additional information on how FSA Methods relate to Means of Compliance.
8.5.1 Maturity Rating.
The FAA normally provides a maturity rating as part of its feedback to help applicants understand the significance of the issues in the method. The maturity levels are shown in Table 1. These maturity levels are unrelated to the complete enough assessment, which occurs after application submission, whereas the methods must be accepted prior to submission. Applicants should be aware that it typically requires a significant effort to move up a single maturity level.
Table 1 - Methodology Maturity Levels Maturity Level Reason(s) 5 – Very Mature May need minor clean-up (minor misstatements, reference problems) 4 – Nearly Mature Documentation is solid, but not yet comprehensive (additional material could expose new issues) Technical approach is nearly acceptable, but there are minor reasons that need to be corrected for the approach to be satisfactory, There are meaningful but not major errors.
3 – In-process Technical approach appears reasonable, but significant gaps remain for the explanation to be complete or correct.
Compliance with § 450.115(c) across the whole analysis has significant gaps.
2 – Immature Significant concerns about the technical approach (the fundamental idea is sound, but the implementation and/or details need a lot of work).
Significantly substandard response to an aspect of § 450.115(c).
1 – Invalid Fundamental problem(s) with the technical approach, i.e. it does not produce the products needed, or it is not viable for the intended application.
Clear lack of understanding of the requirements of an aspect of § 450.115(c) 0 – Not Complete Enough Does not cover all required technical topics or does not discuss an aspect of § 450.115(c) for each technical topic.
8.5.2 Response to Feedback.
Applicants should carefully review all feedback received and submit a revision of the description of the methods for review. Applications should take care to avoid minimizing the comments. The summary of feedback is particularly important as it aims to provide clear identification of the major issues and their severity. If the applicant feels the FAA feedback indicates that a portion of the material was overlooked, is factually incorrect in the feedback, or is inconsistent with the regulation, the applicant should submit the evidence in writing separate from other responses to feedback. Also, if an applicant would like further explanation of the feedback to assist them in responding, a meeting with the FAA may be requested. Applicants should not use the meeting as forum to debate or to seek assurances on a proposed solution.
8.5.3 Recommendations.
To make this process maximally efficient and reduce iterations, the FAA notes the following:
• Applicants should aim to ensure that all feedback has a comprehensive response within the revised description. Only material in the description of methods document(s) are used as the basis for evaluation (e.g. not other responses to comments).
• Attempts to quickly fix a significant issue in a method are rarely found satisfactory.
• Rigorous internal review of all submissions is recommended. An inadequate response to FAA feedback will prolong the review cycle.
• Applicants should specify, for each comment, where in the revised submission the response has been made.
• Applicants are encouraged to submit both a “red-line” and a “clean” version of description revisions to aid reviewers in finding changes, unless the material has undergone such a significant revision that showing individual changes would not be meaningful.
• Configuration management of documents is very important; documents should be assigned unique identification codes with unique revision numbers. A revision history should be maintained within each document. The accuracy of such information should be verified immediately prior to submittal.
8.6 Example.
To illustrate the depth and rigor that provides sufficient material to demonstrate the approach is valid, this section presents two fictional examples. These examples are both descriptions of a three degree of freedom (3DOF) propagation.
8.6.1 Insufficient Description.
An example of a description with insufficient detail is shown in Figure 1. The method is described using a single sentence without any references to the governing mathematics, numerical analysis, or sources of data. This example is an unacceptable submission because there is a wide variety of potential implementations of this that could result in very different results. The FAA cannot determine the fidelity of the physics that are incorporated, whether the numerical approach is valid, or whether the input data is from a suitable source that is processed correctly. Further, without a specification of the software used, there is no basis on which to inspect that the approved software is being used for a particular mission.
Figure 1 - Method Description with Insufficient Detail Debris impact locations are calculated using a 3DOF propagator that incorporates air density and wind using our in-house tool.
8.6.2 Sufficient Description.
An example of a description with sufficient detail is shown in Figure 2. This description is still less than 200 words, but now is precise. This approach required no model development or mathematical derivation by the applicant, existing approaches are simply linked together. The reference to existing approaches also allows the FAA to quickly evaluate the validity of the approach. These requirements are now sufficient for writing an algorithm to perform this calculation, which is then the basis for verification of the implementation. This further provides an unambiguous basis for an inspection by the FAA.
Figure 2 – Method Description with Sufficient Detail A standard approach to three degree-of-freedom (3DOF) computational simulation is used to compute trajectories for uncontrolled, unpowered objects. It is implemented in our SB_BallisticPropagation software module. Input data are the initial position and velocity in earth-centered inertial (ECI) coordinates; the object’s ballistic coefficient as a function of Mach number; and the specification of a 3-D atmospheric model (e.g., a Global Forecast System forecast). Equations of motion are appropriate for a rotating Earth are used to determine the flight path of an object using a 3DOF simulation approach [1]. The equations are integrated with respect to time using a Runge-Kutta method with the Adams-Bashforth predictor-corrector Ref [2] with an initial timestep of 1E-6 seconds (this timestep is much smaller than any meaningful changes in parameters on the scale of rocket flight). Earth parameters through J2 are from WGS84 [3]. Extraction and transformation of air density, speed of sound, and wind data are discussed in 3rd party software documentation [4]. The specific atmospheric data depends on the analysis phase, as discussed in section X.X. The output is the trajectory (time, position, velocity) of the object in ECI coordinates from the initial state to impact with the Earth’s surface at the interval of the integration steps.
1. Weiland, C. (2010). Three and Six Degree of Freedom Trajectory Simulations. In:
Computational Space Flight Mechanics. Springer, Berlin, Heidelberg.
https://doi.org/10.1007/978-3-642-13583-5_8
2. William H. Press ... [and others]. (1992). Numerical recipes in C: the art of scientific computing. Cambridge [Cambridgeshire]; New York: Cambridge University Press, ch. 16.
https://numerical.recipes/
3. Department of Defense World Geodetic System 1984: Its Definition and Relationships with Local Geodetic Systems, Version 1.0.0, 8 July 2014.
https://nsgreg.nga.mil/doc/view?i=4085.
4. XYZ Company, Atmospheric Data Application Programmer’s Interface Reference, version
6.1.
OMB Control Number: 2120-0746 Expiration Date: 11/30/2024 Advisory Circular Feedback Form Paperwork Reduction Act Burden Statement: A federal agency may not conduct or sponsor, and a person is not required to respond to, nor shall a person be subject to a penalty for failure to comply with a collection of information subject to the requirements of the Paperwork Reduction Act unless that collection of information displays a currently valid OMB Control Number. The OMB Control Number for this information collection is 2120-0746. Public reporting for this collection of information is estimated to be approximately 5 minutes per response, inc luding the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. All responses to this collection of information are voluntary per FAA Order 1320.46D Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to: Information Collection Clearance Officer, 800 Independence Ave, Washington, D.C. 20590.
If you find an error in this AC, have recommendations for improving it, or have suggestions for new items/subjects to be added, you may let us know by (1) emailing this form to 9-AST- ASZ210-Directives@faa.gov, or (2) faxing it to (202) 267-5450.
Subject: Click here to enter text. Date: Click here to enter text.
Please check all appropriate line items:
☐ An error (procedural or typographical) has been noted in paragraph Click here to enter text.
on page Click here to enter text..
☐ Recommend paragraph Click here to enter text. on page Click here to enter text. be changed as follows:
Click here to enter text.
☐ In a future change to this AC, please cover the following subject:
(Briefly describe what you want added.)
Click here to enter text.
☐ Other comments:
Click here to enter text.
☐ I would like to discuss the above. Please contact me.
Submitted by: Date: