U.S. Department of Transportation Federal Aviation Administration
Advisory Circular
Subject: Statistical Analysis Considerations for Comparative Test and Analysis Based Compliance Findings for Turbine Engine and Auxiliary Power Unit Replacement, Redesign and Repaired Parts Date: 08/28/2014 Initiated by: ANE-111 AC No: 33-10
1. Purpose.
a. This advisory circular (AC) describes acceptable statistical methods, but not the only methods, to help develop substantiating data for comparative test and analysis compliance findings. The findings support the Federal Aviation Administration (FAA) approval of turbine engine and auxiliary power unit (APU) replacement, redesign and repaired parts produced under:
(1) Parts manufacturer approval (PMA),
(2) Type Certificate (TC),
(3) Supplemental Type Certificate (STC), or
(4) Repair or alteration.
b. The AC describes statistical principles that can be used to help determine adequate sample sizes for a comparative showing of equivalency of parts from different design or manufacturing processes. The guidance in this AC is acceptable for determining sample sizes and/or populations of specimens. The resulting data may be used to support a showing of compliance to the airworthiness requirements of Title 14 of the Code of Federal Regulations (14 CFR) 21.303, 14 CFR part 33, 14 CFR part 43 and Technical Standard Order (TSO) C77.
2. Applicability.
a. The guidance provided in this document is directed to applicants requesting FAA approval for PMA, TC, STC, or repair or alteration of turbine engine and APU parts. Type certificate, PMA, STC, and repair or alteration parts will collectively be referred to as replacement parts for the purpose of this AC. Applicants can use these statistical methods for determining sample sizes when using a comparative test and analysis approach to obtain FAA approval. The guidance in this AC is presented in such a manner that persons experienced in statistics work will be best suited to understand and apply it. However, understanding the basic intent of the guidance does not require a high level of statistics expertise.
b. This material is neither mandatory nor regulatory in nature and does not constitute a regulation. It describes acceptable means, but not the only means, for demonstrating compliance
2 with the applicable regulations. Terms such as “should,” “shall,” “may,” and “must” are only used in relation to the acceptable method of compliance described in this document. The FAA will consider other methods of compliance that an applicant may propose. While the guidelines in this AC are not mandatory, they are derived from extensive FAA and industry experience in determining compliance with the relevant regulations. On the other hand, if we become aware of circumstances that convince us that following this AC would not result in compliance with the applicable regulations, we will not be bound by the terms of this AC, and we may require additional substantiation to make a finding of compliance.
c. This material does not change, create any additional, authorize changes in, or permit deviations from existing regulatory requirements.
3. Related Advisory and Reading Material.
a. FAA related references.
(1) AC 33.83-1, Comparative Method to Show Equivalent Vibratory Stresses and High Cycle Fatigue Capability for Parts Manufacturer Approval of Turbine Engine and Auxiliary Power Unit Parts.
(2) AC 33-8, Guidance for Parts Manufacturer Approval of Turbine Engine and Auxiliary Power Unit Parts under Test and Computation.
b. Related reading material.
(1) David J. Sheskin, Handbook of Parametric and Nonparametric Statistical Procedures, 5th Edition, Chapman & Hall/CRC, 2011.
(2) M. M. Desu, Sample Size Methodology, Academic Press, 1990.
4. Background.
a. The use of statistics is often necessary for comparative test and analysis work when evaluating part dimensions, tolerances or material properties, as these are usually not known to the applicant. The FAA has observed applicants using a variety of statistical principles and methods in their test and computation compliance data to show equivalency. This AC describes acceptable statistical analysis principles that provide acceptable and meaningful results. The AC also describes a practical alternative approach for statistically-derived sample size requirements.
b. The FAA has identified two recurring problem areas that are important considerations when using statistics to demonstrate equivalency between two parts. The first is the use of methods whose purpose is to show differences rather than equivalency. The second is the use of sample sizes that are insufficient to demonstrate equivalence.
c. The approval processes requires an applicant to compare the approved parts to replacement parts. This work could involve comparisons of geometry (dimensional); mechanical properties (creep, tensile, fracture toughness, etc.); and lab analyses for specific physical properties (chemistry, grain size, grain orientation, etc.). When applicants perform comparative test and analysis to show equivalency of the replacement part to the approved part, statistical analysis may be necessary to determine how many parts of each population should be compared.
5. Guidance.
a. Statistical principles. This section discusses the differences between typical industry standard practices versus methods the FAA has determined are statistically correct for determining sample size. Statistical methods exist that can determine whether two groups of parts are statistically equivalent. Equivalency tests begin with the null hypothesis (assumption) that two parts are different and seek evidence to reject that hypothesis. This is in contrast to the more common statistical question “are these two populations different?”
Tests to find a difference begin with the null hypothesis that the two populations are the same, and attempt to show a difference within a given level of confidence (the percent of the time the finding is correct). Many applicants are only familiar with this common test for differences, and assume that if a test for differences fails to detect one, that the two populations must therefore be equivalent. This is an incorrect assumption, as the failure to find a difference does not mean that one does not exist, only that the chosen test cannot detect a difference.
The finding of equivalency requires the opposite approach; a test to determine, with a given level of confidence, that the two populations are not different. The statistical test to make a determination of equivalency is less commonly known. This AC describes the correct methodology to determine statistical equivalency.
(1) Current practice of using a comparative test and analysis method to showing equivalence. The objective of using comparative test and analysis is to show compliance by demonstrating equivalency of two sets of data from two different populations. Many applicants submit data based on statistical methods that are designed to detect statistical differences, not to show equivalency. This results in an incorrect conclusion that the proposed replacement parts and the approved parts are equivalent.
(a) For example, applicants have proposed analyses that use a simple t-test to compare means (averages) of the two populations. In doing so, these applicants assume that this demonstrates that the two populations are equivalent. However, the basic t-test starts with the assumption (null hypothesis) that the two samples are from the same population and it puts the burden on finding a difference. This approach is not statistically valid, and leads to incorrect conclusions about differences.
A test result of “no statistically-significant difference,” means the current evidence is not strong enough to demonstrate that the two parts are different; this is not the same as demonstrating that the two parts are equivalent. This test establishes evidence against the null hypothesis only, not for it. In other words, the absence of evidence that the two part samples are different is not evidence the two part samples are equivalent.
(b) Using small sample sizes in this regard may compromise the method’s ability to identify differences. This situation arises because a small sample size can make it difficult to identify statistical indications that a difference exists. Therefore, if a small sample size methodology is used, the applicant may be unable to detect a statistically-significant difference.
This would result in an incorrect conclusion that the parts are equivalent.
(c) A potential consequence of this current practice is failure to find a difference when one exists, called beta error (β) or “Type II error.” (1 - β) is the “Power” of finding a given difference. Saying a difference exists when a difference does not is called alpha error, α, or “Type I error.” (1 - α) is the “Confidence” level in saying a difference exists. It is beta error that
4 concerns us in a demonstration of equivalence. The common practice of using the simple t-test, high-confidence levels, and small sample sizes leads to a high-beta error – the inability to find real differences between the samples.
(2) Statistically-correct method for showing equivalence. The statistically correct method for showing equivalency is the application of statistical power. When applicants start with the assumption that the two samples must be demonstrated to be equivalent, they must use a statistical test to show a proposed replacement part is equivalent to an approved part. Applicants should determine required sample sizes prior to beginning the test program.
(3) The sample size, N, (for both replacement and approved parts) required for finding an allowed difference d (i.e.., equivalence), given alpha error, α, beta error, β, and standard deviation, s, is:
Formula 1: N = [(zα + zβ)2 * 2 * s2] / d2 Here z is the one-sided normal deviate for the given error level. These values are obtained from a standard normal table, or may be calculated in Excel™ via the NORMSINV function.
NORMSINV returns the inverse of the standard normal cumulative distribution for a probability corresponding to the normal distribution. The input to NORMSINV is (1 - β) or (1 - α). Note that this formula makes the simplifying assumption of known and equivalent standard deviations.
Also note that several calculators are available in the public domain (on the internet), which will compute sample size automatically.
Example calculation:
For a typical α of 0.20 (80% confidence) and β of 0.05 (95% power), the z values are 0.842 and 1.645, respectively. N should be rounded up to the next whole number.
So, for α=0.2, β=0.05, s=1 and d=0.2, we get:
N = [(0.842 + 1.645)2 * 2 * (1)2] / (0.2)2 = 309.3 rounded up to 310 Remember, N is the required number of both replacement and approved part samples; so the total number of parts for both combined is 620.
(4) Define the allowed difference.
In an equivalency demonstration, not only α and β must be defined, but d, the allowed difference between the sample means, must be defined as well. Typically, this value should be less than or equal to 0.2 standard deviation (0.2s) for a finding of equivalence.
Alternatively, d may be established by consent; for example, a proposed definition of equivalency may be “within 3 KSI,” or “within 0.0005 inch tolerance,” or other relevant parameter.
Other specific guidance, such as AC 33.83-1, describes acceptable parameters and values for showing equivalence. These alternative values for d can be established when the effect of the allowed [observed] difference on the usage and requirements of the part and product is understood.
If we compare the allowed difference between sample means to the formula for sample size required considering only alpha error, α, we can readily see the effect of the need to consider beta error:
5 Formula 2: N = [(zα)2 * s2] / d2 Example calculation:
For α=0.05, s=1 and d=0.2, we get:
N = [(1.645)2 * (1)2] / (0.2)2 = 67.65 rounded up to 68
(5) The importance of large sample size.
The z values only hold for large sample sizes, and the conclusion of any statistical test only holds if the samples provide a true representation of the underlying population. This means that parts from more than one lot, melt, or other relevant production factor, establish the variation introduced by these factors. Typically, three different lots is the minimum necessary to capture lot-to-lot variability. Note that the accuracy of a population variability description improves with increasing number of lots represented in the data. Use of two or fewer lots poses a high probability of bias to one side or the other of the distribution, and is not recommended.
The following example compares the sample size needed per sample (replacement sample and approved part sample) for different d values and α=0.2, β=0.05, s=1:
0.2s 310
0.5s 50
0.8s 20
(6) The importance of determining sample size before testing.
(a) Applicants need a method for determining sample sizes for statistical showings of equivalency. This method will tell them early in the project how many specimens will be required to make a valid showing of equivalence. For example, an applicant may have difficulty acquiring a large number of approved parts to test for equivalency, since the demonstration is required before full production can begin and only a limited number of replacement parts may be available. Also, the availability and cost of approved parts, as well as tracking the source of nonserialized parts, may limit the number of parts available for comparative assessment. This situation often results in insufficient test data to support the requirements of the statistical methods for a valid showing of equivalency.
(b) Determining the sample size is a pretest calculation. Often, applicants provide the FAA with posttest calculations from a small number of measured sample parts. Typically, the applicants fail to perform any analysis prior to test measurements to calculate the statisticallyappropriate sample size, and use an incorrect methodology (the simple t-test described above) to show equivalency. The resulting analysis is very unlikely to identify true differences between the approved parts and proposed replacement part populations, unless the differences are very large.
This illustrates why the simple t-test, used as a posttest calculation, is very unlikely to identify differences.
(c) For example, if only nine samples are tested from each population, and 95% confidence is used (α=0.05), you need a difference (d) between the sample means of 1.7s, or more, to be identifiable with 95% power. Therefore, you will not be able to identify true differences between the sample populations unless the differences are very large.
(7) Assumption of normally-distributed data. The statistical concepts discussed in this document apply to data that is at least reasonably-well normally distributed (i.e., the samples are taken from populations that fit the characteristics of a normal (also called Gaussian) distribution).
For data that does not meet these assumptions, transformation into a normal distribution is often possible (square-root normal, log-normal, et cetera). If the data represents other than a normal distribution, and cannot be transformed into a normal distribution, the analysis should use the distributional parameters appropriate to the distribution type.
(8) Measurement error. Measurement error should be minimized through use of a calibrated gauge or other measurement device appropriate for the dimension or parameter being measured. Additionally, the same gauge or device should be used to measure all parts from both replacement and approved parts samples.
(9) Outliers. To ensure the determination of whether an outlier is correct, applicants may not remove sampled parts from the analysis without the approval of the FAA. The assumption that any unusual part must be an outlier is usually a faulty one. A request to label a part as an outlier must include:
(a) A statistical test;
(b) A histogram of all the parts to document the outlier’s status visually; and
(c) Explanations of factors unique to the outlier (handling damage, etc.) that provide a basis for not including the part in the rest of the population.
(10) Summary. In summary, the preceding paragraphs provide an appropriate method for determining the sample size necessary for showing that two sample populations are statistically equivalent.
b. Practical acceptable alternative for sample size requirements. This section discusses a practical alternative to using statistically-derived sample sizes.
(1) Stay within demonstrated experience (i.e., observed measurements).
(a) We recognize that practical constraints may limit applicants from submitting the statistically-required sample sizes of parts to demonstrate that the replacement part is equivalent to the approved part. Under this alternative, the requirement for demonstrating equivalency becomes a restriction that limits the subsequent replacement part design to be within the observed limits of the sample population of approved parts. At the same time, the replacement part sample must meet other requirements as well. These requirements are explained in paragraph (b) below.
(b) Minimum sample sizes depend on the particular parameter being evaluated. For straightforward measurements such as dimensions, the FAA considers a minimum sample of three to five approved parts from each of three or more different lots as acceptable(where a lot is any identified variation in production process or material chemical composition that could result in a measure of random variability in finished part dimensional or material characteristics). In other words, 9-15 parts equally distributed over 3 or more lots is an acceptable minimum sample.
However, we do recommend larger sample sizes.
Multiple lots (not just one high or low lot) are necessary to help ensure that a true range of population variation is observed. The more lots represented, the better the evaluation of population variability. These sample sizes apply to both approved parts and replacement parts.
0812812014 AC 33-10 Also, emphasis must be placed on covering the range of manufacturing variability in the replacement parts rather than performing a single production run. If lot processing information is not available for the approved parts, the applicant must obtain parts from different sources and over a period of time. Other more complex parameters, such as high-cycle fatigue and vibration (modal) characteristics, typically require a more thorough understanding of the population mean and variation to establish equivalence and replacement part specification limits. This necessitates larger sample sizes (see AC 33.83-1).
(c) Applicants should analyze the approved part and replacement part samples for the means, extents (highest and lowest actual measurement) and standard deviation. The approved part data will be used to establish the subsequent replacement part production specification limits. In addition, the replacement part sample data must fit within the observed range of the approved part sample. For example, when measuring a particular dimension, the lowest replacement part dimensional value (as actually measured) must be equal to or greater than the lowest approved part dimensional value (as actually measured); and the highest replacement part dimensional value (as actually measured) must be equal to or less than the highest approved part dimensional value (as actually measured).
Additionally, the replacement part sample standard deviation must be less than or equal to the approved part sample standard deviation. Finally, the difference between the replacement and approved part sample means must meet an 80% confidence test; meaning that at-test must fail to find a significant difference at the 80% confidence level vv:ith the acceptable minimmn sample size. The t-value for 80% confidence is obtained from at-table for the appropriate degrees of freedom.
1 The equation for the comparison of two samples standard deviations unknown but assumed to be equal, is:
T,.W2.d+~2~ ABS(x 1- x 2) I SQRT{[(v1sl 2 + v,s, 2) I (vl+v,)][li(NI) + li(N,)] } Here X 1 and X 2 are the sample means, s12 and sz2 are the sample standard deviations, N1 and N2 are the sample sizes, and v 1 and vz are the sample degrees of freedom v1= CN1 -1) and v2 = (N2 - 1) ]. "ABS" means absolute value,. and "SQRT" means square root.
Note that this formula can be rearranged to solve for the maximum allowable difference between the two samples means to meet the 80% confidence requirement:
Max(x 1 - x ,) ~ t1. ol2.~1+~2 * SQRT { [(v1s12 + v,s, 2) I (v1+v,)][ li(N 1) + l/(N,)] } 7
0812812014 AC 33-10 Example calculation: Approved Parts Replacement Parts Sample mean 2.4 2.3 Sample standard deviation 0.7 0.5 Sample size (N) 9 12 Degrees of freedom, (v = N-1) 8 II
t.so%.19 ~ (2.4- 2.3) I SQRT{[(8*0.7 2 + II *0.52) I (8+ 11)][1/(9) + 11(12)]) ~ 0.383 The two-sided 80% t value for 19 degrees of freedom (v1 + v2) equals 1.328, as obtained from a standard table of the Student's t distribution. Since the calculated value of0.383 is less than the tso%, 19 value, the samples meet the requirement of no significant difference with 80% confidence.
2. If the necessary outcomes noted in paragraph 5.b (1) (c )1 above cannot be met using the initial sample size, then the applicant may increase the number of samples in a further attempt to demonstrate equivalency. These additional samples must be random (no preselection of parts) and can only include parts representative of the manufactured population. These additional parts may be added to the overall sample in an effort to find an approved part dimensional value (as actually measured) that is equal to or lower than the lowest replacement part dimensional value (as actually measured). Parts already included in the analysis may not be eliminated unless they clearly fail an outlier test (see paragraph 5.a (6)), or an entirely new sample under a changed production process is provided as a new demonstration of equivalency.
Note that the statistics and formulae must be recalculated to include the additional parts.
J. If the measured data (replacement or approved parts) used for the comparison represents other than a normal distribution, and cannot be transformed into a normal distribution, then the comparable distributional parameters should be calculated rather than mean and standard deviation.
(d) Testing or measuring a larger sample of approved parts would likely produce a wider observed range of design parameters. The advantage of a larger sample size is that the replacement part may have a wider range of parameter values to stay within, making it easier to show equivalency. If an expanded sample size is used the applicant must show that the replacement part stays within this wider range.
(e) Additional rationale for the requirement to stay within observed measurements
(i.e., demonstrated experience) is that the true approved part manufacturing tolerance may be "inspected in" to the approved part population. In this situation, the approved part manufacturing process produces a normal distribution, but the extents ofthe distribution lie outside the acceptable limits for production acceptance, and parts are either scrapped or reworked. The resulting distribution of measurements appears normal closely about the mean, but the tails of the distribution are truncated. Unless adequate parts are sampled, it is unlikely that the tolerance truncation will be recognized and any attempt to statistically identify tolerance limits will likely result in a replacement part that does not meet the approved part tolerance. The following Figure 1 illustrates this situation:
Figure 1 Parameter Measurements by Number of Parts
(2) Establish the replacement part production specifications.
(a) The replacement part production specification must not exceed either the high or low actual measured value for the parameter within the approved part sample. Be aware that for small sample sizes (less than 25), the sample standard deviation may be artificially large, and that the type design holder may be “inspecting in” a tighter limit than the predicted population extents. For example, if the applicant has inspected 12 approved parts, and those parts range from 55Rc to 58Rc hardness, the replacement part specification for hardness must lie within (or at) those values. Applicants should use similar analysis for other parameters. If the observed range is considered too restrictive, applicants may sample additional approved parts in an attempt to show a larger actual range.
(b) For profile and similar measurements (for example, chord lengths, leading edge radial contour, or twist and related characteristics), deviations from the requirement listed in paragraph 5.b.(2)(a) (directly above) may be accepted to ensure the intended contour of the part remains consistent with the approved part. In those cases where deviation must be used to ensure consistency, the applicant must demonstrate that this smoothing of the geometry does not fall outside the bounds of the measured approved parts. Also, the observed range and (and other parameters as appropriate) may be “smoothed” or averaged across all the linked dimensions.
However, care must be taken for smoothing measured data. Features showing tight local tolerances may indicate the presence of design controls that are necessary for proper function throughout the approved product operating envelope.
0 2 4 6 8 10 12 14 16
0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 1 1.01 1.02 1.03 1.04 1.05 Parameter Measurement Number of Parts
(3) Material properties.
(a) This section provides guidance when selecting sample size to characterize material properties. In traditional material characterization testing, specimens made from the material of interest are fabricated to standard industry specifications. The number and geometry of these specimens can be found in various industry standards documents. When trying to demonstrate similarity between TCH and aftermarket parts, the comparison baseline is the finished part and not the pre-manufactured component material.
Component size limitations may prevent the removal of test material suitable to measure and compare the required material properties. In these instances, the applicant may substitute, by FAA approved test plan, specially processed test material, that is equivalent to the component material. Equivalency is demonstrated by duplicating each material’s form, microstructure hardness, and chemistry.
Other complications to the comparison process are geometric complexities, part-to-part tolerances, effects of coatings, and requirements to test under application specific environmental conditions. As a result, additional specimens may be required to statistically characterize the nominal and lower bound part material characteristics.
1 For basic material properties that are more dependent on alloy constituency than on part manufacture process, such as Young’s Modulus, Poisson’s Ratio, thermal expansion and conductivity coefficients, and density; we recommend a minimum of 10 approved parts or specimens be tested to establish minimum material property characteristics. The 10 approved parts must be from three separate lots with at least three parts per lot. Applicants may also use this comparison standard when establishing grain size, microstructure, and hardness. The same number of replacement parts must then be tested within the same test parameter ranges and the results compared.
2 For properties affected by how the material is processed during part manufacture such as high-cycle fatigue, low-cycle fatigue, creep, tensile strength, crack growth, etc., we recommend a minimum of 30 approved parts or specimens be tested to establish a minimum material property curve. For fatigue testing, we further recommend that at least 25 of the tested parts not be run-outs. A run-out is a part that completes a fatigue test of the planned test duration (cycles) without cracking. The same number of replacement parts must then be tested within the same test parameter ranges and the results compared.
3 For parts exhibiting complex geometry in the gage area (for example fillet radius, cooling holes, or rapid changes in section, near the peak stress location for fatigue testing) or complex manufacturing variables such as grain structure or coatings, additional specimens may be required to capture part-to-part variables. Applicants must also evaluate the criticality of the part (level of accuracy required) to ensure the sample size is sufficiently large to capture the anticipated property variations. So, sample sizes greater than those recommended for basic properties and properties affected by manufacture may be required under certain circumstances.
11 4 Generally, the results comparison should show that the replacement part material properties fall between the minimum and maximum values observed for the approved part. However, the comparison criteria will depend upon the material property being tested. For example:
(aa) Replacement parts or specimens must be tested within the same stress and temperature ranges as the approved part tests, and the results must fall at or above the minimum value observe for the approved part (not just be above the predicted minimum curve).
The minimum specification is then the minimum observed approved part value at each stress and temperature level.
For comparison of the average material properties to that of the approved parts, the proposed replacement part average material property curve must also lie above or at, the approved part average material property curve. Further, the projected replacement part minimum curve must lie above or at the approved part minimum curve.
Excess variability increases the number of parts or specimens required. The expected amount of variation in the material property can be obtained from generic material property curves (for example, nickel and stainless steels) in the industry specifications.
The proposed replacement part curve being above the approved part curve is still not, by itself, a demonstration that the proposed replacement parts are comparable. The replacement part must still lie at or above the minimum observed approved part at each stress level. Run-outs are not to be included in the average or standard deviation calculations.
(bb) If the observed minimum or maximum (as applicable) material property curves are considered too restrictive for actual production needs, the applicant may perform additional approved part tests (to increase sample size) to show different actual observed results.
(b) Other circumstances may occur where specific properties (such as high-cycle fatigue for blades and vanes) must be demonstrated for the "as produced" part. For example, this may occur when complex engine operating conditions with a large number of variables are involved. These instances may necessitate using a large number of samples to ensure the part's “as produced” capability is accurately characterized. See AC 33.83-1 for additional information for blades and vanes in this regard.
(c) Production inspection to maintain material properties.
1 Applicants need a quality control program to ensure the production standard replacement parts meet or exceed the material property minimum or range, including holding the required average material property level. The applicant must develop and document a method to show that material properties within each production lot meet the minimum specification. This program is part of the production approval.
2 For material properties that cannot be adequately inspected without destructive testing, the applicant should predict a minimum material property. Applicants should use data from the replacement part tests that were performed to make their predictions. If this predicted minimum is unacceptable for mission and compliance purposes, and regardless of comparison with approved parts, the applicant has not demonstrated the capability to produce the replacement part. The overall intent of this prediction is to ensure acceptable process and part capabilities rather than for comparison with approved parts.
Colleen M. D' Alessandro Assistant Manager, Engine and Propeller Directorate Aircraft Certification Service 12