Content uploaded by Robert Shishko
Author content
All content in this area was uploaded by Robert Shishko on Feb 10, 2016
Content may be subject to copyright.
American Institute of Aeronautics and Astronautics
1
Developing Analogy Cost Estimates for Space Missions
Robert Shishko*
Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA 91109
The analogy approach in cost estimation combines actual cost data from similar existing
systems, activities, or items with adjustments for a new project’s technical, physical or
programmatic differences to derive a cost estimate for the new system. This method is
normally used early in a project cycle when there is insufficient design/cost data to use as a
basis for (or insufficient time to perform) a detailed engineering cost estimate. The major
limitation of this method is that it relies on the judgment and experience of the
analyst/estimator. The analyst must ensure that the best analogy or analogies have been
selected, and that appropriate adjustments have been made. While analogy costing is
common, there is a dearth of advice in the literature on the “adjustment methodology”,
especially for hardware projects. This paper discusses some potential approaches that can
improve rigor and repeatability in the analogy costing process.
Nomenclature
di,j = distance metric between project i and project j
xik = technical attribute k in project i
Cnew = estimated cost for a new project
Canal = actual cost for a previous project selected as an analogy
N* = number of analogy projects selected
wt = average wage rate, typically dollars per workyear at time t
K = number of attributes in adjustment mechanism
αk = weighting factor the attribute k
λi = weighting factor for analogy project i
I. Introduction
nalogy cost estimation is recognized as a useful approach in preparing an early cost estimate for a new system
or project when there is insufficient historical data to develop a statistically valid cost estimating relationship
(CER), or insufficient information, time, or resources to perform a engineering (“grass-roots”) estimate. An analogy
cost estimate is also useful as a “sanity” cross-check against results produced by these other two methods. The basic
idea behind the analogy approach is that when a new system has functional and performance characteristics similar
to an existing one whose cost is known, the known cost can be adjusted to reflect programmatic and technical
differences to develop a cost estimate for the new system. Analogy estimates can be made for whole projects or
elements of a project. Even when some spacecraft subsystems are entirely new designs, often others are developed
as improved versions of previously successful designs (i.e., heritage designs). If their costs are known, heritage
designs can serve as analogy projects, subsystems, or elements. In developing the analogy cost estimate for the new
system or system element, the analyst/estimator must develop and apply the appropriate adjustments.
The major limitation of analogy cost estimation is that it relies on the judgment and experience of the
analyst/estimator to develop and apply those adjustments. Yet there is a dearth of techniques and practical advice in
the literature on the “adjustment mechanism”, especially for advanced technology hardware projects like space
missions. This paper discusses some potential approaches that can improve rigor and repeatability in the analogy
costing process.
My review of the literature revealed a number of relevant cost estimation handbooks that discussed the analogy
approach. The focus, however, was on the necessity of documenting the choice of analog project(s), the adjustment
factors, and the cost estimate. For example, DoD 5000.4M (Ref. 1) states:
* Principal System Engineer/Economist, Mission and System Architecture Section, Caltech/JPL, 4800 Oak Grove
Drive, M/S 301-180, Pasadena, CA 91109, AIAA Member.
A
American Institute of Aeronautics and Astronautics
2
For estimates made by . . . analogy costing techniques, the rationale and procedures used to prepare such an estimates
must be documented. This should include the cost experience used, and the method by which the information was
evaluated and adjusted to make the current cost estimate. If an analog estimate is made using complexity analysis, the
basis for the complexity analysis (including backgrounds of the individuals making the ratings), the factors used
(including the ranges of values), and a summary of the technical characteristics and cost driving elements shall be
provided.
The Department of the Army Cost Analysis Manual (Ref. 2) states similar requirements:
The analyst must show the validity of the direct comparison. A variation to this methodology is to adjust the historical
data to account for some variation in the proposed system, activity, or item. For example, if commercial vehicle data are
used to estimate some aspect of a tactical vehicle, then the historical data might have to be adjusted to accommodate the
impact of complexity or "militarization." It is very important that the analyst document the "adjustment technology" to
show the applicability of the methodology.
No guidance or help is provided on how the create the adjustment factors, leaving room for the analyst/estimator to
apply any number of reasonable judgments in the process.
In contrast, the FAA Life Cycle Cost Estimating Handbook (Ref. 3) devotes an entire chapter to analogy
estimating. Here, the analyst/estimator is asked to provide three factors to be combined multiplicatively with the
analogy project cost. The three factors are (1) a complexity factor, based on design and performance differences
assuming no special miniaturization and manufacturing technology differences; (2) a miniaturization factor; and (3)
a productivity improvement factor, based on improvements in technology (i.e., how much the “production function”
has shifted). The inclusion of a miniaturization factor reflects concern for “stringent” mass and volume constraints
on components and subsystems. That this factor has not played a role in space system analogy costing is curious
since there is both anecdotal and hard evidence that miniaturization has a strong effect on the cost of planetary
rovers, making very small (“nanorovers”) rovers much more expensive per kilogram than larger ones.
The journal literature on analogy cost estimation was not voluminous and tended to deal more with software
projects than hardware. The focus of many of these articles was on empirical/statistical tests of alternative
techniques for developing analogy cost (or effort) estimates, and on quantifying the accuracy of the estimates.
Software projects are typically characterized by a few variables, most notably source lines of code (SLOC), which
are nearly always collected and available to the developing organization, so the existence of data sets for even a
modest number of similar completed projects (20-30) makes some statistical tests possible. These journal articles
were also useful in framing the methodological issues discussed in the next section. Lastly, some articles dealt with
whether analogy cost estimation is better or worse than traditional cost estimating relationships (CERs). On this
point, controversy remains since the empirical evidence is not conclusive either way.
II. Methodological Issues
In analogy cost estimation, three key methodological issues are: (1) determining which analogy projects are the
most appropriate ones to use, (2) the number of analogy projects to include in the adjustment mechanism, and (3)
what adjustment mechanism will be applied. There are also a number of process issues that organizations wishing to
employ analogy estimation must address. These include building an analogy cost database and making it available to
analyst/estimators, automating the process of generating an analogy estimate for a new project (tools), training in the
use of these tools, reconciling the analogy cost estimate with other approaches (validation), and documenting how
the analogy cost estimate was made.
A. Selection of Analogy Projects
The selection of analogy projects can be accomplished subjectively by the analyst/estimator, if the available set
of projects is small and the choice(s) is (are), more or less, obvious. When that is not the case, the creation and use
of a metric to describe the closeness of one project to another that relies on technical and programmatic attributes is
helpful in finding the most appropriate analog projects.
When technical and programmatic attributes are continuous variables, a number of distance metrics have been
discussed in the analogy cost literature.4,5 The most popular ones are shown as Eqs. (1) through (4). The Euclidean
distance in Eq. (1) is both normalized to the available data set and weighted by a set of external multipliers αk.
Normalization and weighting are attempts to balance the importance of each attribute. Normalization is useful in
guarding against an overweighting of some attributes simply because of the units in which they are measured.
The distance metric in Eq. (2), often used in cluster analysis, is similar to the square of the Euclidean metric as
can be seen from its Taylor expansion. The distance metric of Eq. (3) is known as the “city block” or “Manhattan”
metric as it is the “walking” distance when confined to a n-dimensional orthogonal grid. Last, the Eq. (4) metric is
called the Chebychev or maximum distance metric. For reasons cited above, both the city block and maximum
distance metrics should be normalized.
American Institute of Aeronautics and Astronautics
3
(1)
(2)
(3)
(4)
When technical or programmatic attributes are not continuous variables, but fall into discrete categories, then
typically the contribution of that attribute is set to zero if they are identical in the two projects being compared, or
set to one if they differ. Obviously, there is a need then to standardize the values a categorical attribute can take in
any analogy database so that the correct calculation is made. For example, in the original database of space missions
obtained for use in this paper, the terms solar panel, solar array, and solar were used to describe the main power
source. An automated tool for comparing these entries might result in a mismatch. Another way of translating
categorical attributes into numerical values has been suggested by Idri and Abran.6 They propose developing and
using a membership function from fuzzy set theory to go from a linguistic value, such as high, medium, and low, to a
normalized numerical scale.
B. Number of Analogy Projects to Use
The number of analogy projects to use is a joint decision with the adjustment mechanism. With the small data set
(<20 missions with complete technical and high quality cost data) currently available for analogy cost estimation of
space missions, it is likely that there are only one to three good analogy projects.† However, when the number of
appropriate analogy projects in a database is found to be large, as may be the case with software projects, the cost
analyst can take advantage of this with the right choice of adjustment mechanism. The cost analyst must make a
decision based on an examination of the available data.
C. Adjustment Mechanisms
In this section, I outline some possible adjustment mechanisms based on the number of analogy projects
selected, N*, and the number of attributes used in the adjustment, K.
1. N* = 1 with K ≥ 1
Pick the closest analogy project based on the selected distance metric, and scale the known costs up or down
based on a function of the analogy project’s and new project’s K attributes.
(5)
One method for creating the scaling function is to run a standard parametric cost model for both the analogy project
and the new project, and form the ratio. Kellogg and Phan7 attributed this method to Bob Bitten when they applied it
to space instruments using the Multivariate Instrument Cost Model (MICM), developed at the Goddard Space Flight
Center. (See Figure 1.) Parametric cost models for space missions such as NAFCOM and JPL’s PMCM would, of
course, be used in place of MICM when estimating the cost of a new mission.
(5’)
† To make a quick check of this based on the database of space missions used in this paper, I constructed several
histograms of calculated distance metrics comparing a new project (not in the database) against projects in the
database. The results did not contradict this assertion, which the reader may verify using the data in the Appendix.
kk
ki
ki
i
ki
j
k
i
k
kji xx
xx
d
2
,)(min)(max
2
,ln
kj
k
i
k
ji x
x
d
k
j
k
i
kji xxd ,
j
k
i
kkji xxd max
,
analanal
k
new
k
new CxxfC ),(
anal
anal
k
new
k
new C
xNAFCOM xNAFCOM
C)( )(
American Institute of Aeronautics and Astronautics
4
This method has the advantage that one is
not relying entirely on the credibility of the
parametric cost model since it is being used
only in a relative sense. It can also be applied
when only a few of the parametric cost model’s
inputs are known since defaults can be
substituted for those that are as yet
undetermined. However, further
experimentation is needed to check whether this
adjustment mechanism is valid beyond
instruments to full space missions.
2. N* = 1 with K = 2
Pick the closest analogy project based on
the selected distance metric, and scale costs up
or down based on relative scope and
complexity. This method has been applied by
Stuhr on a space-based radar mission.‡
Typically the method is applied at the lowest
practical level of the WBS. Scope addresses
quantity or size, for example, the number
equipment design or assemblies. Complexity addresses how difficult the task is, for example, the stringency of
technical performance requirements or lack of inheritance. Ultimately, scope and complexity rely on expert
judgment, and quantitative measures to support or guide such judgments may not be available early in a new project.
(6)
3. N* = 2 with K > 1
Pick an upper bound analogy project and a lower bound analogy project based on judgment, and scale costs in
between based on a function of the analogy project’s and new project’s K attributes.
(7)
One method of computing the weights is to use the distance metrics for the upper and lower bound analogy projects.
Simple proportions (linear interpolation) could be used in which the lower bound project proportion is assigned to
the upper bound project cost. Another set of weights could be formed that disproportionately favors the closer of the
two projects, as in Eq. (7’).
(7’)
where
A generalization of Eq.(7) would involve an arbitrary N* and providing N* - 1 weights located on a unit simplex.
4. N* ≥ 2 with K = 1 for software project application
Pick a set of analogy software projects and compute their average productivity in source lines of code (SLOC)
per programmer per period. The estimated cost of a new project would then be given by Eq. (8).
(8)
‡ Private communication with Fred Stuhr, JPL Radar Science and Engineering Section, March 12, 2004
Actual Cost of Analog
Xnew
NAFCOM
Xanal
Est Cost of New Proj
Cost Model Est of Analog
Cost Model Est of New Proj
Actual Cost of Analog
Xnew
NAFCOM
Xanal
Est Cost of New Proj
Cost Model Est of Analog
Cost Model Est of New Proj
Figure 1. Bitten Analogy. Though the use of NAFCOM (NASA /
Air Force Cost Model) is shown, any parametric space mission
cost model could be used to scale an analogy project’s cost to
estimate the cost of the new space mission.
lanaluanallanal
k
uanal
k
new
k
new CCxxxC ____ )1(),,(
lanaluanalnew CCC _2_2 )(cos)(sin
2_,
2_,
2_,
2
sin
uanalnewlanalnew
lanalnew dd
d
*)(Nx
SLOC
wC
prod
new
new
anal
anal
comp
new
comp
anal
scope
new
scope
new C
x
x
x
x
C
American Institute of Aeronautics and Astronautics
5
Typically the average productivity would be calculated from the analogy projects’ SLOCs and costs. Wage rates
serve to adjust the costs to account for inflation, if costs are in nominal dollars in the database.
III. Application
Whereas the last section discussed distance metrics and potential adjustment mechanisms in the abstract, the
objective in this section is to describe some simple “experiments” that used real data from space missions to form an
analogy cost estimate for a new mission. I obtained a database of actual costs, technical, and programmatic attributes
for those JPL missions shown in the Appendix. The costs were already adjusted for inflation so all costs were in
$FY04M. Technical attributes included spacecraft mass, launch vehicle, subsystem type, trajectory, and redundancy
information. Programmatic attributes included
phase durations, program type, spares and reliability
class, and spacecraft developer information. The
new mission (not in the database) was a nearly
completed Mars orbiter project.
From these data, I first computed a variety of
distance metrics, using three types of information
that presumably would be available even in the
earliest phases of a new project. The purpose of this
was to determine whether a consistent closest
analogy project would arise in this test case. Table 1
shows the attributes that were used for each distance
metric, and the results are shown in the Appendix.
Next, I performed an analogy cost estimate using
upper and lower bound analogy projects to constrain
the estimate. The results of these experiments are
reported below.
A. Closest Analogy Project
The closest analogy project was not independent of the types of information used to compute the distance
metrics. One might surmise that the closest analogy project would be another Mars orbiter mission; on this basis, the
schedule-based metrics missed the mark, while the mass-based metrics performed consistently and as experience
might suggest. The metrics based on target body and design choices tended to pick out the Mars missions, but the
Chebychev version identified more than half the database projects as closest analogies. This is a consequence of
using categorical attributes, which is not recommended in this case.
B. Estimating the Cost By Analogy
Since the mass-based distance metrics appeared to be consistent with judgment, I selected the Mars Observer
mission as an upper bound analogy project and the Mars Odyssey (also an orbiter) as a lower bound analogy project.
I computed the weights in Eq. (7) using both Eq. (7’) and simple linear interpolation for each of the four mass-based
metrics, and applied these weights to the actual
cost of the analogy projects. The actual costs
applied represented the full life-cycle costs
(Phases A through E), though I could have chosen
to estimate just Phase C/D costs. Table 2 shows
the results. The simple linear interpolation
performed better in this test case, though all
estimates were within 20% of the actual cost of
the new mission. The Euclidean distance metric
performed uniformly better than the others, and in
combination with the linear interpolation,
produced a remarkably accurate estimate.
k
Schedule
Mass
Target/Design
1
Phase A/B
duration
Flight system
dry mass
Target body
2
Phase C/D
duration
Flight system
wet mass
Parts class
3
Phase E
duration
Main power
source type
4
S-Band
5
Ka-Band
6
X-Band
7
UHF
8
Propulsion type
Table 1. Parameters Used To Build Distance Metrics.
Each of the parameters were normalized and weighted
equally within their respective distance metrics. The design
choices attributes were composed entirely of “categorical”
parameters, which prohibited the use of the log ratio metric.
Target bodies were represented by numerical values.
Weights
Euclidean
Log Ratio
City Block
Chebychev
Using Eq.(7’)
1.132
1.197
1.192
1.186
Using Linear
Interpolation
1.011
1.172
1.132
1.109
Table 2. Ratio of Estimated Cost to Actual Cost. Results
for different distance metrics based on flight system mass
and different interpolation techniques are shown. The
“actual” cost for the test case represents the nearly
completed project’s actuals-to-date plus an estimate of
remaining cost, including reserves.
American Institute of Aeronautics and Astronautics
6
IV. Conclusion
Far more experimentation, test cases, and data are needed to improve analogy cost estimation for space
missions. From the test case involving an upcoming Mars orbiter mission, the distance metrics most aligned with
judgment are those that combine simple mass attributes—flight system dry mass and flight systems wet mass. Those
based on schedule did not perform well, and those based on design choices produced too many analogy projects.
Some combination of mass parameters and design choices remains unexplored. In estimating the cost of the Mars
orbiter mission using upper and lower bound analogy projects to bracket the cost, the Euclidean distance metric and
simple linear interpolation did remarkably well. However, more work is needed to confirm whether this is
repeatable.
American Institute of Aeronautics and Astronautics
7
Appendix
Distance Metrics Based On
Schedule Parameters
Mass Parameters
Target and Design Choices
Mission Name
Euclidean
Log Ratio
City Block
Chebychev
Euclidean
Log Ratio
City Block
Chebychev
Euclidean
Log Ratio
City Block
Chebychev
QuikScat
0.327
1.688
0.556
0.235
0.203
0.537
0.041
0.036
2.236
8.000
4.000
Mars Global Surveyor
0.334
6.950
0.545
0.235
0.217
0.517
0.047
0.030
1.414
2.000
1.000
Genesis
0.116
0.159
0.181
0.084
0.329
1.724
0.108
0.064
2.000
4.000
1.000
Stardust
0.180
0.155
0.272
0.160
0.422
4.024
0.178
0.091
1.414
2.000
1.000
Magellan
0.267
0.512
0.378
0.196
0.158
0.289
0.025
0.024
2.449
7.000
2.000
Mars Odyssey
0.226
0.348
0.355
0.193
0.351
1.852
0.123
0.068
1.000
1.000
1.000
Mars Climate Orbiter
0.497
5.253
0.684
0.462
0.370
2.258
0.137
0.072
1.000
1.000
1.000
Jason 1
0.287
0.574
0.463
0.235
0.355
2.390
0.126
0.078
2.236
8.000
4.000
Cloudsat
0.368
1.060
0.572
0.319
0.222
0.712
0.049
0.044
2.236
8.000
4.000
Grace
0.233
0.211
0.352
0.210
0.173
0.343
0.030
0.027
2.236
8.000
4.000
Mars Pathfinder
0.458
3.495
0.540
0.454
0.217
0.659
0.047
0.042
1.414
2.000
1.000
NM Deep Space 1
0.463
3.857
0.500
0.462
0.386
2.857
0.149
0.080
2.000
4.000
1.000
SIRTF
0.416
1.001
0.625
0.300
0.232
0.787
0.054
0.047
2.000
4.000
1.000
Deep Impact
0.423
1.708
0.629
0.370
0.189
0.489
0.036
0.035
1.732
3.000
1.000
Mars Exploration Rover (MER A of 2 S/C)
0.334
1.040
0.482
0.311
0.183
0.450
0.033
0.033
1.414
2.000
1.000
Mars Polar Lander
0.476
4.221
0.677
0.437
0.332
1.912
0.111
0.070
1.000
1.000
1.000
Mars Observer
0.757
4.599
1.276
0.536
0.105
0.061
0.011
0.008
1.732
3.000
1.000
Cassini
1.003
2.939
1.666
0.765
0.992
2.049
0.984
0.496
2.449
7.000
2.000
Galileo
0.883
1.776
1.172
0.836
0.159
0.106
0.025
0.023
2.646
8.000
2.000
Voyager (2 S/C)
0.586
1.479
0.749
0.569
0.244
0.851
0.059
0.048
2.449
7.000
2.000
Infimum
0.116
0.155
0.181
0.084
0.105
0.061
0.011
0.008
1.000
na
1.000
1.000
American Institute of Aeronautics and Astronautics
8
Acknowledgments
The research described in this paper was carried out at the Jet Propulsion Laboratory (JPL), California Institute
of Technology, under a contract with the National Aeronautics and Space Administration. The author wishes to
acknowledge the support of JPL Costing Office and its manager, Robert Metzger.
References
1Department of Defense, “Cost Analysis Guidance and Procedures”, DoD 5000.4M, December 1992, pp.32.
2U.S. Army Cost and Economic Analysis Center, “Department of the Army Cost Analysis Manual”, May 2002, pp. 34.
3Federal Aviation Administration, “FAA Life Cycle Cost Estimating Handbook” June 2002.
4Angelis, L., and Stamelos, I., “A Simulation Tool for Efficient Analogy Based Cost Estimation,” Empirical Software
Engineering, Vol. 5, 2000, pp. 35-68.
5Briand, L., Langley, T., and Wieczorek, I., “A Replicated Assessment and Comparison of Common Software Cost
Modeling Techniques,” International Software Engineering Network, Technical Report ISERN-99-15, 1999.
6Idri, A., and Abran, A., “A Fuzzy Logic Based Set of Measures for Software Project Similarity: Validation and Possible
Improvements,” 1530-1435/01, Proceedings of the IEEE, 2001, pp. 85-96.
7Kellogg, R., and Phan, S., “An Analogy-based Method for Estimating the Costs of Space-Based Instruments” IEEE
Aerospace Conference, CP1160, Vol. 8, IEEE, New York, 2003, pp. 3843-3847.
