Content uploaded by David G. Ullman
Author content
All content in this area was uploaded by David G. Ullman on Sep 14, 2019
Content may be subject to copyright.
1
Analysis of Alternatives (AoA) Based Decisions
Dr. David G. Ullman and LTC(R) Richard Ast1
MORS, Phalanx, Vol 44, No 3, Sept 2011, pp 24
The Analysis of Alternatives (AoA) is a process that has been adopted by the Office of
Management and Budget (OMB) and the Department of Defense (DoD) to ensure that
multiple alternatives have been analyzed prior to making investment decisions. It is an
assessment approach to evaluate potential solution sets (material, organizational,
structural, or ideological) to satisfy a desired capability. An AoA moves the justification
of a single alternative to the exploration of multiple options in order to establish a basis
for funding the best possible projects in a rational, defensible manner while considering
risk and uncertainty. As with other decision-making tools, techniques, and procedures,
a mature AoA is best used in a higher-level decision-making context.
What Is AoA? Why Conduct Them?
Analysis of Alternatives is the detailed analytical comparison of multiple options before
committing resources to an objective or goal. The practice of comparing multiple
alternative solutions has long been a part of engineering practice (Ullman, 2009,
especially Chapter 7, Concept Generation). However, there is a natural tendency to
pre-select a single alternative and justify it rather than compare multiple options with the
goal of choosing the best one. Justification appears easier than evaluation when
making a learned decision. Thus, government agencies, such as OMB and DoD, have
made it necessary to require the use of a rigorous AoA process when proposing
program solutions.
To facilitate this AoA introduction, there are 4 levels of AoA maturity:
Level 1 – Propose one alternative and justify it.
Level 2 – Propose multiple alternatives and provide a one-dimensional
comparative analysis with some inclusion of uncertainty effects.
Level 3 – Propose multiple alternatives and provide multi-dimensional
comparative analysis with some inclusion of uncertainty effects.
Level 4 – Propose multiple alternatives, and provide multi-dimensional
comparative analysis and support robust resource allocation decisions with the
inclusion of uncertainty effects.
These levels include measures about the number of alternatives considered, the
inclusion of uncertainty in the analysis, and the level of decision support. Additional
levels could be defined by considering these measures separately, but these four levels
are sufficient for the processes used today.
1 Dr Ullman can be reached at ullman@davidullman.com or 1-541-760-2338. Other material by him is at
www.davidullman.com. LTC(R) Richard Ast is a Senior Analyst for Analytic Services Inc. (ANSER), based out of
Arlington, VA.
2
This paper will show how OMB provides funding approval from Level 1 to Level 2, and
how DoD sets a goal to achieve Level 3 AoA. Further, the paper will show that current
AoA methods only go part of the way to achieving the highest potential – Level 4 AoA.
Keep in mind that the ultimate goal of an AoA exercise is to enable making the best
possible decision about resource allocations where this decision is based on uncertain,
incomplete, evolving, and conflicting estimates of cost, performance, and other critical
measures. In this paper, the OMB and DoD approaches to AoA are discussed first, and
then AoA‟s potential will be explored.
The Value of an AoA Depends on Estimation and Risk
The “analysis” in AoA refers to making estimates of future costs, delivery schedule,
performance and other critical measures in order to understand the risk of following a
course of action. As Chapter 3 in Making Robust Decisions (Ullman, 2006) points out
about estimates, “Where the past performance may be known, the present is obscured
by its immediacy and the future is a best guess.” The best guess is clouded in
uncertainty and uncertainty results in risk. Uncertainty comes from many sources2, and
these can be characterized by the types of resulting risks: technical risks, programmatic
risks, operational risks and decision risks. The first three are typically part of every AoA;
but the last one, decision risk, the risk of choosing the wrong alternative when
performing an AoA, is often omitted and needs to be known.
Time, cost, and performance estimations are notoriously inaccurate. In one government
agency, cost overruns range from 31% (small projects) to 315% (very large projects) 3.
As another example of estimation inaccuracy, in the Chaos Report (Standish Group,
2000 and 2004) an annual analysis of information technology (IT) projects, 51% of all IT
projects were delivered late or over budget in 2004 and an additional 15% were
cancelled. Further, projects completed by large companies had only 42% of the
originally designed features and functions. It should be noted that the Chaos Report
numbers may actually be understated, as they are self-reported.
In a simple estimation exercise described in Making Robust Decisions, time estimates
were made for a basic, everyday task by hundreds of attendees at a conference. The
resulting estimates averaged 32 minutes with a standard deviation of 10 minutes. In
other words over 30% of the estimates were more than 10 minutes more or less than
the average. Further, by simply changing the wording of the estimate request, the
average estimate dropped to 17 minutes. In other words, by asking a single estimator
for the time required to do a task, even a common one, will result in an estimate that is
not much better than a guess.
Risk is due to uncertainty - without any uncertainty, reality will match the estimate and
the risk will be zero. Formally, risk is the likelihood of something going wrong times the
consequences if it does. The goal of including uncertainty in AoA is to help analyze
risk. In terms of the estimation exercise example in the previous paragraph, it should be
2 Eleven specific sources are cited in Making Robust Decisions. Ullman 2006.
3 Will need to specify source.
3
possible in an AoA to include the uncertainty of time estimates as they may have a
marked impact on cost and material maturity when it comes to making decisions.
The Office of Management and Budget AoA Effort
Part 7 (Section 300) of the OMB Circular A-11 (OMB, 2008) establishes a policy for
planning, budgeting, acquisition, and managing Federal capital assets, and gives
instructions on budget justification and reporting requirements. This is an effort to move
organizations from justifying a single alternative, Level 1 AoA, to the comparison of
multiple alternatives. Within the OMB and other government agency literature, AoA is
often referred to as “Alternatives Analysis.” Details on alternatives analysis is given in
Appendix A of GSA‟s IT Budget Submission Instructions (GSA, 2007).
In order to achieve Level 2 AoA, Section 300 requires that an organization identify and
consider at least three viable alternatives, in addition to the current baseline (i.e., the
status quo). These alternatives need to be presented in a table that shows:
Alternative Analyzed
Description of Alternative
Risk Adjusted Lifecycle Costs estimate – the overall estimated cost over the life
of the investment that has been adjusted to accommodate any risk identified
Risk Adjusted Lifecycle Benefits estimate – projected benefits and costs for each
viable alternative
The GSA IT Budget Submission Instructions goes on to say that the following
quantitative and qualitative benefits should be addressed when evaluating total annual
benefits for each alternative:
Qualitative Benefits
Cost Savings
Cost Avoidance
Stakeholder Benefits
Non-Monetary Quantitative Benefits
In both the OMB and GSA documents, the comparison is based on Net Present Value
(NPV), an effort to reduce all measures to their dollar values. There is great comfort in
having a single dollar value for each project. But, is this value sufficient for actually
committing resources? Using only NPV has the following shortcomings:
The accuracy of the data is suspect; using a single indicator of project value only
combines inaccurate estimates, thus compounding the error.
Risk estimates are added to NPV and are often no better than an educated
guess, further compounding the error.
NPV penalizes projects with longer-term launch dates.
NPV assumes that risk (uncertainty) is spread out evenly over the life of a
project, which is often not true.
4
It is difficult to measure everything in terms of dollars. Time is money, but time
estimates are often inaccurate.
In concluding an AoA study for the OMB, the organization must also provide information
describing the estimating technique used, why the selected alternative was chosen, and
what specific qualitative benefits will be realized. The level of detail and rigor of a cost
benefit analysis should be commensurate with the size, complexity, and cost of a
project. Cost/benefit projections should be calculated for all viable alternatives.
To accommodate the risks in the estimates, the OMB and GSA give little guidance. The
best to be found is in Section 9 of Guidelines and Discount Rates for Benefit-Cost
Analysis of Federal Programs (OMB, 2006). Here it states that estimates of benefits
and costs are typically uncertain because of imprecision in both underlying data and
modeling assumptions. The guidance in the half-page of coverage is limited to
“objective estimates of probabilities should be used whenever possible,” and the
suggestion that any limitations of the analysis because of uncertainty should be
discussed.
What should be drawn from the above discussion is that OMB forces AoA Level 1
policies to Level 2 using solely NPV. NPV is certainly one appropriate measure, but it is
not the only measure that should be included in an AoA.
The Department of Defense AoA Effort
In DoD AoA is used in the decision-making process to support acquisition of new
capabilities and systems. By definition, “(t)he AoA shall focus on identification and
analysis of alternatives, measures of effectiveness, cost, schedule, concepts of
operations, and overall risk. The AoA shall assess the critical technology elements
(CTEs) associated with each proposed materiel solution, including technology maturity,
integration risk, manufacturing feasibility, and, where necessary, technology maturation
and demonstration needs” 4. DoD instruction 5000.2 (2008) provides AoA procedural
guidance for potentially high cost projects referred to as Acquisition Category I (ACAT 1
or 1A) programs. The procedure describes the requirement of the Milestone Decision
Authority (MDA) to approve AoA guidance to the service lead or to the Principle Staff
Assistant in charge of the mission area. Which in turn designate responsibility to
complete the AoA study plan. In order to avoid conflict of interest, the program
manager (PM) will not be assigned that responsibility. The AoA study plan must be
approved by the MDA prior to the start of the AoA. The OSD Cost, Assessment &
Program Evaluation office provides an independent assessment of the AoA to the MDA
(CAPE - formally known as Program Analysis & Evaluation (PA&E)) in the following
areas5:
Illuminated capability advantages and disadvantages
Considered joint operational plans
Examined sufficient feasible alternatives
4 DoD Instruction 5000.2, December 8, 2008, page 15.
5 DoD Instruction 5000.2, December 8, 2008, page 58.
5
Discussed key assumptions and variables and sensitivity to changes in these
Calculated costs
Assessed Technology risk and maturity, Alternative ways to improve the energy
efficiency, and Appropriate system training
Although the AoA is only required by statute for the initial milestone decision, (milestone
A) updates may be necessary for follow-on critical decisions (milestone B and C). Each
update has renewed guidance and the need for an approved study plan. The AoA
study plan describes “how‟ the AoA will be conducted to include approach and
methodology.
Each Service has its own AoA methodology, for example, the Army headquarters
Operations (G3/5/7) issues study guidance as a formal tasking to the analytical agency
of choice. Typically, the Training and Doctrine Command (TRADOC) is the lead
organization. TRADOC in turn issues a tasking to a subordinate analytical center within
its organization. Although the multilayered process may appear cumbersome, the
detailed command structure clearly identifies an approval chain enabling the AoA to
progress in the desired direction. The analytical organization responsible for the AoA
uses a series of regulations and pamphlets out of the Army Headquarters and TRADOC
to provide additional guidance on the conduct of AoA, but not to the level of specificity
that would hamper analytical organizations from tailoring each AoA to individually
assess and evaluate the potential program6. The Army‟s TRADOC Analysis Center
(TRAC) often is responsible for the conduct of ACAT 1 and II programs while the
mission area leads (school houses) are responsible for less cost prohibitive programs.
Similar to the Army, the Assistant Secretary of the Navy Research, Development and
Acquisition (ASN (RDA)) releases guidance on AoA preparation and a proposal is
prepared in coordination with major stakeholders. The interested parties may include
decision makers (program sponsor) and the program manager team. The AoA is
conducted by a service provider which can be a Navy Study Center (for example, the
Center for Naval Analysis (CNA)) or one of the many federally funded research and
development centers (FFRDC). The Navy forms an oversight board and receive
feedback of AoA development to include study plan approval that includes the
methodology and approach7.
Perhaps the best documented AoA methodology and approach is detailed in the Air
Force Materiel Command Analysis of Alternatives (AoA) Handbook (USAF, 2008). It is
a handbook of useful analysis tools and techniques. We will use material from it in this
discussion.
6 The primary references for the Army are Army Regulation 71-9 (AR-71-9), December 28, 2009; TRADOC
Regulation 10-5-7 which describes the mission role of TRAC organizations, TRADOC Regulation 71-20 used for
force development, and TRADOC Pamphlet 350-70-6 on the conduct of analysis.
7 Defense Acquisition University, Analysis of Alternatives, April 2006 page 4.
6
DoD AoA studies span two main categories of measures, effectiveness, and cost. The
combination of effectiveness and cost results in a set of multiple measures; thus DoD
AoA studies are at Level 3.
Cost analysis is performed similar to the methods suggested in the OMB and GSA
literature. But instead of translating all measures into NPV, the DoD also considers
effectiveness analysis. Measuring effectiveness is normally the most complex element
of the AoA and consumes a significant fraction of AoA resources. The goal of the
effectiveness analysis is to determine the military worth of the alternatives relative to
qualitative or quantitative measures. These focus on a system„s performance or
characteristics that indicates the degree to which it performs the goal task. These
measures of effectiveness (MOEs) are:
Quantitative when feasible (e.g., "the number of targets held at risk," or "the
number of targets by type that you can hold at risk in daytime and nighttime
conditions")
Qualitative when necessary, calling on the opinion of a knowledgeable person or
group, (e.g., "In your opinion does the solution provide a day-night capability?")
Universal across all the alternatives, as all alternatives are evaluated using all
MOEs
Independent – not strongly correlated with one another (to avoid
overemphasizing particular aspects of the alternatives)
There is only minimal consideration of risk and uncertainty in the handbook. The
section covering this topic (7.3.4) concludes with, “Several approaches are available to
treat risk in an estimate; they range from very subjective to those with complex
statistics. Whatever risk methodology the cost analyst decides to employ, it should be
adequately described in the study plan. The results of the risk analysis will be included
in the final cost estimates.”
The Air Force Handbook clearly recognizes that the goal of an AoA study is to make a
decision, but not sufficiently to be considered Level 4. The methods for alternative
comparison in the handbook are paraphrased and will serve as a basis for discussion in
the final part of this paper.
The Handbook suggests a filtering of possible alternatives to eliminate those that are
not viable, cost effective, or otherwise lacking as shown in Figure 1, taken from the
Handbook. This filtering (similar to that suggested in The Mechanical Design Process),
is strong, but grows weak when discussing how to select among the finalists (Options 2,
6, and 7, in Figure 1).
7
Figure 1, Filtering of Alternatives
What is suggested is that plots of cost versus effectiveness be made to support the
alternatives comparison. These can show the cost- effectiveness trade-off. The
Handbook is never quite clear about how to combine the MOEs into a single
“effectiveness,” metric or how to find uncertainty. In the discussion, the authors go on to
say, in highlighted font, that there is generally no requirement for an AoA to identify a
SINGLE solution. But if the goal is to support decision making, then guidance about
how to get to a single solution may be necessary.
Once the analysis is complete, it is useful to present a summary of the key
discriminators for each alternative side-by-side before presenting the conclusions and
recommendations drawn from all of the analysis. Figure 2 shows an example of this
sort of presentation, an alternative comparison matrix, where LCC represents Life Cycle
Cost. This kind of depiction ensures that the report reader or briefing audience has a
summary picture of the results in mind (and for reference) as the conclusions and
recommendations are made.
Figure 2, Alternative Comparison Matrix
8
The next step in this process is to find a way to clearly identify for the decision makers
the advantages and disadvantages of each alternative, especially how the alternatives
address the required capabilities and answer the high-level issues/questions in the AoA
guidance.
Where the DoD approach to AoA is more mature than that of OMB, it could still improve
in two areas: handling of uncertainty and a decision centric approach to the problem.
Taking the AoA to the Next Level
As stated in the introduction, the ultimate goal of AoA is to support making resource
allocation decisions. Where OMB pushes Federal agencies to Level 2 AoA and the
DoD manages Level 3, neither meets the qualifications for Level 4. In this section we
will explore what it will take to meet the stated definition:
Level 4 – Propose multiple alternatives, and provide multi-dimensional comparative
analysis and support robust resource allocation decisions with the inclusion of uncertainty
effects.
This definition has the following constituent parts:
A decision process approach
Multi-dimensional qualitative and quantitative comparisons
An integration of estimation uncertainty
The ability to fuse evaluations to give guidance to the process
Of these, the second part is developed in the USAF AoA Handbook and the third and
fourth parts are recognized but could use additional support. The following describes
what is needed to fulfill all these AoA needs.
At times, it may seem that the conduct of an AoA is a requirement to do analysis for the
sake of analysis - unless the final goal is kept in sight: from a set of alternatives, choose
the one most likely to be successful or identify what needs to be done next to make this
decision. Thus, an AoA should center on the elements of a decision process. As
shown in Figure 3, there are four main activities necessary for information generation
and refinement during the decision-making process: understand, evaluate, fuse, and
decide. Note that the decision to choose an alternative – the last item in “what-to-do-
next” – is only one activity of the decision-making process.
9
Figure 3, Decision-making Process
For an AoA, the following apply to the items in Figure 3:
The issue is, “Choose the best solution before committing resources.”
The alternatives are the projects or proposed solution sets. The value of the
study is only as good as the options proposed and evaluated.
The criteria are the measures for evaluating the alternatives. The USAF
Handbook spends a considerable effort on how to construct criteria (i.e., MOEs)
that can lead to a good study.
Evaluations have two components: evaluating how important each of the criteria
is for the stakeholders; and assessing the value of the alternatives by comparing
them to the criteria. The USAF Handbook is very strong on the evaluation of the
alternatives relative to the criteria. There are two “howevers.”
o All evaluations are uncertain, and this uncertainty needs to be identified
and analytically included as part of the qualitative and quantitative
evaluations.
o The importance held by each of the various stakeholders may be
inconsistent. One way to build decision buy-in is to honor each of these
viewpoints in the analysis.
Decisions depend on fused evaluation results. Fusion has three difficult factors:
Understand the Problem
Clarify the issue
Generate alternatives
Develop criteria
Evaluate
The importance of criteria
The value of alternatives
relative to the criteria
Fuse evaluation results to develop
decision measures
Stakeholders and
decision makers
Decide what-to-do-next
Work to gain consensus
Reduce uncertainty
Refine criteria
Refine alternatives
Choose an alternative and
document the deliberation and
decision
Move to next
issue
10
o Evaluations are often a mix of qualitative and quantitative. These must be
combined in some consistent manner.
o Evaluation results might come from multiple sources and may be
inconsistent with one another. This is especially true for qualitative
evaluations.
o Evaluation results are uncertain and the uncertainty must be fused in a
logical manner.
Based on the fused results, decide “what-to-do-next.” This decision will direct the
process down one of three possible paths:
o Improve understanding: refine the alternatives and criteria
o Refine evaluation: work to gain consensus and reduce uncertainty
o Select an alternative
One feature of Figure 3 is that two of the paths leading from “decide what-to-do-next” go
back to earlier activities. These arrows emphasize the information evolution and
refinement that are inherent in making a robust decision. The diagram also shows that
the stakeholders are involved in all parts of the decision-making process.
Conclusion: AoA in Context
The effectiveness of an AoA depends upon the decision-making context in which it is
implemented. Attempting to use an AoA in the context of reactive, quick knee-jerk
decision making is an exercise in futility. An AoA simply will not work in this type of
framework. Knowledge-based decision making may or may not be conducive to the use
of AoA; that would depend on the quality of the data, information, and information used
for decision making.
Systematic decision making may or may not use an AoA in a documented, repeatable
process. Similarly, aligned decision making strengthens the use of AoA by connecting
the decision-making process to organizational purpose, vision, mission, strategies,
goals, objectives, plans, programs, projects, routine tasks, and performance metrics.
Integrated decision making, by considering internal and external factors and
stakeholders, is an even more robust context for an AoA.
Architected decision making is the framework that enables the most robust criteria-
based decision making. Framing decisions within a holistic context of the enterprise‟s
people, processes, technology, leadership/ management infrastructure, and change-
management approach, and forcing decision makers to construct criteria that reflect the
totality of the organization‟s architecture, provides the most promise of decision making
and decisions. It is in this type of framework where Analysis of Alternatives will yield the
most effective and useful decisions.
11
References
Department of the Army, Headquarters, United States Army Regulation 71-9, Force
Development Warfighting Capabilities Determination. Washington, DC, 2009.
http://www.fas.org/irp/doddir/army/ar71-9.pdf
Department of the Army, Headquarters, United States Army Training and Doctrine
Command, TRADOC Regulation 10-5-7, Organization and Functions United
States Army TRADOC Analysis Center, Fort Monroe, VA, 2010.
http://www.tradoc.army.mil/tpubs/regs/r10-5-7.pdf
Department of the Army, Headquarters, United States Army Training and Doctrine
Command, TRADOC Regulation 71-20, Force Development Concept
Development, Experimentation, And Requirements Determination, Fort Monroe,
VA, 2011.
http://www.tradoc.army.mil/tpubs/regs/tr71-20.pdf
Department of the Army, Headquarters, United States Army Training and Doctrine
Command, TRADOC Pamphlet 350-70-6, Training Systems Approach to
Training Analysis, Fort Monroe, VA, 2004
http://www.tradoc.army.mil/tpubs/pams/p350-70-6.pdf
Department of the Navy, Acquisition and Capabilities Guidebook, The Deputy Assistant
Secretary of the Navy (Research, Development and Acquisition) Acquisition and
Logistics Management, Washington, DC, 2008.
https://acquisition.navy.mil/rda/content/download/5797/26501/version/1/file/SEC
NAV+M-5000.2+-+
Greenberg, Marc and Gates, James; Analysis of Alternatives. Defense Acquisition
University, Business, Cost Estimating, and Financial Management Department.
April 2006, https://acc.dau.mil/adl/en-US/30371/file/5377/05%20-
%20Analysis%20of%20Alternatives%20TN-%20Apr%2006.pdf
Standish Group, Chaos Reports 2000 and 2004, https://secure.standishgroup.com/
Ullman, David G. Making Robust Decisions: Decision Management for Technical,
Business, & Service Teams. Victoria, B.C.: Trafford Publishing, 2006.
------- The Mechanical Design Process, 4th edition. New York: McGraw Hill, 2009.
United States Air Force. Analysis of Alternative (AoA) Handbook: A practical Guide to
Analysis of Alternatives. Kirtland AFB, NM: Air Force Materiel Command
(AFMC„s) Office of Aerospace Studies (OAS), 2008.
http://www.oas.kirtland.af.mil/AoAHandbook/AoA%20Handbook%20Final.pdf.
United States General Services Administration. IT Budget Submission Instructions:
Guide for Major IT Initiatives (BY2009 Exhibit 300 & Exhibit 53). Washington,
DC: Office of the Chief Information Officer, 2007,
http://colab.cim3.net/file/work/CPIC/GSA%20BY09%20for%20IT%20Majors_7.11
_v3.1.doc.
United States Government Accountability Office (GAO), Defense Acquisitions, Many
Analyses of Alternatives Have Not Provided a Robust Assessment of Weapon
System Options, GAO-09-665, 2009.
http://www.gao.gov/new.items/d09665.pdf
12
United States Office of Management and Budget. Circular No. A–11, Preparation,
Submission, And Execution Of The Budget. Washington, DC: Executive Office of
the President, 2008.
http://www.whitehouse.gov/omb/circulars/a11/current_year/a_11_2008.pdf.
United States Office of Management and Budget. Circular No. A-94, Guidelines and
Discount Rates for Benefit-Cost Analysis of Federal Programs. Washington, DC:
Executive Office of the President, 1992, Revised 2006.
http://www.whitehouse.gov/omb/circulars/a094/a094.html.
