Multi-criteria decision making: An operations research approach

Abstract

The core of operations research is the development of approaches for optimal decision making. A prominent class of such problems is multi-criteria decision making (MCDM). The typical MCDM problem deals with the evaluation of a set of alternatives in terms of a set of decision criteria. This paper provides a comprehensive survey of some methods for eliciting data for MCDM problems and also for processing such data.

Key words: Decision making, optimization, pairwise comparisons, sensitivity analysis, operations research.

Full text

1
Multi-Criteria Decision Making: An
Operations Research Approach
E. Triantaphyllou, B. Shu, S. Nieto Sanchez, and T. Ray
Department of Industrial and Manufacturing Systems Engineering
3128 CEBA Building
Louisiana State University
Baton Rouge, LA 70803-6409, U.S.A.
E-mail: trianta@lsu.edu
Abstract: The core of operations research is the development of approaches for optimal decision making. A
prominent class of such problems is multi-criteria decision making (MCDM). The typical MCDM problem deals with
the evaluation of a set of alternatives in terms of a set of decision criteria. This paper provides a comprehensive
survey of some methods for eliciting data for MCDM problems and also for processing such data.
Key words: Decision making, optimization, pairwise comparisons, sensitivity analysis, operations research.
1Multi-Attribute Decision Making: A General Overview
Multi-Attribute Decision Making is the most well known branch of decision making. It is a branch of a
general class of Operations Research (or OR) models which deal with decision problems under the presence of a
number of decision criteria. This super class of models is very often called multi-criteria decision making (or
MCDM). According to many authors (see, for instance, [Zimmermann, 1991]) MCDM is divided into
Multi-Objective Decision Making (or MODM) and Multi-Attribute Decision Making (or MADM).
MODM studies decision problems in which the decision space is continuous. A typical example is
mathematical programming problems with multiple objective functions. The first reference to this problem, also
known as the "vector-maximum" problem, is attributed to [Kuhn and Tucker, 1951]. On the other hand, MADM
concentrates on problems with discrete decision spaces. In these problems the set of decision alternatives has been
predetermined.
Although MADM methods may be widely diverse, many of them have certain aspects in common [Chen and
Hwang, 1992]. These are the notions of alternatives, and attributes (or criteria, goals) as described next.
Alternatives:
Alternatives represent the different choices of action available to the decision maker. Usually, the set of alternatives
is assumed to be finite, ranging from several to hundreds. They are supposed to be screened, prioritized and
eventually ranked.
Multiple attributes:
Published in:
Encyclopedia of Electrical and Electronics Engineering, (J.G. Webster, Ed.), John Wiley & Sons, New
York, NY, Vol. 15, pp. 175-186, (1998).

2
Each MADM problem is associated with multiple attributes. Attributes are also referred to as "goals" or "decision
criteria". Attributes represent the different dimensions from which the alternatives can be viewed.
In cases in which the number of attributes is large (e.g., more than a few dozens), attributes may be arranged
in a hierarchical manner. That is, some attributes may be major attributes. Each major attribute may be associated
with several sub-attributes. Similarly, each sub-attribute may be associated with several sub-sub-attributes and so
on. Although some MADM methods may explicitly consider a hierarchical structure in the attributes of a problem,
most of them assume a single level of attributes (e.g., no hierarchical structure).
Conflict among attributes:
Since different attributes represent different dimensions of the alternatives, they may conflict with each other. For
instance cost may conflict with profit, etc.
Incommensurable units:
Different attributes may be associated with different units of measure. For instance, in the case of buying a used car,
the attributes "cost" and "mileage" may be measured in terms of dollars and thousands of miles, respectively. It is
this nature of having to consider different units which makes MADM to be intrinsically hard to solve.
Decision weights:
Most of the MADM methods require that the attributes be assigned weights of importance. Usually, these weights
are normalized to add up to one. How these weights can be determined is described in section 6.2.
Decision matrix:
An MADM problem can be easily expressed in matrix format. A decision matrix A is an (M × N) matrix in which
element aij indicates the performance of alternative Ai when it is evaluated in terms of decision criterion Cj, (for i =
1,2,3,..., M, and j = 1,2,3,..., N). It is also assumed that the decision maker has determined the weights of relative
performance of the decision criteria (denoted as Wj, for j = 1,2,3,..., N). This information is best summarized in figure
1. Given the previous definitions, then the general MADM problem can be defined as follows [Zimmermann, 1991]:
Definition 1-1:
Let A = { Ai, for i = 1,2,3,... ,M} be a (finite) set of decision alternatives and G = {gi, for j = 1,2,3,..., N} a (finite)
set of goals according to which the desirability of an action is judged. Determine the optimal alternative A* with
the highest degree of desirability with respect to all relevant goals gi.
Criteria
C1 C2 C3 ... CN
Alt. W1 W2 W3 ... WN
_______________________________________
A1 a11 a12 a13 ... a1N
A2 a21 a22 a23 ... a2N
A3 a31 a32 a33 ... a3N
. . . . . .
. . . . . .
. . . . . .
AM aM1 aM2 aM3 ... aMN
Figure 1: A Typical Decision Matrix.
Very often, however, in the literature the goals gi are also called decision criteria, or just criteria (since the

3
alternatives need to be judged (evaluated) in terms of these goals). Another equivalent term is attributes. Therefore,
the terms MADM and MCDM have been used very often to mean the same class of models (i.e., MADM). For these
reasons, in this paper we will use the terms MADM and MCDM to denote the same concept.
2 Classification of MCDM Methods
As it was stated in the previous section, there are many MADM methods available in the literature. Each
method has its own characteristics. There are many ways one can classify MADM methods. One way is to classify
them according to the type of the data they use. That is, we have deterministic, stochastic, or fuzzy MADM methods
(for an overview of fuzzy MADM methods see [Chen and Hwang, 1992]). However, there may be situations which
involve combinations of all the above (such as stochastic and fuzzy data) data types.
Another way of classifying MADM methods is according to the number of decision makers involved in the
decision process. Hence, we have single decision maker MADM methods and group decision making MADM (for
more information on the later class, the interested reader may want to check the journal of Group Decision Making).
In this paper we concentrate our attention on single decision maker deterministic MADM methods.
In [Chen and Hwang, 1992] deterministic -- single decision maker -- MADM methods were also classified
according to the type of information and the salient features of the information. The WSM, AHP, revised AHP,
WPM, and TOPSIS methods are the ones which are used mostly in practice today and are described in later sections.
Finally, it should be stated here that there are many other alternative ways for classifying MADM methods [Chen and
Hwang, 1992]. However, the previous ones are the most widely used approaches in the MADM literature.
3Some MCDM Application Areas
Some of the industrial engineering applications of MCDM include the use of decision analysis in integrated
manufacturing [Putrus, 1990], in the evaluation of technology investment decisions [Boucher and McStravic, 1991],
in flexible manufacturing systems [Wabalickis, 1988], layout design [Cambron and Evans, 1991], and also in other
engineering problems [Wang and Raz, 1991]. As an illustrative application consider the case in which one wishes
to upgrade the computer system of a computer integrated manufacturing (CIM) facility. There is a number of
different configurations available to choose from. The different systems are the alternatives. A decision should also
consider issues such as: cost, performance characteristics (i.e., CPU speed, memory capacity, RAM size, etc.),
availability of software, maintenance, expendability, etc. These may be some of the decision criteria for this problem.
In the above problem we are interested in determining the best alternative (i.e., computer system). In some other
situations, however, one may be interested in determining the relative importance of all the alternatives under
consideration. For instance, if one is interested in funding a set of competing projects (which now are the
alternatives), then the relative importance of these projects is required (so the budget can be distributed proportionally
to their relative importances).
Multi-criteria decision-making (MCDM) plays a critical role in many real life problems. It is not an
exaggeration to argue that almost any local or federal government, industry, or business activity involves, in one way
or the other, the evaluation of a set of alternatives in terms of a set of decision criteria. Very often these criteria are
conflicting with each other. Even more often the pertinent data are very expensive to collect.
4Multi-Criteria Decision Making Methods
4.1 Background Information
With the continuing proliferation of decision methods and their modifications, it is important to have an
understanding of their comparative value. Each of the methods uses numeric techniques to help decision makers
choose among a discrete set of alternative decisions. This is achieved on the basis of the impact of the alternatives
on certain criteria and thereby on the overall utility of the decision maker(s).
Despite the criticism that multi-dimensional methods have received, some of them are widely used. The
weighted sum model (or WSM) is the earliest and probably the most widely used method. The weighted product
model (or WPM) can be considered as a modification of the WSM, and has been proposed in order to overcome some
of its weaknesses. The analytic hierarchy process (or AHP), as proposed by Saaty [Saaty, 1980, 1983, 1990, and

4
1994], is a later development and it has recently become increasingly popular. Professors Belton and Gear [1983]
suggested a modification to the AHP that appears to be more powerful than the original approach. Some other widely
used methods are the ELECTRE [Benayoun, et al., 1966] and TOPSIS [Hwang and Yoon, 1981]. In the sub-section
that follows these methods are presented in detail.
4.2 Description of Some MCDM Methods
There are three steps in utilizing any decision-making technique involving numerical analysis of alternatives:
1) Determining the relevant criteria and alternatives.
2) Attaching numerical measures to the relative importance of the criteria and to the impacts
of the alternatives on these criteria.
3) Processing the numerical values to determine a ranking of each alternative.
This section is only concerned with the effectiveness of the four methods in performing step 3. The central decision
problem examined in this paper is described as follows. Given is a set of M alternatives: A1, A2, A3, ..., AM and a set
of N decision criteria C1, C2, C3, ..., CN and the data of a decision matrix as the one described in Figure 1. Then the
problem is to rank the alternatives in terms of their total preferences when all the decision criteria are considered
simultaneously.
4.2.1 The Weighted Sum Model
The weighted sum model (or WSM) is probably the most commonly used approach, especially in single
dimensional problems. If there are M alternatives and N criteria then, the best alternative is the one that satisfies (in
the maximization case) the following expression [Fishburn, 1967]:
(4-1)A(
WSM 'max
ij
N
j'1qij wj,for i'1,2,3, ..., M.
where: AWSM* is the WSM score of the best alternative, N is the number of decision criteria, aij is the actual value of
the i-th alternative in terms of the j-th criterion, and Wj is the weight of importance of the j-th criterion.
The assumption that governs this model is the additive utility assumption. That is, the total value of each
alternative is equal to the sum of products given as (4-1). In single-dimensional cases, in which all the units are the
same (e.g., dollars, feet, seconds), the WSM can be used without difficulty. Difficulty with this method emerges when
it is applied to multi-dimensional decision-making problems. Then, in combining different dimensions, and
consequently different units, the additive utility assumption is violated and the result is equivalent to "adding apples
and oranges".
Example 4-1:
Suppose that an MCDM problem involves four criteria, which are expressed in exactly the same unit, and three
alternatives. The relative weights of the four criteria were determined to be: W1 = 0.20, W2 = 0.15, W3 = 0.40, and
W4 = 0.25. The corresponding aij values are assumed to be as follows:
A'
25 20 15 30
10 30 20 30
30 10 30 10
.
Therefore, the data (i.e., decision matrix) for this MCDM problem are as follows:
Criteria
C1C2C3C4

5
Alt. ( 0.20 0.15 0.40 0.25)
_______________________________
A125 20 15 30
A210 30 20 30
A330 10 30 10
When formula (4-1) is applied on the previous data, the scores of the three alternatives are:
A1(WSM score) = 25×0.20 + 20×0.15 + 15×0.40 + 30×0.25 = 21.50.
Similarly, A2(WSM score) = 22.00,
and A3(WSM score) = 20.00.
Therefore, the best alternative (in the maximization case) is alternative A2 (because it has the highest WSM score;
22.00). Moreover, the following ranking is derived: A2 > A1 > A3 (where ">" stands for "better than"). €
4.2.2 The Weighted Product Model
The weighted product model (or WPM) is very similar to the WSM. The main difference is that instead of
addition in the model there is multiplication. Each alternative is compared with the others by multiplying a number
of ratios, one for each criterion. Each ratio is raised to the power equivalent to the relative weight of the
corresponding criterion. In general, in order to compare the alternatives AK and AL, the following product (Bridgman
[1922] and Miller and Starr [1969]) has to be calculated:
(4-2)R(AK/AL)'k
N
j'1(aKj /aLj )wj,
where: N is the number of criteria, aij is the actual value of the i-th alternative in terms of the j-th criterion, and Wj
is the weight of importance of the j-th criterion.
If the term R(AK / AL) is greater than to one, then alternative AK is more desirable than alternative AL (in the
maximization case). The best alternative is the one that is better than or at least equal to all the other alternatives.
The WPM is sometimes called dimensionless analysis because its structure eliminates any units of measure.
Thus, the WPM can be used in single- and multi-dimensional decision-making problems. An advantage of the method
is that instead of the actual values it can use relative ones. This is true because:
(4-3)
aKj
aLj
'
aKj /j
N
i'1aKi
aLj /j
N
i'1aLi
'a/
Kj
a/
Lj
.
A relative value a/Kj is calculated by using the formula: where the aKj's are the actual values.a/
Kj 'aKj /j
N
i'1aKi
Example 4-2:
Consider the problem presented in the previous example 4-1 (note that now the restriction to express all criteria in
terms of the same unit is not needed). When the WPM is applied, then the following values are derived:
R(A1/A2) = (25/10)0.20 × (20/30)0.15 × (15/20)0.40 × (30/30)0.25 = 1.007 > 1.
Similarly, R(A1/A3) = 1.067 > 1,
and R(A2/A3) = 1.059 > 1.
Therefore, the best alternative is A1, since it is superior to all the other alternatives. Moreover, the ranking of these
alternatives is as follows: A1 > A2 > A3. €

6
An alternative approach is one to use only products without ratios. That is, to use the following variant of
formula (4-2):
(4-4)P(AK)'k
N
j'1(aKj )wj,
Then, when the previous data are used, exactly the same ranking is derived.
4.2.3 The Analytic Hierarchy Process
The analytic hierarchy process (or AHP) ([Saaty, 1980, 1983, 1990, and 1994]) is based on decomposing
a complex MCDM problem into a system of hierarchies (more on these hierarchies can be found in [Saaty, 1980]).
The final step in the AHP deals with the structure of an M×N matrix (where M is the number of alternatives and N
is the number of criteria). This matrix is constructed by using the relative importances of the alternatives in terms
of each criterion. The vector (ai1, ai2, ai3, ..., aiN) for each i is the principal eigenvector of an N×N reciprocal matrix
which is determined by pairwise comparisons of the impact of the M alternatives on the i-th criterion (more on this,
and some other related techniques, is presented in section 6).
Some evidence is presented in [Saaty, 1980] which supports the technique for eliciting numerical evaluations
of qualitative phenomena from experts and decision makers. However, we are not concerned here with the possible
advantages and disadvantages of the use of pairwise comparisons and the eigenvector method for determining values
for the aij's. Instead, we examine the method used in AHP to process the aij values after they have been determined.
The entry aij, in the M×N matrix, represents the relative value of the alternative Ai when it is considered in terms of
criterion Cj. In the original AHP the sum is equal to one.
j
N
i'1aij
According to AHP the best alternative (in the maximization case) is indicated by the following relationship
(4-5):
(4-5)A(
AHP 'max
ij
N
j'1qij wj,for i'1,2,3, ..., M.
The similarity between the WSM and the AHP is evident. The AHP uses relative values instead of actual ones. Thus,
it can be used in single- or multi-dimensional decision making problems.
Example 4-3:
Again, consider the data used in the previous two examples (note that as in the WPM case the restriction to express
all criteria in terms of the same unit is not needed). The AHP uses a series of pairwise comparisons (more on this can
be found in section 6) to determine the relative performance of each alternative in terms of each one of the decision
criteria. In other words, instead of the absolute data, the AHP would use the following relative data:
Criteria
C1C2C3C4
Alt. ( 0.20 0.15 0.40 0.25)
_______________________________
A125/65 20/55 15/65 30/65
A210/65 30/55 20/65 30/65
A330/65 5/55 30/65 5/65
That is, the columns in the decision matrix have been normalized to add up to 1. When formula (4-5) is applied on
the previous data, the following scores are derived:
A1(AHP score) = (25/65)×0.20 + (20/55)×0.15 + (15/65)×0.40 + (30/65)×0.25 = 0.34.

7
Similarly, A2(AHP score) = 0.35,
and A3(AHP score) = 0.31.
Therefore, the best alternative (in the maximization case) is alternative A2 (because it has the highest AHP score;
0.35). Moreover, the following ranking is derived: A2 > A1 > A3. €
4.2.4 The Revised Analytic Hierarchy Process
Belton and Gear [1983] proposed a revised version of the AHP model. They demonstrated that an
inconsistency can occur when the AHP is used. They presented a numerical example which deals with three criteria
and three alternatives. In that example the indication of the best alternative changes when an identical alternative to
one of the nonoptimal alternatives is introduced now creating four alternatives. According to the authors the root for
that inconsistency is the fact that the relative values for each criterion sum up to one. Instead of having the relative
values of the alternatives A1, A2, A3, ..., AM sum up to one, they propose to divide each relative value by the maximum
value of the relative values. In particular, they elaborated on the following example.
Example 4-4 (from [Belton and Gear, 1983], p. 228):
Suppose that the actual data of an MCDM problem with three alternatives and three criteria are as follows:
Criteria
C1C2C3
Alt. ( 1/3 1/3 1/3)
________________________
A1198
A2919
A3111
Observe that in real life problems the decision maker may never know the previous real data. Instead, he/she can use
the method of pairwise comparisons (as described in section 6) to derive the relative data. When the AHP is applied
on the previous data, the following decision matrix with the relative data is derived:

8
Criteria
C1C2C3
Alt. ( 1/3 1/3 1/3)
________________________
A11/11 9/11 8/18
A29/11 1/11 9/18
A31/11 1/11 1/18
Therefore, it can be easily verified that the vector with the final AHP scores, is: (0.45, 0.47, 0.08). That is, the three
alternatives are ranked as follows: A2 > A1 > A3.
Next, we introduce a new alternative, say A4, which is an identical copy of the existing alternative A2 (i.e.,
A2 . A4). Furthermore, it is also assumed that the relative weights of importance of the three criteria remain the same
(i.e., 1/3, 1/3, 1/3). When the new alternative A4 is considered, it can be easily verified that the new decision matrix
is as follows:
Criteria
C1C2C3
Alt. ( 1/3 1/3 1/3)
________________________
A11/20 9/12 8/27
A29/20 1/12 9/27
A31/20 1/12 1/27
A49/20 1/12 9/27
Similarly as above, it can be verified that the vector with the final AHP scores, is: (0.37, 0.29, 0.06, 0.29). That is,
the four alternatives are ranked as follows: A1 > A2 . A4 > A3. The authors claim that this result is in logical
contradiction with the previous result (in which A2 > A1).
When the revised AHP is applied on the last data, the following decision matrix is derived:
Criteria
C1C2C3
Alt. ( 1/3 1/3 1/3)
________________________
A11/9 18/9
A211/9 1
A31/9 1/9 1/9
A411/9 1
The vector with the final scores, is: (2/3, 19/27, 1/9, 19/27). That is, the four alternatives are ranked as follows: A2
. A4 > A1 > A3. The last ranking is, obviously, the desired one. €
The revised AHP was sharply criticized by Saaty [1990]. He claimed that identical alternatives should not
be considered in the decision process. However [Triantaphyllou and Mann, 1989] have demonstrated that similar
logical contradictions are possible with the original AHP, as well as with the revised AHP, when non-identical
alternatives are introduced.
4.2.5 The ELECTRE Method
The ELECTRE (for Elimination and Choice Translating Reality; English translation from the French
original) method was first introduced in [Benayoun, et al., 1966]. The basic concept of the ELECTRE method is to

9
deal with "outranking relations" by using pairwise comparisons among alternatives under each one of the criteria
separately. The outranking relationship of Ai v Aj describes that even when the i-th alternative does not dominate the
j-th alternative quantitatively, then the decision maker may still take the risk of regarding Ai as almost surely better
than Aj [Roy, 1973]. Alternatives are said to be dominated, if there is another alternative which excels them in one
or more attributes and equals in the remaining attributes.
The ELECTRE method begins with pairwise comparisons of alternatives under each criterion. Using physical
or monetary values gi(Aj) and gi(Ak) of the alternatives Aj and Ak respectively, and introducing threshold levels for the
difference gi(Aj)-gi(Ak), the decision maker may declare that he/she is indifferent between the alternatives under
consideration, that he/she has a weak or a strict preference for one of the two, or that he/she is unable to express any
of these preference relations. Therefore, the set of binary relations of alternatives, the so-called outranking relations,
may be complete or incomplete. Next, the decision maker is requested to assign weights or importance factors to the
criteria in order to express their relative importance.
Through a series of consecutive assessments of the outranking relations of the alternatives, ELECTRE elicits
the so-called concordance index, defined as the amount of evidence to support the conclusion that Aj outranks, or
dominates, Ak, as well as the discordance, the counter-part of concordance index.
Finally, the ELECTRE method yields a whole system of binary outranking relations between the alternatives.
Because the system is not necessarily complete, the ELECTRE method is sometimes unable to identify the preferred
alternative. It only produces a core of leading alternatives. This method has a clearer view of alternatives by
eliminating less favorable ones, especially convenient while encountering few criteria with large number of alternatives
in a decision making problem [Lootsma, 1990]. The organization of the ELECTRE method is best illustrated in the
following steps [Benayoun, et al., 1966]:
Step 1. Normalizing the Decision Matrix
This procedure transforms various units in the decision matrix into dimensionless comparable units by using
the following equation:
xij 'aij
j
M
i'1a2
ij
.
Therefore, the normalized matrix X is defined as follows:
X'
x11 x12 x13 ... x1N
x21 x22 x23 ... x2N
. .
. .
. .
xM1 xM2 xM3 ... xMN
,
where M is the number of alternatives and N is the number of criteria, and xij is the new and dimensionless preference
measure of the i-th alternative in terms of the j-th criterion.
Step 2. Weighting the Normalized Decision Matrix
The column of the X matrix is then multiplied by its associated weights which were assigned to the criteria
by the decision maker. Therefore, the weighted matrix, denoted as Y, is:
Y = XW,
where:

10
Y
'
y11 y12 y13 ... y1N
y21 y22 y23 ... y2N
. .
. .
. .
yM1 yM2 yM3 ... yMN
'
w1x11 w2x12 w3x13 ... wNx1N
w1x21 w2x22 w3x23 ... wNx2N
. .
. .
. .
w1xM1 w2xM2 w3xM3 ... wNxMN
,
and:
W '
w10 0 ... 0
0w20... 0
. .
. .
. .
0 0 0 ... wM
,and also j
N
i'1wi'1.
Step 3. Determine the Concordance and Discordance Sets
The concordance set Ckl of two alternatives Ak and AR, where M$k, l$1, is defined as the set of all criteria for
which Ak is preferred to Al. That is, the following is true:
Ckl = {j, such that: ykj $ ylj}, for j = 1, 2, 3, ..., N.
The complementary subset is called the discordance set and it is described as follows:
Dkl = {j, such that: ykj < ylj}, for j = 1, 2, 3, ..., N.
Step 4. Construct the Concordance and Discordance Matrices
The relative value of the elements in the concordance matrix C is calculated by means of the concordance
index. The concordance index ckl is the sum of the weights associated with the criteria contained in the concordance
set. That is, the following is true:
Ckl'
j
j,Ckl
wj,for j'1,2,3, ..., N.
The concordance index indicates the relative importance of alternative Ak with respect to alternative Al.
Apparently, 0 # ckl # 1. Therefore, the concordance matrix C is defined as follows:
C'
&c12 c13 ... c1M
c21 &c23 ... c2M
. .
. .
. .
cM1 cM2 cM3 ... &
.
It should be noted here that the entries of matrix C are not defined when k = l.

11
The discordance matrix D expresses the degree that a certain alternative Ak is worse than a competing
alternative Al. The elements dk l of the discordance matrix are defined as follows:
(4-6)
dkl'
max
j,Dkl
*ykj&ylj*
max
j*ykj&ylj*.
The discordance matrix is defined as follows:
D'
&d12 d13 ... d1M
d21 &d23 ... d2M
. .
. .
. .
dM1dM2dM3... &
.
As before, the entries of matrix D are not defined when k = l.
It should also be noted here that the previous two M × M matrices are not symmetric.
Step 5. Determine the Concordance and Discordance Dominance Matrices
The concordance dominance matrix is constructed by means of a threshold value for the concordance index.
For example, Ak will only have a chance to dominate Al if its corresponding concordance index ckl exceeds at least a
certain threshold value c. That is, the following is true:
ckl $ c,
The threshold value c can be determined as the average concordance index. That is, the following relation
is true:
(4-7)c'1
M(M&1)×j
M
k'1
and k…l
j
M
l'1
and l…k
ckl.
Based on the threshold value, the concordance dominance matrix F is determined as follows:
fkl = 1, if ckl $ c,
fkl = 0, if ckl < c.
Similarly, the discordance dominance matrix G is defined by using a threshold value d, where d is defined
as follows:
(4-8)d'1
M(M&1)j
M
k'1
and k…l
j
M
l'1
and l…k
dkl ,
and: gkl = 1, if dkl $ d,
gkl = 0, if dkl < d.
Step 6. Determine the Aggregate Dominance Matrix
The elements of the aggregate dominance matrix E are defined as follows:
ekl = fkl × gkl. (4-9)
Step 7. Eliminate the Less Favorable Alternatives
From the aggregate dominance matrix, we could get a partial-preference ordering of the alternatives. If ekl

12
= 1, then this means that Ak is preferred to Al by using both concordance and discordance criteria.
If any column of the aggregate dominance matrix has at least one element equal to 1, this column is
"ELECTREally" dominated by the corresponding row. Therefore, we simply eliminate any column(s) which have
an element equal to 1. Then, the best alternative is the one which dominates all other alternatives in this manner.
4.2.6 The TOPSIS Method
TOPSIS (the Technique for Order Preference by Similarity to Ideal Solution) was developed by Hwang and
Yoon [1981] as an alternative to the ELECTRE method. The basic concept of this method is that the selected
alternative should have the shortest distance from the ideal solution and the farthest distance from the negative-ideal
solution in a geometrical sense.
TOPSIS assumes that each attribute has a tendency of monotonically increasing or decreasing utility.
Therefore, it is easy to locate the ideal and negative-ideal solutions. The Euclidean distance approach is used to
evaluate the relative closeness of alternatives to the ideal solution. Thus, the preference order of alternatives is yielded
through comparing these relative distances.
The TOPSIS method evaluates the following decision matrix which refers to M alternatives which are
evaluated in terms of N criteria:
D'
x11 x12 x13 ... x1N
x21 x22 x23 ... x2N
. .
. .
. .
xM1xM2xM3... xMN
.
where xij denotes the performance measure of the i-th alternative in terms of the j-th criterion. For a clear view of
this method, the TOPSIS method is presented next as a series of successive steps.
Step 1. Construct the Normalized Decision Matrix
This process tries to convert the various attribute dimensions into nondimensional attributes similarly as with
the ELECTRE method. An element rij of the normalized decision matrix R can be calculated as follows:
(4-10)rij'xij
j
M
i'1x2
ij
.
Step 2. Construct the Weighted Normalized Decision Matrix
A set of weights W = (w1, w2, w3, ..., wN), (where: Ewi = 1) defined by the decision maker is accommodated
to the decision matrix to generate the weighted normalized matrix V as follows:

13
V'
w1r11 w2r12 w3r13 ... wNr1N
w1r21 w2r22 w3r23 ... wNr2N
. .
. .
. .
w1rM1w2rM2w3rM3... wNrMN
.
Step 3. Determine the Ideal and the Negative-ideal Solutions
The ideal A* and the negative-ideal A- solutions are defined as follows:
A* = { (max vij | j 0 J), (min vij | j 0 J/)| i = 1,2,3, ..., M} =
i i
= { v1*, v2*, ..., vN* }. (4-11)
A- = { (min vij | j 0 J), (max vij | j 0 J/)| i = 1,2,3, ..., M} =
i i
= { v1-, v2-, ..., vN- }. (4-12)
where: J = {j = 1,2,3, ..., N | j associated with benefit criteria},
J/= {j = 1,2,3, ..., N | j associated with cost criteria}.
For the benefit criteria, the decision maker wants to have a maximum value among the alternatives. For the
cost criteria, the decision maker wants to have a minimum value among alternatives. Obviously, A* indicates the most
preferable alternative or ideal solution. Similarly, A- indicates the least preferable alternative or negative-ideal
solution.
Step 4. Calculate the Separation Measure
The N-dimensional Euclidean distance method is next applied to measure the separation distances of each
alternative to the ideal solution and negative-ideal solution.
Si* = ( E(vij - vj*)2 )1/2, i = 1,2,3, ..., M, (4-13)
where Si* is the separation (in the Euclidean sense) of each alternative from the ideal solution.
Si- = ( E(vij - vj-)2 )1/2, i = 1,2,3, ..., M, (4-14)
where Si- is the separation (in the Euclidean sense) of each alternative from the negative-ideal solution.
Step 5. Calculate the Relative Closeness to the Ideal Solution
The relative closeness of an alternative Ai with respect to the ideal solution A* is defined as follows:
Ci* = Si- / (Si* + Si-), 0 # Ci* # 1, i = 1,2,3, ..., M. (4-15)
Apparently, Ci* = 1, if Ai = A*, and Ci- = 0, if Ai = A-.
Step 6. Rank the Preference Order
The best satisfied alternative can now be decided according to preference rank order of Ci*. Therefore, the
best alternative is the one that has the shortest distance to the ideal solution. The relationship of alternatives reveals
that any alternative which has the shortest distance to the ideal solution is guaranteed to have the longest distance to
the negative-ideal solution.

14
5Sensitivity Analysis of MCDM Methods
As it was stated earlier, often data in MCDM problems are difficult to be quantified or are easily changeable.
Thus, often the decision maker needs to first estimate the data with some accuracy, and later estimate more critical
data with higher accuracy. In this way, the decision maker can rank the alternatives with high confidence and not
overestimate non critical data. The above considerations lead to the need of performing a sensitivity analysis on a
MCDM problem.
The objective of a typical sensitivity analysis of an MCDM problem is to find out when the input data (i.e.,
the aij and wj values) are changed into new values, how the ranking of the alternatives will change. In the literature
there has been some discussion on how to perform a sensitivity analysis in MCDM. Insua [1990] demonstrated that
decision making problems may be remarkably sensitive to some reasonable variations in the parameters of the
problems. His conclusion justified the necessity of sensitivity analysis in MCDM. Evans [1984], explored a linear
programming -like sensitivity analysis in the decision making problems consisting of a single set of decision
alternatives and states of nature. In his method, the optimal alternative is represented as a bounded convex polyhedron
in the probability state space. Using the geometric characteristics of the optimal regions, he defined the confidence
sphere of the optimal alternatives. The larger the confidence sphere, the less sensitive the optimal alternative will be
to the state probabilities.
Masuda [1990] studied some sensitivity issues of the AHP method. In his paper, he focused on how changes
on entire columns of the decision making matrix may affect the values of the composite priorities of the alternatives.
In his method, he generated the sensitivity coefficient of the final priority vector of the alternatives to each of the
column vectors in the decision matrix. A large coefficient means that the values of the final priorities of the
alternatives will change more greatly if there is a slight change in the corresponding column vector of the decision
matrix. However, that does not guarantee that a ranking reversal among the alternatives due to the change of the
column vectors is sure to happen. Finally, Triantaphyllou and Sanchez [1997] proposed a unified approach for a
sensitivity analysis for three major MCDM methods. These methods are: the WSM, the WPM and the AHP (original
and revised). Their approach examines the effect of the changes of a single parameter (i.e., an aij or wj value) on the
final rankings of the alternatives. That approach can be seen as an extension of Masuda's method with its focus on
the ranking reversal of the alternatives which is more useful in practical applications. Also in that paper, the authors
have done some empirical studies to determine the most critical criterion (wj) as well as the most critical performance
value (aij) in a general MCDM problem.
Sensitivity analysis is a fundamental concept for the effective use and implementation of quantitative decision
models [Dantzig, 1963]. It is just too important to be ignored in the application of an MCDM method to a real life
problem.
6Data Estimation for MCDM Problems
One of the most crucial steps in many decision making methods is the accurate estimation of the pertinent
data. This problem is particularly crucial in methods which need to elicit qualitative information from the decision
maker. Very often qualitative data cannot be known in terms of absolute values. For instance, what is the worth of
the i-th alternative in terms of a political impact criterion? Although information about questions like the previous
one may be vital in making the correct decision, it is very difficult, if not impossible, to quantify it correctly.
Therefore, many decision making methods attempt to determine the relative importance, or weight, of the alternatives
in terms of each criterion involved in a given decision making problem.
An approach based on pairwise comparisons which was proposed by Saaty (see, for instance, [Saaty, 1980
and 1983]) has long attracted the interest of many researchers. Pairwise comparisons are used to determine the
relative importance of each alternative in terms of each criterion. In this approach a decision maker has to express
his/her opinion about the value of one single pairwise comparison at a time. Usually, the decision maker has to choose
his/her answer among 10-17 discrete choices. Each choice is a linguistic phrase. Some examples of such linguistic
phrases are: "A is more important than B", or "A is of the same importance as B", or "A is a little more important
than B", and so on. The focus here is not on the wording of these linguistic statements, but, instead, on the numerical

15
values which should be associated with such statements.
The main problem with the pairwise comparisons is how to quantify the linguistic choices selected by the
decision maker during their evaluation. All the methods which use the pairwise comparisons approach eventually
express the qualitative answers of a decision maker into some numbers which, most of the time, are ratios of integers.
A case in which pairwise comparisons are expressed as differences (instead of ratios) was used to define similarity
relations and is described by Triantaphyllou in [1993]. The next section examines the issue of quantifying pairwise
comparisons. Since pairwise comparisons are the keystone of these decision making processes, correctly quantifying
them is the most crucial step in multi-criteria decision making methods which use qualitative data.
Many of the previous problems are not bound only to the AHP. They are present with any method which has
to elicit information from pairwise comparisons. These problems can be divided into the following three categories:
(i) How to quantify the pairwise comparisons.
(ii) How to process the resulted reciprocal matrices.
and (iii) How to process the decision matrices.
Next we consider some of the main ideas related with pairwise comparisons. In the sub-sections that follow, we
consider each one of the previous challenges, and discuss some remedies which have been proposed.
6.1 Problem #1: On the Quantification of Pairwise Comparisons
Pairwise comparisons are quantified by using a scale. Such a scale is an one-to-one mapping between the
set of discrete linguistic choices available to the decision maker and a discrete set of numbers which represent the
importance, or weight, of the previous linguistic choices. There are two major approaches in developing such scales.
The first approach is based on the linear scale proposed by Saaty [1980] as part of the AHP. The second approach
was proposed by Lootsma in [1988 and 1990] and in [Lootsma, et al., 1990] and determines exponential scales.
Both approaches depart from some psychological theories and develop the numbers to be used based on these
psychological theories.
6.1.1 Scales Defined on the Interval [9, 1/9]
In 1846 Weber stated his law regarding a stimulus of measurable magnitude. According to his law a change
in sensation is noticed if the stimulus is increased by a constant percentage of the stimulus itself [Saaty, 1980]. That
is, people are unable to make choices from an infinite set. For example, people cannot distinguish between two very
close values of importance, say 3.00 and 3.02. Psychological experiments have also shown that individuals cannot
simultaneously compare more than seven objects (plus or minus two) [Miller, 1956]. This is the main reasoning used
by Saaty to establish 9 as the upper limit of his scale, 1 as the lower limit and a unit difference between successive
scale values.
The values of the pairwise comparisons are determined according to the scale introduced by Saaty [1980].
According to this scale (which we call Scale1), the available values for the pairwise comparisons are members of the
set: {9, 8, 7, 6, 5, 4, 3, 2, 1, 1/2, 1/3, 1/4, 1/5, 1/6, 1/7, 1/8, 1/9}. The above numbers illustrate that the values
for the pairwise comparisons can be grouped into the two intervals [9, 1] and [1, 1/9]. As it was stated above, the
values in the interval [9, 1] are evenly distributed, while the values in the interval [1, 1/9] are skewed to the right
end of this interval.
There is no good reason why for a scale defined on the interval [9, 1/9] the values on the sub-interval [9, 1]
should be evenly distributed. An alternative scale could have the values evenly distributed in the interval [1, 1/9],
while the values in the interval [9, 1] could be simply the reciprocals of the values in the interval [1, 1/9]. This
consideration leads to the scale (which we call Scale2) with the following values: {9, 9/2, 9/3, 9/4, 9/5, 9/6, 9/7, 9/8,
1, 8/9, 7/9, 6/9, 5/9, 4/9, 3/9, 2/9, 1/9}. This scale was originally presented by Ma and Zheng in [1991]. In the
second scale each successive value on the interval [1, 1/9] is (1 - 1/9) / 8 = 1/9 units apart. In this way, the values
in the interval [1, 1/9] are evenly distributed, while the values in [9, 1] are simply the reciprocals of the values in
[1, 1/9]. It should be stated here that the notion of having a scale with a group of values evenly distributed, is
followed in order to be in agreement with the same characteristic of the original Saaty scale. As it will be seen in the

16
next section, other scales can be defined without having evenly distributed values.
Besides the second scale, many other scales can be generated. One way to generate new scales is to consider
weighted versions between the previous two scales. That is, for the interval [1, 1/9] the values can be calculated
using the formula:
NewValue = Value(Scale1) + (Value(Scale2) - Value(Scale1) )*("/100).
In the previous formula the values of " can range from 0 to 100. Then, the values in the interval [9, 1] are the
reciprocals of the above values. For " = 0 Scale1 is derived, while for " = 100 Scale2 is derived.
6.1.2 Exponential Scales
A class of exponential scales has been introduced by Lootsma [1988 and 1990] and [Lootsma, et al., 1990].
The development of these scales is based on an observation in psychology about stimulus perception (denoted as ei).
According to that observation, due to Roberts [1979], the difference en+1 - en must be greater than or equal to the
smallest perceptible difference, which is proportional to en. As a result of Robert's observation the numerical
equivalents of these linguistics choices need to satisfy the following relations:
en+1 - en = , en, (where , > 0) or:
en+1 = (1 + ,) en = (1 + ,)2 en-1 = ...
... = (1 + ,)n+1 eo, (where: eo = 1) or: en = e( × n.
In the previous expressions the parameter ( is unknown (or, equivalently, , is unknown), since ( = ln(1 + ,), and
e is the basis of the natural logarithms (please note that ei is just the notation of a variable).
Another difference between exponential scales and the Saaty scale is the number of categories allowed by the
exponential scales. There are only four major linguistically distinct categories, plus three so-called threshold
categories between them. The threshold categories can be used if the decision maker hesitates between the main
categories. For a more detailed documentation on psychophysics we refer the reader to Marks [1974], Michon, et
al., [1769], Roberts [1979], Zwicker [1982], and Stevens and Hallowell Davis [1983]. The reader will find that the
sensory systems for the perception of tastes, smells, and touches follow the power law with exponents near 1.
6.1.3 Evaluating Different Scales
In order for different scales to be evaluated, two evaluative criteria were developed by Triantaphyllou, et al.,
in [1994]. Furthermore, a special class of pairwise matrices was also developed. These special matrices were then
used in conjunction with the two evaluative criteria in order to investigate some stability properties of different scales.
The most important observation of that study is that the results illustrate very clearly that there is no single
scale which is the best scale for all cases. Similarly, the results illustrate that there is no single scale which is the
worst scale for all cases. However, according to these computational results, the best (or worst) scale can be
determined only if the number of the alternatives is known and the relative importance of the weights of the two
evaluative criteria has been assessed.
6.2 Problem #2: Processing Reciprocal Matrices with Pairwise Comparisons
At this point it is assumed that the decision maker has determined the values of all the pairwise comparisons.
That is, available are the values aij (for i,j = 1,2,3,...,N), where aij represents the relative performance of alternative
Ai when it is compared with alternative Aj in terms of a single criterion. Given these values, the decision maker needs
to determine the relative weights, say Wi (i = 1,2,3,...,N), of the alternatives in terms of the single criterion. Saaty
[1980] has proposed a method which asserts that the desired weights are the elements of the right principal eigenvector
of the matrix with the pairwise comparisons. This method has been evaluated under a continuity assumption by
Triantaphyllou and Mann in [1990]. Moreover, other authors have proposed alternative approaches.
For instance, Chu, et al., in [1979] observed that, given the data aij, the values Wi to be estimated are desired
to have the following property:
aij . Wi/Wj.
This is reasonable, since aij is meant to be the estimate of the ratio Wi/Wj. Then, in order to get the estimates for the
Wi given the data aij, they proposed the following constrained optimization problem:

17
minimize S'j
N
i'1j
N
j'1(aij Wj&Wi)2
subject to:j
N
i'1Wi'1,
and Wi>0,for any i'1,2,3,...,N.
They also gave an alternative expression S1 that is more difficult to solve numerically. Specifically, they proposed:
minimize S1'j
N
i'1j
N
j'1(aij &Wj/Wi)2.
In Federov, et al., [1982], a variation of the previous least-squares formulation was proposed. For the case
of only one decision maker the authors recommended to use the following models:
log aij 'log Wi&log Wj%Q1(Wi,Wj)gij ,
and aij 'Wi/Wj%Q2(Wi,Wj)gij ,
where Wi and Wj are the true (and thus unknown) weights; and are given positive functionsQ1(X,Z)Q2(X,Z)
(where X,Z > 0). The random errors gij are assumed to be independent with zero mean and unit variance. However,
they fail to give a way of selecting the appropriate two previous positive functions.
In the following paragraphs we present the main idea which was originally described in Triantaphyllou, et
al., [1990]. In that treatment the assumption of the human rationality is made. According to that assumption the
decision maker is a rational person. Rational persons are defined here as individuals who try to minimize their regret
[Simon, 1961], to minimize losses, or to maximize profit [Write and Tate, 1973]. In the present context, minimization
of regret of losses, or maximization of profit could be interpreted as the effort of the decision maker to minimize the
errors involved in the pairwise comparisons.
As it was stated in the previous paragraphs, in the inconsistent case, the entry aij of matrix A is an estimate
of the real ratio Wi/Wj. Since it is an estimate, the following is true:
aij = (Wi/Wj)dij, for i, j = 1, 2, 3, ..., N. (5-1)
In the previous relation, dij denotes the deviation of aij from being a perfectly accurate judgment. Obviously, if dij =
1, the aij value was perfectly estimated. From the previous formulation, we conclude that the errors involved in these
pairwise comparisons are given by:
gij = dij - 1,
or by using (5-1) above,
gij = aij (Wj/Wi) - 1. (5-2)
When the set of alternatives (or criteria) contains N elements, then N(N-1)/2 total pairwise comparisons need
to be estimated. The corresponding N(N-1)/2 errors are (after using relations (5-1) and (5-2)):
gij = aij (Wj/Wi) - 1, for i, j = 1, 2, 3, ..., N, and j > 1. (5-3)
Since the Wi's are relative weights which (in most cases) have to add up to 1, the following relation should also be
satisfied:
(5-4)
j
N
i'1Wi'1.00,and Wi>0,for i'1,2,3, ..., N.
When the data (e.g., the pairwise comparisons) are perfectly consistent, then relations (5-3) and (5-4) can be written
as follows:
B × W = b. (5-5)
The vector b has zero entries everywhere, except that the last entry is equal to 1; the matrix B has the following
structure (blank entries represent zeros):

18
B'
1 2 3 4 5 6 7 ...N&1N
&1a1,21
&1a1,32
&1a1,43
. . .
. . .
. . .
. . .
.a1,N&1.
&1a1,NN&1
&1a2,31
&1a2,42
&1a2,53
. . .
. . .
. . .
.a2,N&1.
&1a2,NN&2
. .
. .
. .
&1aN&1,N1
1 1 1 1 1 1 1 ... 1 1
The error minimization issue is interpreted in many cases (for instance, in regression analysis and in the linear
least-squares problem) as the minimization of the sum of squares of the residual vector r = b - B×W [Stewart, 1973].
In terms of the previous formulation (5-5), this means that, in a real-life situation (i.e., when errors are not zero any
more), the real intention of the decision maker is to minimize the following expression
(5-6)f2(x)'2b&BW22
2,
which, apparently, expresses a typical linear least-squares problem.
In [Triantaphyllou, et al., 1990] all the previous methods were tested in terms of an example originally
presented by Saaty in [1977] and also later used by other authors (e.g., Chu, et al., [1979] and [Federov, et al.,
1982]). In that test it was found that the proposed human rationality approach results in much smaller residuals.
Moreover, in the same study it was found, on thousands of randomly generated test problems, that the eigenvalue
approach may result in considerably higher residual values than the proposed least-squares approach which uses the
previous human rationality assumption.
6.3 Problem #3: Processing the Decision Matrices
In Triantaphyllou and Mann [1989] the AHP, revised AHP, weighted sum model (WSM) [Fishburn, 1967],

19
and the weighted product model (WPM) [Miller and Starr, 1969] were examined in terms of two evaluative criteria.
That study focused on the last step of any MCDM method which involves the processing of the final decision matrix.
That is, given the weights of relative performance of the decision criteria, and the performance of the alternatives in
terms of each one of the decision criteria, then determine what is the ranking (or relative priorities) of the alternatives.
As it was shown in Triantaphyllou and Mann [1989], however, these methods can give different answers to
the same problem. Since the truly best alternative is the same regardless of the method chosen, an estimation of the
accuracy of each method is highly desirable. The most difficult problem that arises here is how one can evaluate a
multi-dimensional decision making method when the true best alternative is not known. Two evaluative criteria were
introduced in [Triantaphyllou and Mann, 1989] for the above purpose.
The first evaluative criterion has to do with the premise that a method which is accurate in
multi-dimensional problems should also be accurate in single-dimensional problems. There is no reason for an
accurate multi-dimensional method to fail in giving accurate results in single-dimensional problems, since
single-dimensional problems are special cases of multi-dimensional ones. Because the first method, the WSM,
gives the most acceptable results for the majority of single-dimensional problems, the result of the WSM was used
as the standard for evaluating the other three methods in this context.
The second evaluative criterion considers the premise that a desirable method should not change the
indication of the best alternative when an alternative (not the best) is replaced by another worse alternative (given
that the importance of each criterion remains unchanged).
In Triantaphyllou and Mann [1989] the previous two evaluative criteria were applied on random test problems
with the numbers of decision criteria and alternatives taking the values 3, 5, 7, ..., 21. In those experiments it was
found that all the previous four MCDM methods were inaccurate. Furthermore, these results were used to form a
decision problem in which the four methods themselves were the alternatives. The decision criteria were derived by
considering the two evaluative criteria. To one's greatest surprise, one method would recommend another, rival
method, as being the best method! However, the final results seemed to suggest that the revised AHP was the most
efficient MCDM method of the ones examined. This was reported in Triantaphyllou and Mann [1989] as a decision
making paradox. Finally, a different approach of evaluating the performance of the AHP and the revised AHP is
described by Triantaphyllou and Mann in [1995]. In that treatment it was found that these two methods may yield
dramatically inaccurate results (more than 80% of the time on all the problems).
7 Concluding Remarks
There is no doubt that many real life problems can be dealt with as MCDM problems. Although the
mathematical procedures for processing the pertinent data are rather simple, the real challenge is in quantifying these
data. This is a non trivial problem. In matter of fact, it is not even a well defined problem. For these reasons, the
literature has an abundance of competing methods. The main problem is that often nobody can know what is the
optimal alternative. Operations research provides a systematic framework for dealing with such problems.
This paper discussed some of the challenges facing practitioners and theoreticians in some of the
methodological problems in MCDM theory. Although it is doubtful that the "perfect" MCDM approach will ever
be found, it is always a prudent idea for the user to be aware of the main controversies in the field. Although the
search for finding the best MCDM method may never end, research in this area of decision making is still critical and
valuable. REFERENCES
1. Belton, V., and T. Gear, "On a short-coming of Saaty's method of analytic hierarchies," Omega, 228-230,
1983.
2. Benayoun, R., B. Roy, and N. Sussman, "Manual de reference du programme electre, Note de Synthese et
Formation," No. 25, Direction Scientifique SEMA, Paris, Franch, 1966.
3. Boucher, T.O. and E.L. McStravic, "Multi-attribute Evaluation Within a Present Value Framework and its
Relation to the Analytic Hierarchy Process," The Engineering Economist, 37, 55-71, 1991.

20
4. Bridgman, P.W., Dimensional Analysis, Yale University Press, New Haven, CN, 1922.
5. Cambron, K.E. and G.W. Evans, "Layout Design Using the Analytic Hierarchy Process," Computers &
Industrial Engineering, 20, 221-229, 1991
6. Chen, S.J. and C.L. Hwang, Fuzzy Multiple Attribute Decision Making: Methods and Applications,
Lecture Notes in Economics and Mathematical Systems, No. 375, Sringer-Verlag, Berlin, Germany, 1992.
7. Chu, A.T.W., R.E. Kalaba, and K. Spingarn, "A comparison of two methods for determining the weights
of belonging to fuzzy sets," Journal of Optimization Theory and Applications, 27, 531-538, 1979.
9. Dantzig, G.B., Linear programming and extensions. Princeton University Press, Princeton, NJ, 1963.
10. Evans, J.R. "Sensitivity analysis in decision theory," Decision Sciences, 1(15), 239-247, 1984.
11. Federov, V.V., V.B. Kuz'min, and A.I. Vereskov, "Membership degrees determination from Saaty matrix
totalities," Institute for System Studies, Moscow, USSR. Paper appeared in: Approximate Reasoning in
Decision Analysis, M.M. Gupta, and E. Sanchez (editors), North-Holland Publishing Company, 23-30,
1982.
12. Fishburn, P.C., Additive Utilities with Incomplete Product Set: Applications to Priorities and Assignments,
Operations Research Society of America (ORSA) Publication, Baltimore, MD, 1967.
13. Hwang C.L. and K. Yoon, Multiple Attribute Decision Making: Methods and Applications, Springer-Verlag,
New York, NY, 1981.
14. Kuhn, H.W. and A.W. Tucker, "Nonlinear Programming," Proc. 2nd Berkeley Symp. Math. Stat. Prob.,
481-492, 1951.
15. Lootsma, F.A., "Numerical scaling of human judgment in pairwise-comparison methods for fuzzy
multi-criteria decision analysis," Mathematical Models for Decision Support. NATO ASI Series F,
Computer and System Sciences, Springer-Verlag, Berlin, Germany, 48, 57-88, 1988.
16. Lootsma, F.A., T.C.A. Mensch, and F.A. Vos, "Multi-criteria analysis and budget reallocation in long-term
research planning," European Journal of Operational Research, 47, 293-305, 1990.
17. Lootsma, F.A., "The French and the American school in multi- criteria decision analysis," Recherche
Operationnele / Operations Research, 24(3), 263-285, 1990.
18. Ma, D. and X. Zheng, "9/9-9/1 scale method of the AHP," Proceedings of the 2nd International Symposium
on the AHP, Vol. 1, Pittsburgh, PA, 197-202, 1991.
19. Marks, L.E., Sensory Processes, The New Psychophysics, Academic Press, New York, NY, 1974.
20. Masuda, T. "Hierarchical sensitivity analysis of the priorities used in the Analytic Hierarchy Process,"
Systems Science, 21(2), 415-427, 1990.
21. Michon, J.A., E.G.J. Eijkman, and L.F.W. de Klerk, Handboek der Psychonomie, (in Dutch), Van Loghum
Slaterus, Deventer, The Netherlands, 1976.
22. Miller, D.W., and M.K. Starr, Executive Decisions and Operations Research, Prentice-Hall, Inc.,
Englewood Cliffs, NJ, 1969.

21
23. Miller, G.A., "The magical number seven plus or minus two: some limits on our capacity for processing
information," Psychological Review, 63, 81-97, 1956.
24. Putrus, P. "Accounting for Intangibles in Integrated Manufacturing (nonfinancial justification based on the
Analytical Hierarchy Process)," Information Strategy, 6, 25-30, 1990.
25. Ríos Insua, D. Sensitivity analysis in multi-objective decision making. Lecture Notes in Economics and
Mathematical Systems, Springer-Verlag, Berlin, Germany, 1990.
26. Roberts, F.S., Measurement Theory, Addison-Wesley, Reading, Mass, 1979.
27. Roy, B., "How the outranking relation helps multiple criteria decision making." In: Multiple Criteria
Decision Making, Cochrane and Zeleny (Eds.), University of South Carolina Press, SC, 179-201, 1973.
28. Saaty, T.L., "A scaling method for priorities in hierarchical structures," Journal of Mathematical
Psychology, 15, 57-68, 1977.
29. Saaty, T.L., The Analytic Hierarchy Process, McGraw-Hill International, New York, NY, 1980.
30. Saaty, T.L., "Axiomatic foundations of the Analytic Hierarchy process," Management Science, 32(2), 841-
855, 1983.
31. Saaty, T.L., "An exposition of the AHP in reply to the paper 'Remarks on the Analytic Hierarchy Process',"
Management Science, 36(3), 259-268, 1990.
32. Saaty, T.L., Fundamentals of Decision Making and Priority Theory with the AHP, RWS Publications,
Pittsburgh, PA, U.S.A., 1994.
33. Simon, H.A., Models of Man, 2nd Edition, John Wiley and Sons, New York, NY, 1961.
34. Stevens, S.S. and M.D. Hallowell Davis, Hearing, its Psychology and Physiology, American Institute of
Physics, New York, NY, 1983.
35. Stewart, S.M., Introduction to Matrix Computations, Academic Press, New York, NY, 1973.
36. Triantaphyllou, E., "A quadratic programming approach in estimating similarity relations," IEEE
Transactions on Fuzzy Systems, 1(2), 138-145, 1993.
37. Triantaphyllou, E., F.A. Lootsma, P.M. Pardalos, and S.H. Mann, "On the Evaluation and Application of
Different Scales For Quantifying Pairwise Comparisons in Fuzzy Sets," Multi-Criteria Decision Analysis,
3, 1-23, 1994.
38. Triantaphyllou, E., and S.H. Mann, "An evaluation of the AHP and the revised AHP when the eigenvalue
method is used under a continuity assumption," Computers and Industrial Engineering, 2(1), 35-44, 1995.
39. Triantaphyllou, E., and S.H. Mann, "An evaluation of the eigenvalue approach for determining the
membership values in fuzzy sets," Fuzzy Sets and Systems. 35(3), 295-301, 1990.
40. Triantaphyllou, E., and S.H. Mann, "An examination of the effectiveness of multi-dimensional
decision-making methods: A decision-making paradox," International Journal of Decision Support Systems,
5, 303-312, 1989.

22
41. Triantaphyllou, E., P.M. Pardalos, and S.H. Mann, "A minimization approach to membership evaluation
in fuzzy sets and error analysis," Journal of Optimization Theory and Applications, 66(2), 275-287, 1990.
42. Triantaphyllou, E., P.M. Pardalos, and S.H. Mann, "The Problem of Evaluating Membership Values in Real
World Situations," Operations Research and Artificial Intelligence: The Integration of Problem Solving
Strategies, (D.E. Brown and C.C. White III, editors), Kluwer Academic Publishers, 197-214, 1990.
43. Triantaphyllou, E. and A. Sanchez, "A Sensitivity Analysis Approach for Some Deterministic Multi-Criteria
Decision-Making Methods," Decision Sciences, 28(1), 151-194, 1997.
44. Wabalickis, R.N., "Justification of FMS With the Analytic Hierarchy Process," Journal of Manufacturing
Systems, 17, 175-182, 1988.
45. Wang, L., and T. Raz, "Analytic Hierarchy Process Based on Data Flow Problem," Computers & Industrial
Engineering, 20, 355-365, 1991.
46. Write, C., and M.D. Tate, Economics and Systems Analysis: Introduction for Public Managers, Addison-
Wesley, Reading, MA, 1973.
47. Zimmermann, H.-J., Fuzzy Set Theory and Its Applications, Kluwer Academic Publishers, Second Edition,
Boston, MA, 1991.
48. Zwicker, E., Psychoakustic, Springer-Verlag, Berlin, Germany, 1982.