A comparative analysis of multi-criteria decision-making methods

Abstract

In this work, we empirically compare the rankings produced by several multi-criteria decision-making methods. We analyzed multi-MOORA, TOPSIS and three different settings for VIKOR. Using decision matrices with different number of alternatives and criteria, we compared the rankings produced using the Spearman’s correlation coefficient index. Our results showed that VIKOR could fail to obtain a ranking due to the failure of certain calculations. The rankings produced by TOPSIS and multi-MOORA were very similar, while the rankings produced by the different VIKOR variants showed a great variability.

Full text

Prog Artif Intell
DOI 10.1007/s13748-016-0093-1
REGULAR PAPER
A comparative analysis of multi-criteria decision-making methods
Blanca Ceballos1·María Teresa Lamata1·David A. Pelta1
Received: 20 January 2016 / Accepted: 4 April 2016
© Springer-Verlag Berlin Heidelberg 2016
Abstract In this work, we empirically compare the rank-
ings produced by several multi-criteria decision-making
methods. We analyzed multi-MOORA, TOPSIS and three
different settings for VIKOR. Using decision matrices with
different number of alternatives and criteria, we compared
the rankings produced using the Spearman’s correlation
coefficient index. Our results showed that VIKOR could
fail to obtain a ranking due to the failure of certain cal-
culations. The rankings produced by TOPSIS and multi-
MOORA were very similar, while the rankings produced
by the different VIKOR variants showed a great variabil-
ity.
Keywords TOPSIS ·VIKOR ·Multi-MOORA ·Rankings
comparison
1 Introduction
Multi-criteria decision-making (MCDM) methods are math-
ematical models that help to take decisions in scenarios where
the possible alternatives are evaluated over multiple conflict-
ing criteria.
BDavid A. Pelta
dpelta@decsai.ugr.es
Blanca Ceballos
bceballos@decsai.ugr.es
María Teresa Lamata
mtl@decsai.ugr.es
1Department of Computer Science and Artificial Intelligence,
University of Granada, 18014 Granada, Spain
The application areas of these methods are huge [8].
Examples can be found in supplier selection [18], techni-
cal evaluation of tenderers [10], evaluation of service quality
[9] or in renewable energy [1].
When facing a specific MCDM problem, there are no
clear guidelines on which MCDM method should be used
to solve it. This issue is controversial and it has been
studied in the literature since many decades ago [15–
17,19]. It is true that depending on the MCDM method
applied, the solution could be different, specially when
the alternatives are very similar. Therefore, we seek to
do a comparative analysis among some MCDM meth-
ods, in order to better understand their similarities and
differences. The long-term goal is to have guidelines to sup-
port the decision-maker in the selection of which MCDM
method to apply. We consider this work a step towards such
goal.
There are many MCDM methods in the literature, as
PROMETHEE [3,4], AHP [14], ELECTRE [13], etc. In
this work, we focus on multi-MOORA [2,5], TOPSIS [7]
and VIKOR [11,12]. Multi-MOORA applies aggregation
operators, while TOPSIS and VIKOR operates calculating
distances to “ideal” or “reference” points. We selected these
methods for comparison because they have the same input
and all of them rely on a normalization procedure.
The comparison among methods is done over a set of
randomly generated decision matrices, as in [19] and then the
ranking “agreement” between pairs of methods is assessed
through the Spearman’s correlation coefficient.
The remainder of this paper is organized as follows. Sec-
tion 2provides a brief overview on what is an MCDM
problem, and describes the basic calculations of multi-
MOORA, TOPSIS and VIKOR. Section 3describes the
experimental framework and the results of the experiments.
Finally, Sect. 4is devoted to conclusions.
123

Prog Artif Intell
2 Multi-criteria decision-making problem and
methods
An MCDM problem [15] is composed by a finite set of alter-
natives represented as A={Ai|i=1,2,...,m},mbeing
the number of the alternatives. The alternatives are evalu-
ated according to certain criteria, denoted as C={Cj|j=
1,2,...,n}, where nis the number of the criteria. The criteria
can have different domains, and may represent a cost (which
is desirable to minimize) or a benefit (desirable to maximize).
In addition, each criterion is assigned an importance weight,
represented as W={wj|j=1,2,...,n}. These weights
are normalized to add up to one, i.e., n
j=1wj=1.
This information is organized in a decision matrix (Mm×n)
as in Table 1, where each element xij represents the value
of the alternative Aiwith respect to the criterion Cj.The
matrix Mand the vector of weights W={w1,w
2,...,w
n}
are the fundamental inputs for the MCDM methods that we
will consider here.
2.1 TOPSIS method
The TOPSIS method [7] evaluates the alternatives in terms
of their distance to the so-called “positive” and “negative”
ideal solution. TOPSIS is composed by the following steps:
Step 1 Normalize the decision matrix replacing every xij
by nij using the following formula:
nij =xij
m
j=1(xij)2
,(1)
where i=1,2,...,mand j=1,2,...,n.
Step 2 Calculate the weighted normalized values as vij =
wj∗nij, where wjcorrespond to the weight of the jth cri-
terion, i=1,2,...,mand j=1,2,...,n.
Step 3 Calculate the positive ideal solution (PIS), A+, and
the negative ideal solution (NIS), A−, as follows:
(PIS)=A+={v+
1,v
+
2,...,v
+
j,...,v
+
n},
(NIS)=A−={v−
1,v
−
2,...,v
−
j,...,v
−
n},(2)
where v+
j=maxi(vij)and v−
j=mini(vij)if the jth crite-
rion is benefit; and v+
j=mini(vij)and v−
j=maxi(vij)if
the jth criterion is cost, i=1,2,...,mand j=1,2,...,n.
Table 1 Decision matrix of an
MCDM problem MCDM C1C2··· Cn
A1x11 x12 ··· x1n
A2x21 x22 ··· x2n
··· ··· ··· xij ···
Amxm1xm2··· xmn
Step 4 Calculate the distances from every alternative to
the ideal solutions, d+
ibeing the distance to A+, and d−
ithe
distance to A−as following:
d+
i=⎧
⎨
⎩
n

j=1
(vij −v+
j)2⎫
⎬
⎭
1/2
,
d−
i=⎧
⎨
⎩
n

j=1
(vij −v−
j)2⎫
⎬
⎭
1/2
,(3)
which correspond to the m-dimensional Euclidean distance
and i=1,2,...,m.
Step 5 Calculate the relative closeness to both ideal solu-
tions as following:
Ri=d−
i
d+
i+d−
i
,(4)
where i=1,2,...,m.If Ri=0, then d−
i=0 means that it
is the worst possible case. On the other hand, if Ri= 1, then
d+
i= 0 means that it is the best possible case. In general,
0≤Ri≤1.
Step 6 Rank the alternatives according to Riin descending
order. The best alternative is the one with the highest Ri.
2.2 VIKOR method
The VIKOR method [11] is, as TOPSIS method, also based
in the idea of the distances to “ideal solutions”. However,
some differences exist between both methods, as stated in
[12].
VIKOR method follows these steps:
Step 1 Determine the best f∗
jand worst f−
jvalues of
each criterion as f∗
j=maxi(xij)and f−
j=mini(xij),if
the jth criterion is benefit, and as f∗
j=mini(xij)and f−
j=
maxi(xij)if the jth criterion is cost, i=1,2,...,mand
j=1,2,...,n.
Step 2 Normalize the xij values as follows:
nij =
f∗
j−xij
f∗
j−f−
j
,(5)
where i=1,2,...,mand j=1,2,...,n.
Step 3 Calculate the values Siand Ri,i=1,2,...,m
and j=1,2,...,n:
Si=
n

j=1
wj∗nij,(6)
Ri=max
jwj∗nij(7)
123

Prog Artif Intell
Step 4 Calculate Qias follows:
Qi=v(Si−S∗)
(S−−S∗)+(1−v) (Ri−R∗)
(R−−R∗),(8)
where S∗=mini(Si),S−=maxi(Si),R∗=mini(Ri),
R−=maxi(Ri), and v∈[0,1]. Parameter vbalances the
relative importance of indexes Sand R.
Step 5 Sort Qin increasing order. The best-ranked alter-
native is the one with the lowest value of Q.
Step 6 Compromise solution: the so-called compromise
solution is the alternative awhich is the best ranked accord-
ing to Q(minimum) if the following two conditions are
satisfied:
–Condition 1: Acceptable advantage.Q(a)−Q(a)≥
DQ where a is the second best alternative according to
Qand DQ =1/(m−1)(mis the number of alternatives).
–Condition 2: Acceptable stability in decision-making.
Alternative amust be also the best ranked according
to Sand/or R(the alternative with the lowest value).
If one of the conditions is not satisfied, then a set of com-
promise solutions is proposed, which consists of:
– The alternatives aand a if condition 1 is true and con-
dition 2 is false, or
– The set of alternatives a,a,...,a(p)if condition 1 is
false; pbeing the position in the ranking of the alternative
a(p)verifying Q(a(p))−Q(a)<DQ.
The best alternative, ranked by Q, is the one with the
minimum value of Q. The compromise ranking result is the
compromise ranking list of the alternatives with the “advan-
tage rate”.
2.3 Multi-MOORA method
Multi-MOORA constructs a ranking departing from three
calculations: the “Ratio System”, the “Reference Point” and
the “Full Multiplicative Form of Multiple Objectives” [5].
2.3.1 Ratio system
The first step is the normalization of the decision matrix.
Normalization is done according to Eq. 1(as in TOPSIS)
and the values are denoted as nij. Then, the ratio y∗
iof every
alternative is calculated as follows:
y∗
i=
g

j=1
nij ∗wj−
n

j=g+1
nij ∗wj,(9)
where i=1,2,...,m,j=1,2,...,gare the benefit cri-
teria and j=g+1,2,...,mare the cost criteria. A higher
ratio y∗
iimplies a better ranking of the alternative.
2.3.2 Reference point
Initially, a reference point rjis calculated using the normal-
ized values and the weights. It is defined as rj=max j(nij ∗
wj)if Cjis a benefit criteria, and as rj=min j(nij ∗wj)
if Cjis a cost criteria. Then, every alternative is assigned a
value using the following metric:
min
i(max
j|rj−nij ∗wj|)(10)
The lower the value, the better the alternative is.
2.3.3 Full multiplicative form
An additional value Uiis calculated for every alternative:
Ui=g
j=1nwj
ij
n
j=g+1nwj
ij
,(11)
where i=1,2,...,m,j=1,2,...,gare the benefit crite-
ria and j=g+1,2,...,mare the cost criteria. Finally, the
best-ranked alternative according to the full multiplicative
form is the one that has the highest value of U.
In order to construct the final ranking, multi-MOORA cal-
culates a “summary of rankings” from the Ratio System,
Reference Point and Full Multiplicative Form by applying
the “Theory of Dominance” [6].
3 Experiments and results
3.1 Methodology
Our experiments departs from a set of randomly generated
MCDM problems (i.e., decision matrices). These matrices
were generated according to the procedure described in [19],
where the following parameters were used:
1. Number of criteria:n∈{5,10,15,20}.
2. Number of alternatives:m∈{3,5,7,9}.
3. Values of the alternatives:xij: randomly generated from
a uniform distribution in [0.01,...,1].
4. Criteria Weights: all of them are considered equally
important, thus wi=1/n.
5. Number of replications: we have generated 100 matrices
for each combination of mand n, t hus producing 4 ×4×
100 =1600 MCDM problems.
6. Methods: five methods are considered: TOPSIS, multi-
MOORA (MM) and three different parametrization of
VIKOR using v={0,0.5,1}(named as VIKOR0,
VIKOR0.5,VIKOR
1). When using VIKOR, we just take
123

Prog Artif Intell
the ranking generated with the Qindex, without consid-
ering the conditions of the “Compromise solution”, since
these are strong conditions (hard to meet) for the number
of alternatives in our experiments.
Every MCDM problem is solved using every method, giv-
ing a total of 8000 rankings.
Then, we measure the agreement between every pair of
rankings (one from each method) using the Spearman’s cor-
relation coefficient (ρ). An index value closer to 1 indicates a
high level of agreement (a “1” is obtained if the rankings are
equal). If the index is close to 0, then there is no agreement
between the rankings. Finally, if the index is close to −1, the
rankings are almost inverted.
For every combination of number of alternatives and cri-
teria, we analyze the results in terms of the average values of
ρover the corresponding 100 MCDM problems.
The analysis of results is separated in three parts. The
first part focuses on a problem affecting two variants of the
VIKOR method (VIKOR0and VIKOR0.5). The second part
provides a comparison of the methods on those cases where
VIKOR variants worked. Finally, in the third part, we com-
pare multi-Moora, TOPSIS and VIKOR1over the whole set
of problems.
3.2 A problem concerning the VIKOR method
We have observed that on some MCDM problems, VIKOR0
and VIKOR0.5failed to produce a ranking. This failure does
not appear with VIKOR1.
The reason lies at the core of the Rcalculation performed
in VIKOR (Eq. 7). In some problems, we observed that all
the alternatives have the same Rivalue. As a consequence
R∗=R−and the calculation of the Q(Eq. 8) index becomes
indeterminate for any v= 1. Potentially, the same situation
may occur with the Sivalues, but this never happened in our
experiment.
Table 2shows, for every combination of alternatives and
criteria, the percentage of problems solved by VIKOR0
and VIKOR0.5. Increasing the number of criteria, led to a
decrease in the percentage of problems solved (see the table
by rows). Also, if the number of criteria is fixed (reading the
Table 2 Percentage of problems solved (out of 100) by VIKOR0and
VIKOR0.5
Alternatives (m) Number of criteria (n)(%)
5 101520
332430
59447122
7 100 86 56 28
9 100 98 88 56
3579
0
25
50
75
100
Percentage
Number of alternatives
Percentage of solutions in VIKOR
Criteria
5
10
15
20
Fig. 1 Percentage of problems solved (out of 100) by VIKOR0and
VIKOR0.5
table by columns), the problem becomes “more solvable”
as the number of alternatives increases. For example, when
n=5, the percentage of problems solved raised from 32 %
when m=3 to 100 % when m=7 and m=9. Just in these
last combinations (n=5,m={7,9}), VIKOR is able to
provide a ranking for all the MCDM problems available.
Figure 1graphically shows the results of Table 2.Itcan
be observed that the rate of problems solved according to the
number of criteria varies in terms of the number of alterna-
tives available. When having n=20 criteria, the decision
problem should have much more than m=9 alternatives to
produce results. However, when n=5, it is almost enough
to have just m=5 alternatives to produce a ranking when
the MCDM problems are generated as we did. In general, the
percentage of problems solved increases with the number of
the alternatives available.
3.3 First comparison: all methods over a subset of
problems
Now, we will show an all-against-all comparison of the meth-
ods restricted to the cases where VIKOR0and VIKOR0.5
solved more than 80 % of the problems. These six cases
are n=5,m={5,7,9};n=10,m={7,9}and
n=15,m=9. Recall that we had tested 100 problems
in every case. The global results are shown in Table 3, where
the average values of ρ(over 600 MCDM problems) between
every pair of methods is shown.
On average, the agreement of the rankings produced by
TOPSIS and VIKOR1, and TOPSIS with multi-MOORA is
123

Prog Artif Intell
Table 3 Mean of ρover 600 MCDM problems (considering n=
5,m={5,7,9};n=10,m={7,9}and n=15,m=9)
Methods Mean ρ
TOPSIS–VIKOR10.88
TOPSIS–MM 0.87
VIKOR1–VIKOR0.50.83
VIKOR1–MM 0.82
VIKOR0.5–MM 0.78
VIKOR0.5–TOPSIS 0.75
VIKOR0–VIKOR0.50.61
VIKOR0–MM 0.44
VIKOR0–VIKOR10.35
VIKOR0–TOPSIS 0.33
MM multi-MOORA
VIKOR1−TOPSIS
VIKOR1−VIKOR0.5
MM−TOPSIS
MM−VIKOR0.5
MM−VIKOR1
VIKOR0.5−TOPSIS
VIKOR0−VIKOR0.5
VIKOR0−MM
VIKOR0−VIKOR1
VIKOR0−TOPSIS
0.25
0.5
0.75
1
Mean of ρ
Methods
5 Criteria
Alternatives
5
7
9
Fig. 2 Mean of ρfor 5 criteria and m={5,7,9}.MM multi-MOORA
very high (ρ≥0.87). Less similar rankings are produced
by VIKOR0and TOPSIS (ρ=0.33). Moreover, as the last
four rows correspond to VIKOR0, we can say that this method
produced rankings that shows very little similarity with those
produced by other methods. A similar observation can be
done regarding VIKOR0.5.
If we split the analysis in terms of the problem size, we
obtain the results depicted in Figs. 2,3and 4. Figures show
the mean of ρfor every pair of methods. Figure 2shows
the case for 5 criteria and different number of alternatives
(methods are sorted according with the results when m=9).
Figure 3considers 10 criteria and m={7,9}(methods are
also sorted according with the results when m=9). Finally,
VIKOR1−TOPSIS
MM−TOPSIS
VIKOR1−VIKOR0.5
MM−VIKOR1
MM−VIKOR0.5
VIKOR0.5−TOPSIS
VIKOR0−VIKOR0.5
VIKOR0−MM
VIKOR0−VIKOR1
VIKOR0−TOPSIS
0.25
0.5
0.75
1
Mean of ρ
Methods
10 Criteria
0.85 0.87
0.76
0.83
0.69 0.66
0.30
0.24 0.21
0.55
0.89 0.86 0.86 0.82 0.80 0.78
0.34
0.24 0.23
0.50
Alternatives
7
9
Fig. 3 Mean of ρfor 10 criteria and m={7,9}.MM multi-MOORA
VIKOR1−TOPSIS
MM−TOPSIS
MM−VIKOR1
VIKOR1−VIKOR0.5
MM−VIKOR0.5
VIKOR0.5−TOPSIS
VIKOR0−VIKOR0.5
VIKOR0−MM
VIKOR0−VIKOR1
VIKOR0−TOPSIS
0.25
0.5
0.75
1
Mean of ρ
Methods
15 Criteria
0.89 0.88 0.86 0.82
0.76 0.73
0.21 0.20
0.45
0.27
Fig. 4 Mean of ρfor 15 criteria and m=9. MM multi-MOORA
Fig. 4corresponds to n=15,m=9. The corresponding
values for Fig. 2are shown in Table 4.
The similarities among methods when considering the
problem sizes are quite similar to those shown in the global
analysis (Table 3). This fact can be checked seeing the order
of the methods in the X axis of the plots in Figs. 2,3and 4.In
addition, we can mention that the mean of ρis higher when
the number of alternatives is increased.
123

Prog Artif Intell
Table 4 Mean of ρfor MCDM problems with 5 criteria
Methods Alternatives (m)
579
VIKOR1–TOPSIS 0.79 0.88 0.91
VIKOR1–VIKOR0.50.80 0.84 0.89
MM–TOPSIS 0.87 0.86 0.88
MM–VIKOR0.50.73 0.82 0.87
MM–VIKOR10.78 0.81 0.85
VIKOR0.5–TOPSIS 0.67 0.77 0.85
VIKOR0–VIKOR0.50.67 0.72 0.74
VIKOR0–MM 0.47 0.58 0.64
VIKOR0–VIKOR10.39 0.46 0.53
VIKOR0–TOPSIS 0.36 0.45 0.52
Table is sorted according to m=9
MM multi-MOORA
A fact somehow “surprising” is the influence of the vvalue
in the output of VIKOR method with respect to TOPSIS.
When considering v=1(VIKOR
1), the output of the meth-
ods are almost similar. However, when v=0(VIKOR
0), the
outputs produced are quite different. In other words, VIKOR
shows a wide range of behaviors depending on a single para-
meter that makes it to behave or not like TOPSIS. To the
best of our knowledge, there are no guidelines to set up such
value when facing a new problem, so it is hard to imagine
how such value should be defined.
It should also be highlighted that TOPSIS and multi-
MOORA show a high mean value of ρ, stating that their
outputs are quite similar.
3.4 Second comparison: a subset of methods over all the
problems
In this subsection, we provide an all-against-all compari-
son of multi-MOORA, TOPSIS and VIKOR with v=1
(VIKOR1) over all the test problems considered (1600 dif-
ferent decision matrices).
Table 5shows the mean of ρfor every pair of methods, and
for all combinations of number of criteria and alternatives.
The values are also shown as plots in Fig. 5.
Focusing first on the comparison of multi-MOORA vs.
TOPSIS, we can observe a quite high level of agreement
between the rankings. The less similar case achieved an aver-
age of ρ=0.81 when m=3,n=10. As Fig. 5ashows,
such similarity is independent of the number of alternatives
and criteria.
The situation is different when multi-MOORA is com-
pared against VIKOR1. Now, the average ρranges from 0.58
(when m=3,n=10) to 0.85 (when m=9,n=5). From
Fig. 5b, it is clear that as the number of alternatives increased,
the outputs of the methods became more similar.
The comparison of TOPSIS vs. VIKOR1led to a similar
result. We should highlight the case where m=9 and n=5,
with an average ρ=0.91. This means that the outputs of
both methods are almost the same over the 100 tested cases.
Again, as Fig. 5c indicates, there is a clear tendency showing
that a higher number of alternatives implies a higher rank
similarity (independent of the number of criteria).
4 Conclusions
In this work, we performed a comparison of different MCDM
methods over 1600 randomly generated decision problems to
understand their similarities and differences in terms of the
rankings they produced.
The MCDM methods compared were: multi-MOORA,
TOPSIS and VIKOR, considering v=0, v=0.5 and
v=1, i.e., considering the rankings produced by S,Qand
R, respectively, in VIKOR.
From this comparison, two main points can be outlined:
the first one is relative to VIKOR method in particular, while
the second one is relative to multi-MOORA, TOPSIS and
VIKOR with v=1.
The first observation we did concerns the VIKOR method
in two aspects. Firstly, we point out that its output strongly
depends on the parameter v. The rankings produced when
using v={0.5,1}are quite similar. However, the use of
v=0 led to rankings that showed quite low similarities with
those from the other methods considered.
Secondly, when using VIKOR with v={0,0.5}we found
that the method failed to produce a ranking for many decision
Table 5 Mean of ρfor
all-against-all method’s
comparison
mn
MM vs. TOPSIS MM vs. VIKOR1TOPSIS vs. VIKOR1
5 10 15 20 5 10 15 20 5 10 15 20
3 0.88 0.81 0.87 0.88 0.64 0.58 0.67 0.71 0.61 0.51 0.61 0.68
5 0.87 0.86 0.84 0.86 0.78 0.79 0.77 0.76 0.79 0.78 0.78 0.80
7 0.86 0.87 0.85 0.85 0.81 0.83 0.84 0.83 0.88 0.85 0.86 0.87
9 0.88 0.86 0.88 0.84 0.85 0.82 0.86 0.83 0.91 0.89 0.89 0.88
MM multi-MOORA
123

Prog Artif Intell
3579
0.5 0.75 1
Mean of ρ
Number of alternatives
Multi−MOORA vs. TOPSIS
Criteria
5
10
15
20
3579
0.5 0.75 1
Mean of ρ
Number of alternatives
Multi−MOORA vs. VIKOR(v=1)
Criteria
5
10
15
20
(b)(a)
3579
1
5
7.
0
5.0
Mean of ρ
Number of alternatives
TOPSIS vs. VIKOR(v=1)
Criteria
5
10
15
20
(c)
Fig. 5 Mean of ρfor all-against-all method’s comparison
problems. In the most extreme case, VIKOR could not solved
any of the 100 problems with 3 alternatives and 20 criteria.
We may assume that this is a quite unrealistic situation, but it
also failed to provide a ranking in 44 problems (out of 100)
with 9 alternatives and 20 criteria. The problem relies on the
calculation of the Qindex, which is a convex combination of
two terms. One of them (R) produced a division by zero thus
leading to a indetermination in the Qvalue. In such cases,
VIKOR will fail for any v= 1. In other words, just one
possible value for vis “safe”.
The second observation address the results of the compar-
ison among multi-MOORA, TOPSIS and VIKOR1. Multi-
MOORA and TOPSIS obtained very similar results (their
rankings are almost the same), being the lowest value
of ρ0.81. A high level of similarity was obtained inde-
pendently of the number of criteria/alternatives consid-
ered.
The similarity of multi-MOORA with VIKOR1, and
VIKOR1with TOPSIS is not as high, but it clearly increased
as more alternatives are available. In both cases, the number
123

Prog Artif Intell
of criteria showed a quite minor influence on the similarity
values.
In short, our conclusions are in two lines: (1) if the problem
will be solved using several methods, then the user should not
choose simultaneously multi-MOORA and TOPSIS; and (2)
VIKOR’s ranking is very sensitive to the parameter v, thus it
should be carefully defined.
Acknowledgments Authors acknowledge support through projects
TIN2014-55024-P and P11-TIC-8001 from the Spanish Ministry of
Economy and Competitiveness, and Consejería de Economía, Inno-
vación y Ciencia, Junta de Andalucía (both including FEDER funds),
respectively. B. Ceballos enjoys a training Grant research staff under
project TIN2011-27696-C02-01 (Spanish Ministry of Economy and
Competitiveness).
References
1. Abu Taha, R., Daim, T.: Multi-criteria applications in renewable
energy analysis, a literature review. In: Daim, T., Oliver, T., Kim,
J. (eds.) Research and Technology Management in the Electricity
Industry: Methods, Tools and Case Studies, pp. 17–30. Springer,
London (2013)
2. Baležentis, T., Baležentis, A.: A survey on development and
applications of the multi-criteria decision making method MULTI-
MOORA. J. Multi-Criteria Decis. Anal. 21(3–4), 209–222 (2014)
3. Brans, J.P., Mareschal, B., Vincke, P.: Promethee: a new family
of outranking methods in multicriteria analysis. In: Operational
Research ’84, Proceedings of the Tenth International Conference,
pp. 477–490. Washington, DC (1984)
4. Brans, J.P., Vincke, Ph, Mareschal, B.: How to select and how
to rank projects: the Promethee method. Eur. J. Oper. Res. 24(2),
228–238 (1986)
5. Brauers, W.K.M., Zavadskas, E.K.: Project management by mul-
timoora as an instrument for transition economies. Technol. Econ.
Dev. Econ. 16(1), 5–24 (2010)
6. Brauers, Willem Karel M., Zavadskas, Edmundas Kazimieras:
Robustness of multimoora: a method for multi-objective optimiza-
tion. Informatica 23(1), 1–25 (2012)
7. Chen, S.J., Hwang, C.L., Hwang, F.P.: Fuzzy multiple attribute
decision making(methods and applications). Lecture Notes in Eco-
nomics and Mathematical Systems (1992)
8. Mardani, A., Jusoh, A., Zavadskas, E.K.: Fuzzy multiple criteria
decision-making techniques and applications: two decades review
from 1994 to 2014. Expert Syst. Appl. 42(8), 4126–4148 (2015)
9. Mardani, A., Jusoh, A., Zavadskas, E.K., Khalifah, Z., Nor,
K.M.D.: Application of multiple-criteria decision-making tech-
niques and approaches to evaluating of service quality: a systematic
review of the literature. J. Bus. Econ. Manage. 16(5), 1034–1068
(2015)
10. Mohamed, N.H., Lamsali, H., Sathyamoorthy, D.: Multi-criteria
decision making (mcdm) for technical evaluation of tenderers: a
review of methods employed. Defence S T Tech Bull 8(1), 90–102
(2015)
11. Opricovic, S.: Multicriteria Optimization of Civil Engineering Sys-
tems. Faculty of Civil Engineering, Belgrade (1998)
12. Opricovic, S., Tzeng, G.-H.: Compromise solution by MCDM
methods: a comparative analysis of VIKOR and TOPSIS. Eur. J.
Oper. Res. 156(2), 445–455 (2004)
13. Roy, B., Hugonnard, J.C.: Ranking of suburban line extension
projects on the Paris metro system by a multicriteria method.
Transp. Res. A: Gen. 16(4), 301–312 (1982)
14. Saaty, T.L.: The Analytic Hierarchy Process. McGraw-Hill, New
York (1980)
15. Triantaphyllou, E.: Multi-criteria decision making methods. In:
Multi-criteria Decision Making Methods: A Comparative Study,
pp. 5–21. Springer, New York (2000)
16. Triantaphyllou, E., Chi-Tun, L.: Development and evaluation of
five fuzzy multiattribute decision-making methods. Int. J. Approx-
imate Reasoning 14(4), 281–310 (1996)
17. Triantaphyllou, E., Mann, S.H.: An examination of the effective-
ness of multi-dimensional decision-making methods: a decision-
making paradox. Decis. Support Syst. 5(3), 303–312 (1989)
18. Yildiz, A., Yayla, A.Y.: Multi-criteria decision-making methods
for supplier selection: a literature review. S. Afr. J. Ind. Eng 26(2),
158–177 (2015)
19. Zanakis, S.H., Solomon, A., Wishart, N., Dublish, S.: Multi-
attribute decision making: a simulation comparison of select
methods. Eur. J. Oper. Res. 107(3), 507–529 (1998)
123