The Delphi method as a research tool: an application in transportation and logistics systems evaluations

Abstract

Delphi is a technique by which a group of experts attempts to generate ideas or find a solution for a specific problem by amassing ideas. In other words, it is a method for the systematic collection and aggregation of informed judgments from a group of experts on specific questions or issues. The Delphi method has proven a popular tool in various application areas for identifying and prioritizing issues for managerial decision-making. Review of Delphi method's application in different transportation and logistics systems evaluations with guidelines for making design choices during the process that ensure a valid study will be presented in this paper.

Full text

6th International Quality Conference
June 08th 2012
Center for Quality, Faculty of Engineering, University of Kragujevac
6th IQC June, 8 2012 401
Branka Dimitrijević1)
Vladimir Simić1)
Vesna Radonjić 1)
Aleksandra Kostić-
Ljubisavljević1)
1) University of Belgrade,
Faculty of Transport and
Traffic Engineering, Vojvode
Stepe 305, Belgrade, Serbia,
mail:{brankad, vsima,
a.kostic,v.radonjic}
@sf.bg.ac.rs
THE DELPHI METHOD AS A RESEARCH
TOOL: AN APPLICATION IN
TRANSPORTATION AND LOGISTICS
SYSTEMS EVALUATIONS
Abstract: Delphi is a technique by which a group of
experts attempts to generate ideas or find a solution for
a specific problem by amassing ideas. In other words, it
is a method for the systematic collection and
aggregation of informed judgments from a group of
experts on specific questions or issues. The Delphi
method has proven a popular tool in various
application areas for identifying and prioritizing issues
for managerial decision-making. Review of Delphi
method’s application in different transportation and
logistics systems evaluations with guidelines for making
design choices during the process that ensure a valid
study will be presented in this paper.
Keywords: Delphi Method, Application, Transport,
Logistics
1. INTRODUCTION
The Delphi technique was developed
during the 1950s by workers at the RAND
Corporation while involved on a U.S. Air
Force sponsored project. The objective
was to develop a technique to obtain the
most reliable consensus of a group of
experts. Since it is designed, the Delphi
technique has become a widely used tool
for measuring and aiding forecasting,
planning and decision making tool in a
variety of disciplines.
While researchers have developed
variations of the method since its
introduction, Delphi may be, in general,
characterized as a method for structuring a
group communication process so that the
process is effective in allowing a group of
individuals, as a whole, to deal with a
complex problem. It allows access to the
attributes of interacting groups (knowledge
from a variety of sources, creative
synthesis, etc.), while pre-empting their
negative aspects (attributable to social,
personal and political conflicts, etc.).
Further more, the method allows input
from a larger number of participants than
could feasibly be included in a group or
committee meeting, and from members
who are geographically dispersed.
Delphi is not a procedure intended to
challenge statistical or model-based
procedures, against which human
judgment is generally shown to be inferior.
Delphi researchers employ this method
primarily in cases in which pure model-
based statistical methods are not practical
or possible because of the lack of
appropriate historical, economic or
technical data, and thus where some form
of human judgmental input is necessary,
and typically use a series of questionnaires
interspersed with controlled opinion
feedback. The Delphi method captures a
wide range of interrelated variables and
multidimensional features common to
most complex problems, both of which are
necessary elements for detailed scientific
analysis. Such input needs to be used as

402 B. Dimitrijević, V. Simić, V. Radonjić, A. Kostić-Ljubisavljević
efficiently as possible, and for this purpose
the Delphi technique might serve a role.
Delphi is a relatively inexpensive
method to organize and administer. It is
one of the most popular forecasting
techniques for technological and industry-
wide forecasting and it is estimated that
90% of technological forecasts and studies
are based on Delphi 9.
This paper is organized in the
following way: a variety of important
factors need to be considered in Delphi
researches are identified in Section 2;
review of Delphi method’s application (as
a judgment or forecasting or decision-
aiding tool) in different studies that treat
transportation and logistics systems
problems is presented in Section 3, and
concluding remarks are given in Section 4.
2. SOME IMPORTANT NOTES
ON DELPHI METHOD
Four key features may be regarded as
necessary for defining a procedure as a
‘Delphi’: anonymity, iteration, controlled
feedback, and the analyzed and statistical
aggregation of group response.
Anonymity is achieved through the
use of questionnaires. By allowing the
individual group members to express their
opinions and judgments privately, undue
social pressures (from dominant or
dogmatic individuals, from a majority)
should be avoided. Furthermore, with the
iteration of the questionnaire over a
number of rounds, the individuals are
given the opportunity to change their
opinions and judgments without fear of
losing face in the eyes of the (anonymous)
others in the group [17].
At the beginning Delphi researchers
clearly outline one or more questions that
research study investigates. Factors which
will answer the questions have to be
identified in the next step. Pre-testing is
also an important reliability assurance for
the Delphi method. It can be used to verify
if the identified factors are indeed
pertinent.
Selecting an appropriate group of
experts who are qualified to answer the
questions is one of the most critical
requirements. A Delphi study does not
depend on a statistical sample that attempts
to be representative of any population. It is
a group decision mechanism requiring
qualified experts who have deep
understanding of the issues.
The size and constitution of the group
of experts depends on the nature of the
research question and the dimensions
along which the experts will probably
vary. It is common practice to divide
experts into panels which represent
different stakeholder groups (for example:
academics, practitioners, government
officials, etc.). These groups probably
would have somewhat different
perspectives. Since it is a goal to obtain a
reasonable degree of consensus, it would
be best to have panels that separate these
groups. This design also permits
comparison of the perspectives of the
different stakeholder groups. Delphi
literature recommends that each panel size
is 10 to 18 people. The design of iterative
process which will ensure the
identification and invitation of the most
qualified experts can be found in [17].
The initial questionnaire for a Delphi
survey is very simple, since it consists of
an open-ended solicitation of ideas. These
individual factors are then consolidated
into a single set by the monitor team, who
produce a structured questionnaire from
which the views, opinions and judgments
of the Delphi panelists may be elicited in a
quantitative manner on subsequent rounds.
After each of these rounds, responses are
analyzed and statistically summarized,
which are then presented to the panelists
for further consideration. The forecast or
assessment for each item in the
questionnaire is typically represented by
the statistical average (mean/median) on
the final round. A number of different
metrics for measuring non-parametric

6th IQC June, 8 2012 403
assessments or forecasts exists, but
Kendall’s W coefficient of concordance is
widely recognized as the best 17. The
value of W ranges from 0 to 1, with 0
indicating no consensus, and 1 indicating
perfect consensus between lists. Hence,
from the third round onwards, panelists are
given the opportunity to alter prior
estimates on the basis of the provided
feedback (the mean assessment of the item
for the panel, the panelist’s ranking of the
item in the former round, an indication of
the current level of consensus, etc).
Furthermore, if panelists’ assessments fall
outside “pre-specified agreement level”
they may be asked to give reasons why
they believe their selections are correct
against the majority opinion. This
procedure continues until certain stability
in panelists’ responses is achieved.
Most commonly, round one is
structured in order to make the procedure
simpler for the monitor team and panelists;
the number of rounds is variable, though
rarely goes beyond one or two iterations
(during which time most change in
panelists’ responses occurs); and often,
panelists may be asked for just a single
statistic, or for written justifications of
extreme opinions or judgments. These
simplifications are particularly common in
laboratory studies and have important
consequences for the generalization of
research findings 16.
3. AN APPLICATION IN
TRANSPORT AND LOGISTICS
SYSTEMS EVALUATION
The Delphi technique has been around
for over half a century. One moment of
history that is worth emphasizing, is that of
1975, when the first edition of Linstone
and Turoff's [9] edited book on Delphi first
appeared and brought notice of the
approach to a wider audience. Slowly at
first, but at a seemingly growing rate, the
technique has flourished, appearing in
more and more academic domains and
being used for more and more purposes.
Thus, a significant literature is associated
with the Delphi method.
Opposed to theoretical or
methodology related lessons, practical
lessons from the conduct of ‘real’ Delphis
are elaborated in a number of case studies.
Review papers of Delphi applications in
various application areas such as business,
education, health care, real estate,
engineering, environment, social science,
can also be found in literature. Note that
the three most popular areas for Delphi
applications are education, business, and
health care 21, 22.
To the best of our knowledge there are
no online published research paper
rewieving Delphi applications in transport
and logistics areas. That is why we
attempted to gather together details of a
number of recently published (English-
language) studies involving evaluation of
the Delphi technique in transportation and
logistics field. We searched through
Science Direct and Springer/Kluwer
databases as well as EBSCO and ProQuest
services. This search produced 22 papers
relevant to our present concern.
Regarding Table 1, the studies have
been classified according to: area of
application, task, Delphi group size,
number of rounds, nature of Delphi
feedback, and consensus and sensitivity
analysis issues. Delphi applications in
studies from Table 1 have extended from
the prediction of long-range trends in
transport, traffic and logistics issues to
applications in policy formation and
decision making. A number of papers look
at the role of Delphi, not as a standalone
approach, but as a method that may be
enhanced by other approaches, or that may
contribute as input to others. Delphi
method is very often used to develop an
evaluation criteria system which comprises
criteria identification, selection and
prioritization. In 12 of the 22 papers (55%)
Delphi provided inputs for MCDM.

404 B. Dimitrijević, V. Simić, V. Radonjić, A. Kostić-Ljubisavljević
Also, Table 1 shows that various
differences exist in Delphi conduction
through researches. One of the aims of
using Delphi is to achieve consensus
amongst panelists. In 10 of the 22 papers
(45%) consensus issue was not discussed
at all. Evidence from most Delphi studies
shows that convergence towards the
‘group’ average is typical.
Reaching consensus is directly related
to the nature of the feedback presented to
panelists (see Table 1). The feedback
recommended in the ‘classical’ Delphi
comprises medians or distributions plus
arguments of panelists whose estimates
fall outside the upper and lower quartiles.
In the majority of observed studies
feedback usually comprises only medians
or means.
Table 1 - Summary of the methodological features of Delphi in experimental studies
Legend:

- Are consensus and sensitivity analyses conducted in the research? ( (yes),  (no))
NS - Not specified
Ref.
Area of
implementation
Task
Delphi
group
size
Rou-
nds
Nature of Delphi
feedback

[1]
Supply chain
management
Provides a list of journals,
keywords and lines of research
7
NS
NS

[2]
Logistics tool
selection
Determining inputs for multi
criteria decision making
NS
NS
NS

[3]
Low cost carriers’
destination
selection
Ascertaining the weighting,
preference and threshold of
attributes
NS
2
Mode value,
median value, etc

[4]
Evaluation of state
and privately
owned airlines
Determining goals and criteria to
evaluate performances related to
the goals
21
2
Average rating,
standard deviation

[5]
Transportation
planning
Predict future developments
46
1


[6]
Evaluation of
hazardous waste
transportation firms
Criteria definition and evaluation
15
3
NS

[8]
Vehicle emissions
control - policy
making criteria
Criteria evaluation
300
3
NS

[9]
3PL in a value
chain
Criterion weight evaluation
NS
NS
NS

[11]
Aviation industry
Anticipating probable and
wildcard scenarios on the future
57
3
mean, standard
deviation and
interquartile range

[12]
Alliance partner
selection in the
airline industry
Pairwise comparation of the
elements in each level of ANP
and AHP MCDM methods
25
2
consensus index
and weighted
priorities

[13]
Transshipment Port
Selection
Criteria verification and
categorization
10
2
Mode quantity,
average value, etc .

[14]
Driver Support
Systems
Predicting future developments
117
3
median and
interquartile range


6th IQC June, 8 2012 405
Table 1. Continued
[15]
ATTelematics
implementation
Criteria evaluation
66
NS
NS

[16]
External sources of
vehicle propulsion
Predicting future developments
45
3
NS

[19]
Port management
Indicators determination and
evaluation
NS
NS
NS

[20]
Road freight
transport
Indicators determination and
evaluation
100
2
Mean and standard
deviation

[21]
Traffic safety
Criteria definition and their
weight evaluation
40
2
NS

[24]
Efficiency of goods
transportation
Criteria evaluation
15
2
Mean values

[25]
Sustainable
transportation
system
Scenarios construction and their
evaluation
63
2
Mean values

[26]
Public
transportation
Determining inputs for multi
criteria decision making
NS
2
Mean values

[27]
International
distribution center
location
Determining inputs for multi
criteria decision making
3
2
NS

[28]
Bus scheduling
Indicators determination and
evaluation
20
2
Medium values

4. CONCLUSION
The Delphi method is a versatile
research tool that researchers can employ
at various points in their research. Use of
the Delphi method for forecasting and
issue identification/prioritization in
transport and logistics systems are
observed in this paper. The number of
papers analyzed and reviewed here leads
us to a conclusion that Delphi method has
significant impact on researches in those
areas and that Delphi conducted according
to ‘ideal’ specifications can perform more
quality results.
REFERENCES:
[1] Bellmunt T.V, et al., “Supply chain management: A multidisciplinary content analysis of
vertical relations between companies, 1997–2006”, Ind. Mark. Man. 40 (2011) 1347-1367
[2] Buyukozkan, G., et al. “Logistics tool selection with two-phase fuzzy multi criteria
decision making: A case study for personal digital assistant selection”, Expert Systems
with Applications 39 (2012) 142–153.
[3] Changa, Y-C., Hsu, C-J., et al., “Low cost carriers’ destination selection using a Delphi
method”, Tourism Management 29 (2008) 898–908.
[4] Cooper, W.W., et al. “A Delphi study of goals and evaluation criteria of state and
privately owned Latin American airlines”, Socio-Econ. Plann. Sci. 29 (4) (1995) 273-285.
[5] Gray, P., Helmer, O., “The Use of Futures Analysis for Transportation Research
Planning”, Transportation Journal 16 (2) (1976) 5-12.
[6] Gumus, A.T., “Evaluation of hazardous waste transportation firms by using a two step
fuzzy-AHP and TOPSIS methodology”, Expert Syst. with Applicat. 36 (2009) 4067–4074

406 B. Dimitrijević, V. Simić, V. Radonjić, A. Kostić-Ljubisavljević
[7] He, J., Hung, W., “Perception of policy-makers on policy-making criteria: The case of
vehicle emissions control”, Science of the Total Environment 417-418 (2012) 21–31.
[8] Kayakutlu, G., Buyukozkan, G., “Assessing performance factors for a 3PL in a value
chain”, Int. J. Production Economics 131 (2011) 441–452.
[9] Linstone, H.A., Turoff, M., “The Delphi Method: Techniques and Applications”,
Reading, MA, Addison-Wesley (2002)
[10] Linz, M., “Scenarios for the aviation industry: A Delphi-based analysis for 2025”, Journal
of Air Transport Management 22 (2012) 28-35.
[11] Liou, J., Tzengb, G-H, et al., “A hybrid ANP model in fuzzy environments for strategic
alliance partner selection in the airline industry”, Appl. Soft Comp. 11 (2011) 3515–3524
[12] Lirn, T.C., et al., “An application of AHP on transhipment port selection: a global
perspective”, Maritime Economics & Logistics, 6, (2004) 70–91.
[13] Marchau,V.,vander Heijden R.,“Policy aspects of driver support systems implementation:
results of an international Delphi study”, Transport Policy 5 (1998) 249-258.
[14] McDonald, M., et al., “Deployment of interurban ATT test scenarios (DIATS):
implications for the European road network”, Transport Reviews, 21 (3) (2001) 303-335.
[15] Mulder, K., van de Weijer, C., Marchau, V., “Prospects for external sources of vehicle
propulsion”, Futures 28 (10) (1996) 919-945.
[16] Nowack, M., et al., “Review of Delphi-based scenario studies: Quality and design
considerations”, Technological Forecasting & Social Change 78 (2011) 1603–1615.
[17] Okoli, C., Pawlowski, S.D., “The Delphi method as a research tool: an example, design
considerations and applications”, Information & Management 42 (2004) 15–29.
[18] Peris-Mora, E., Diez Orejas, J.M., “Development of a system of indicators for sustainable
port management”, Marine Pollution Bulletin 50 (2005) 1649–1660.
[19] Piecyk, M.I., McKinnon, A.C., “Forecasting the carbon footprint of road freight transport
in 2020”, Int. J. Production Economics 128 (2010) 31–42.
[20] Pirdavani, A., et al., “A multiple criteria decision-making approach for prioritizing
accident hotspots in the absence of crash data”, Transport Reviews, 30 (1) (2010) 97–113.
[21] Rowe, G., Wright, G., “The Delphi technique as a forecasting tool: issues and analysis”,
International Journal of Forecasting 15 (1999) 353–375.
[22] Rowe, G., Wright, G. (eds), A special issue of technological forecasting and social change
dedicated to Delphi method, Technological Forecasting & Social Change 78 (2011)
[23] Samuelsson, A., Tilanus, B., “A framework efficiency model for goods transportation
with an application to regional less-than-truckload distribution”, Transport Logistics, 1
(2) (1997) 139–151.
[24] Shiftan, Y., Kaplan, S., Hakkert, S., “Scenario building as a tool for planning a
sustainable transportation system”, Transportation Research Part D 8 (2003) 323–342.
[25] Tzenga, G-H., Lina, C-W., Opricovic, S, “Multi-criteria analysis of alternative-fuel buses
for public transportation”, Energy Policy 33 (2005) 1373–1383.
[26] Wibowo, S., Deng, H., “A fuzzy rule-based approach for screening international
distribution centres”, Computers and Mathematics with Applications,
doi:10.1016/j.camwa.2012.03.026.
[27] Zhiqiang, L., Li, Z. et al., “Evaluation Method about Bus Scheduling Based on Discrete
Hopfield Neural Network”, J Transpn Sys Eng & IT 11(2) (2011) 77−83.
Acknowledgment: This work was partially supported by Ministry of Science and
Technological Development of the Republic of Serbia through the project TR 36006 for the
period 2011–2014.