Content uploaded by Nicola Clibbens
Author content
All content in this area was uploaded by Nicola Clibbens on May 19, 2016
Content may be subject to copyright.
37
© RCN PUBLISHING / NURSE RESEARCHER January 2012 | Volume 19 | Number 2
Nurse Researcher
Delphi research: issues
raised by a pilot study
Correspondence
joe.bloggs@gmail.com
Accepted: April 8 2011.
Cite this article as: Clibbens N, Walters S, Baird W (2012) Delphi research: issues raised by a pilot study.
Nurse Researcher. 19, 2, 37-43.
Abstract
Aim This paper will explore the issues raised by the
pilot stage of a three-round Delphi study.
Background The Delphi method involves a range
of complex activity for the expert panellists and
the researcher and yet there is a lack of debate
in the academic literature about how Delphi
research should be piloted.
Data sources A Delphi study aimed at establishing
areas of agreement between service users
and registered nurses about therapeutic
nursing on acute mental health wards. A pilot
Delphi study tested the first-round questions,
the use of two measurement approaches and
the process of analysis and administration across
three rounds.
Review methods A brief review of published
Delphi pilot studies in health care between
2001 and 2011; ten of 25 relevant papers
are included here.
Discussion Approaches to pilot tests for the Delphi
method are discussed.
Conclusion Delphi researchers should publish
greater detail about their approach to pilot studies.
Pilot Delphi studies can support the development
of first round questions but also offer a means to
test measurement methods and define consensus in
subsequent rounds.
Implications for research/practice Pilot studies in
Delphi research provide useful guidance about
first-round questions as well as measurement
methods, consensus thresholds and controlled
feedback in subsequent rounds. They support the
involvement of professionals and service users and
they need to trial the recruitment strategy to avoid
between-round delays. Delphi researchers should
publish details of their approach to pilot studies.
Keywords Pilot study, Delphi research, involving
service users, mental health nursing
Introduction
DESPITE THE administrative and methodological
complexity of Delphi research (Rowe et al 2005,
Baker et al 2007), there has been limited debate
about pilot studies. Few Delphi researchers describe
pilot stages in their published research (Keeney et al
2001). This paper describes the issues raised by
the pilot stage of a three-round Delphi study. Data
were collected using postal questionnaires with
two panels of experts recruited from one region in
England in 2006. These were:
Registered nurses.
Service users.
The full Delphi study aimed to establish areas of
Correspondence to:
Nicola Clibbens
n.clibbens@shu.ac.uk
Nicola Clibbens PhD, MA (Ed),
RMN is senior lecturer nursing
at the faculty of health and
wellbeing, Sheffield Hallam
University, Sheffield, UK
Stephen Walters is professor in
medical statistics at the health
services research (HSR) and
National Institute for Health
Research Design Service for
Yorkshire & Humber (RDSYH),
University of Sheffield, Sheffield,
UK
Wendy Baird is director National
Institute for Health Research
Research Design Service for
Yorkshire & Humber at the
School of Health and Related
Research University of Sheffield,
Sheffield, UK
Author guidelines
www.nurseresearcher.co.uk
agreement between service users and registered
nurses about therapeutic nursing on acute mental
health wards.
A number of critical reports suggested that
acute wards were unsafe and counter-therapeutic
(Sainsbury Centre for Mental Health (SCMH) 1998,
Standing Nursing and Midwifery Committee 1999).
The central criticisms were that nurses were not
spending time with service users and were not
using evidence-based therapeutic skills. Since
then, social policy has been striving to support
improvement in this care environment (Department
of Health (DH) 2002, Garcia et al 2005, Star Wards
2005, SCMH 2006) and the UK Care Quality
38 January 2012 | Volume 19 | Number 2 © RCN PUBLISHING / NURSE RESEARCHER
Nurse Researcher
Commission (CQC) has noted some improvements
(CQC 2009).
One of the barriers to change has been that the
role of the mental health nurse on acute mental
health wards is difficult to define (Butterworth
1994, Gijbels 1995, DH 2006). Forchuck and
Reynolds (2001) described mental health nurses as
belonging to a ‘house divided’ because of the barrage
of ongoing claims and counterclaims between
‘biological’ and ‘therapeutic relationship’ camps.
Acute ward nurses have also been criticised for
their over-reliance on the custodial aspects of their
role at the expense of their therapeutic role (Dodds
and Bowles 2001, MIND 2004, Buchanan-Barker and
Barker 2005, Baker et al 2007, Rae 2007, Ashmore
2008). User-led research has also shown that
custodial and coercive practices are frightening,
counter-therapeutic and damaging to therapeutic
relationships (Gilburt et al 2008). The importance
of exploring the different perspectives of receivers
and providers of care has been acknowledged
(Faulkner 1997, Campbell 1999, DH 2001) and
Ziglio (1996) noted that Delphi research was one way
to achieve this.
While there is evidence to support specific
psychotherapeutic interventions in mental health
nursing more generally, there has been a lack of
focus on how effective these interventions are
on acute mental health wards (DH 2002). Delphi
research was chosen because it is a useful approach
when the research topic does not lend itself to more
precise analytical techniques and there is a limited
body of knowledge on which to build a scientific
study (Linstone and Turoff 1975).
Ethical approval was granted by the local research
ethics committee and the university ethics panel. The
study was also independently reviewed by a public
sector research and development department and an
independent service user research group.
The Delphi method
Delphi is a hybrid survey design that aims to
reach consensus on important issues (Linstone
and Turoff 1975, Beech 1999). It is characterised
by a specific sequence of events: selection of
an expert panel, formulation of questions,
generation of statements of opinion, reduction and
categorisation, rating, analysis and iteration (Mead
and Moseley 2001).
Although there are a number of Delphi variations
(Thompson 2009), the first round is usually
qualitative and designed to establish the breadth
of expert opinion on a defined topic (Mead and
Moseley 2001). Subsequent rounds use numerical
measures and descriptive statistics to establish
strength of opinion on the items generated by
the experts in the first round. Typically, one
of these rounds will be iterative, and the experts
will be given controlled feedback, which is where
the Delphi researcher shares the expert’s own score
from previous rounds with additional information
such as comments made by other experts and the
statistical patterns of scoring for the whole panel
of experts. Experts rescore in light of this feedback
(Okoli and Pawlowski 2004).
There are a number of ways to define expertise
in Delphi research and some definitions could
exclude service users (Baker et al 2006). To ensure
nurses and service users could legitimately be
described as experts in this study, expertise was
defined by reference to experience (Hasson et al
2000) by ensuring that experts had up-to-date
knowledge of the topic under investigation (Jairath
and Weinstein 1994) and by recruiting participants
who are experts in the eyes of those being studied
(Walker et al 2000).
Where service users have been recruited as
experts, they have most often been involved in
one ‘round’ of the Delphi study (Kennedy 2000,
Walker et al 2000). In this study, the service
users were involved in all three rounds. The pilot
study was therefore crucial to ensuring that the
information and design of the Delphi enabled the
expert views of the nurses and the service users to
be fully appreciated and to avoid tokenism (Hui
and Stickley 2007).
The definition of consensus is crucial to the
rigour of Delphi research (Butterworth and Bishop
1995, Powell 2003) and yet no universal definition
exists (Hasson et al 2000). Although the use of
percentages to indicate majority agreement is
common (McKenna 1994, Wengstrom and Häggmark
1998), this approach has been criticised (Dajani et al
1979). Some observers have suggested that putting
measures in place to indicate stability of response
across rounds is a more rigorous approach (Duffield
1993, Crisp et al 1997). The pilot study here was
useful in refining the definition of consensus
in advance of the full Delphi (Dajani et al 1979,
Boje and Murnighan 1982, Keeney et al 2001,
Rowe et al 2005).
Jairath and Weinstein (1994) suggested
that Delphi research should be piloted and
yet Keeney et al (2001) noted that ‘few Delphi
The importance of exploring the different
perspectives of receivers and providers
of care has been acknowledged
39
© RCN PUBLISHING / NURSE RESEARCHER January 2012 | Volume 19 | Number 2
Delphi method
Nurse Researcher
Figure 1 Pilot and full study ‘round by round’
Figure 2 Design and sample pilot Delphi
researchers report undertaking pilot tests’. Pilot
studies could offer a means to ensure greater rigour,
particularly in light of criticisms about the design
of first-round questions as suggested by Moseley
and Mead (2001) and to the choice of measurement
methods and their analysis in subsequent rounds
as suggested by Hardy et al (2004). Further to
this, Keeney et al (2001) noted that ‘it is unclear
how many pilot tests should be undertaken… for
instance, should there be one for every round or only
one for the initial round?’
To establish the ‘best’ approach to piloting
Delphi research in this study, we reviewed 25 Delphi
research papers published in health care between
2000 and 2011. Six of these papers stated that a
pilot study had been conducted (Quinn and Sullivan
2000, Hassan and Barnett 2002, Cramer et al 2008,
Gabb et al 2006, Hung et al 2008, Valdez 2009); only
three published details of the pilot study design.
Biondo et al (2008) stated that their measures had
been pilot tested before the first round of the full
Delphi study, but failed to outline the process. A
further study had the term ‘pilot study’ in the title
(Dyck 2008) but used the term ‘pilot’ to refer to the
small size of the study rather than to a preparatory
test of the Delphi method.
The most common approach was to pilot the
first round of the Delphi study to increase the
validity of its questions (Hassan and Barnett 2002,
Gabb et al 2006, Cramer et al 2008, Hung et al 2008,
Valdez 2009). Hung et al (2008) suggested that this
was important because question design is difficult
and the questions in the first round are the basis
for subsequent rounds. However, limiting the pilot
study to the first round fails to test the complex
processes of analysis and measurement occurring
later in the Delphi research process.
Quinn and Sullivan (2000) described a five-round
pilot study. This was the only study we located that
piloted all stages of the Delphi process. Conducting
their pilot Delphi study in this way resulted in
‘the phrasing being altered while important and
pertinent instructions and information were
highlighted’.
To ensure rigour across all processes in our
Delphi study, we piloted all three rounds. We
considered two options for the design of the Delphi
pilot. The first was to trial each round of the pilot
study immediately before conducting each round
of the full study (Figure 1). While this could have
helped the full study to start sooner, managing the
pilot sample and the full study sample at the same
time could be complex. Greatorex and Dexter (2000)
warned that overly complex processes in Delphi
research risk ‘between round’ delays and attrition of
the full study sample. This design also failed
to consider the Delphi process as a whole,
which risked important issues being missed,
particularly at later stages.
Having reviewed the advantages and
disadvantages of both pilot study designs (Table 1,
page 40), we chose the second design (Figure 2),
where a three-round pilot study was completed in
advance of recruiting to the full Delphi study. As
well as completing the postal questionnaires, we
asked experts in the pilot study to give specific
feedback about the Delphi process including the
time taken to complete the questionnaires, preferred
Pilot Delphi –
round one
Three rounds
First round
Open breadth
questions
Second round
7-point ordinal
scale, visual
analogue scales,
depth questions
Second round
Seven-point ordinal
scale. Visual
analogue scales,
depth questions
Third round
Feedback and
iteration
Registered nurse
experts
(n=4)
Service user
experts
(n=2)
Registered nurse
experts
(n=4)
Service user
experts
(n=3)
Registered nurse
experts
(n=4)
Service user
experts
(n=3)
Two expert panels
Full study Delphi –
round one
Pilot Delphi –
round two
Pilot Delphi –
round three
Full study Delphi –
round two
Full study Delphi –
round three
January 2012 | Volume 19 | Number 2 © RCN PUBLISHING / NURSE RESEARCHER
40
Nurse Researcher
Table 1 Benefits and disadvantages of different approaches to piloting Delphi surveys
expertise or academic qualifications). This coupled
with the fact that the pilot study participants were
known to the researcher may have resulted in a
stronger motivation to respond than was found in
the full study (McKenna 1994, Hasson et al 2000).
Ensuring that the sampling approach, inclusion
criteria and recruitment strategy in pilot Delphi
studies reflect the full study design could
increase the likelihood that recruitment and
retention problems are identified before the full
study commences. Recruitment problems in the
early stages of Delphi can snowball through all
rounds of the process, causing methodological
and administrative problems. This can then
reduce the validity of the consensus because
of delays between rounds and sample attrition
(Keeney et al 2001).
First pilot study round The questions were designed
following reviews of previous Delphi studies (for
example Walker et al 2000) and with advice from
a service user research group and a professional
scientific panel. This resulted in the two expert
panels being asked: ‘What should registered nurses
on acute wards do to help service users get better?’
Only the registered nurse expert panel was asked:
‘What specific therapeutic interventions should be
used by nurses on acute wards?’
Careful formulation and testing of the first round
questions are crucial to the findings of the whole
Delphi study because they generate responses from
which subsequent rounds are conducted (Mead
and Moseley 2001, Thomson 2009). Failing to take
account of question formulation and testing could
lead to misinterpretation by the experts, flawed
responses and untrustworthy findings (Streiner and
Norman 2008).
Pilot of the whole study Pilot round by round
Benefits Full review of all aspects of the study gives
a stronger sense of the whole.
Review of all complex processes improves
rigour.
Avoids complexity of managing two
samples at the same time.
Avoids unnecessary delays between
rounds.
Full study begins sooner.
Gives contemporaneous and round-specific feedback.
Disadvantages Significantly delays the start of the full
study.
Causes delays between rounds, potentially increasing sample attrition.
Causes added complexity by managing two samples.
Danger of finding methodological problems in round
two or three that should have been dealt with earlier.
method of rating and experiences of taking part in a
Delphi study.
Issues raised by the pilot Delphi study
Sampling and recruitment It is usual to use
purposive sampling in Delphi to ensure that the
experts meet pre-defined definitions of expertise
(Rowe et al 1991, Hardy et al 2004, Baker et al 2006).
The approach in the pilot study was to use elements
of purposive and convenience sampling. The sample
was purposive in that the volunteers met some of
the pre-defined inclusion criteria of the full study; it
was a convenience sample because each expert was
already in contact with the researcher through the
university. This resulted in four registered nurses
who had experience of working on acute mental
health wards, as well as three service users who
had previously been cared for on acute mental
health wards, agreeing to take part (n=7).
Two limitations to this approach emerged:
the recruitment strategy was not piloted and
the experts did not fulfil the same definition
of expertise as the full study. The registered
nurses in the pilot study were lecturers at the
university (n=2) and nurse consultants in acute
mental health care (n=2); the service users were
independent researchers (n=3). The pilot sample had
a greater level of academic education and awareness
of the research process than the experts in the full
study could be assumed to have (the full study
inclusion criteria did not require previous research
Frequent reference is made to establishing
validity in the first round questions but
terms such as validity have limited meaning
© RCN PUBLISHING / NURSE RESEARCHER January 2012 | Volume 19 | Number 2
41
Delphi method
Nurse Researcher
Table 2 Summary of response categories in the first
round of the pilot Delphi
Table 3 Time taken (in minutes) for each person to complete each round
of the Delphi
Categories Number of
statements
Service users
Specific helping actions 15
Medication 4
How nurses should be 8
Information 6
Recreation 4
Access to nurses 5
Registered nurses
Therapeutic care 16
Profession, education and
research
13
How nurses should be 12
Process of care 15
Working with others 4
Nurses Round 1 Round 2 Round 3
Nurses
Nurse 1 45 40 20
Nurse 2 30 30 20
Nurse 3 40 20 20
Nurse 4 20 25 Missing data
Mean time taken 33 28 20
Service users
Service user 1 60 120 30
Service user 2 45 70 30
Service user 3 30 20 Service user
dropped out
Mean time taken 45 70 30
Frequent reference is made to establishing
validity in first round questions (Moseley and Mead
2001, Hung et al 2008, Valdez 2009), but terms
such as validity have limited meaning because the
first round is usually qualitative (Maxwell 2002).
It is important, however, to establish if the questions
elicit responses that meet the aims of the study.
In the pilot study, we achieved this in two ways
by asking the experts in the pilot study to provide
feedback about the questions and by conducting
the first round pilot study analysis to establish if the
responses met the study aims.
Expert feedback
All seven of the experts gave responses relevant to
the aims of the study in the first round. The service
users generated 42 statements in six categories
and the registered nurses generated 60 statements
in five categories (Table 2). In the feedback, one
nurse commented that the ‘questions enabled me to
express my views and beliefs – I was happy with
the questions and the space for my thoughts’. This
confirmed that the question was open (Powell 2003)
and unambiguous (Streiner and Norman 2008).
The pilot feedback gave reassurance: ‘The
questionnaire and instructions are clear’ [service
user]; the ‘pack is easy to follow – no major
problems’ [registered nurse]. Following analysis
of the first round, we made minor amendments
to the instructions in the questionnaire pack
to encourage experts to list their ideas rather
than write essay-style responses: the essay-style
responses required deeper analysis than the
reduction and categorisation usually associated
with the first round of Delphi research (Mead
and Moseley 2001), resulting in some of the
experts’ language being lost during analysis.
Experts are thought to be more motivated to
remain part of the study if they can recognise
their contributions to the process (McKenna 1994,
Baker et al 2007) and listed responses helped with
this more readily.
Second and third pilot study rounds The second-
and third-round questionnaires asked the experts to
rate and rank the statements generated in the first
round. Despite the frequent use of Likert scales in
Delphi research (Beech 1997, Pelletier et al 1997,
Gibson 1998, Hasson et al 2000, Kennedy 2000), this
approach has been criticised by Mead and Moseley
(2001): ‘The unthinking use of simple Likert scales
may well be sub-optimal.’ The pilot study provided
an opportunity to explore alternative approaches to
the measures used.
Likert scales, where a nominal measurement
approach is adopted, are problematic when
January 2012 | Volume 19 | Number 2 © RCN PUBLISHING / NURSE RESEARCHER
42
Nurse Researcher
Delphi method
meaningful statistical feedback has to be presented
to the experts as part of an iterative questionnaire.
Transforming nominal codes to numeric feedback
could result in the experts failing to recognise
their contributions or derive meaning from the
feedback (Okoli and Pawlowski 2004, Rowe et al
2005), thus reducing the rigour of the iteration.
Crawford et al (2004) simplified this process by
adopting a numerical presentation that used a
seven-point ordinal scale. Mead and Moseley (2001)
suggested that visual analogue scales (VASs) could
offer more robust statistical measures and could
improve the rigour of the findings, although few
Delphi researchers have adopted this approach
(Wengstrom and Häggmark 1998, Hennessy and
Hicks 2003) (Figure 3).
Measurement approaches
Two measurement approaches were piloted in round
two: a 100mm VAS and a seven-point ordinal scale
(Figure 3). The pressure to keep the questionnaires
to a manageable size meant we piloted the first
half of the first-round questionnaire using a VAS
and the second half using the ordinal scale. As a
result of this, we could not make direct statistical
comparisons between the VAS and the ordinal scale.
The pilot test of the measures was therefore limited
to testing the practical application and analysis of
the two measures in the Delphi process. Future pilot
studies should consider direct comparison between
measures to identify biases in the measures and
indicate the validity of the measurement approach
(Oppenheim 1992, Streiner and Norman 2008).
The service users found the second and third
round questionnaires difficult to complete.
Feedback included statements such as: ‘I find
scale-type questions difficult to answer’ and ‘third
questionnaire: more difficult to understand what
to do’. One of the nurses also commented that,
‘Ranking is difficult as I felt a lot of these items were
very important and struggled to rank only ten’. The
second questionnaire took significantly longer to
complete for two of the service users and one service
user dropped out of the pilot study during round
three (Table 3), emphasising the need for clarity
in the presentation of the questionnaires and
the instructions.
Despite the nurses’ preference for the VAS,
it was rejected because it was complex and time
consuming to administer, analyse and feed back.
The pilot study also demonstrated that the VAS
did not offer greater depth of analysis than the
ordinal scale, as suggested by McCabe and Stevens
(2004): ‘It is not obvious that VAS offers anything
that is not provided by the ranking exercise.’ The
service users preferred the ordinal scale and it
was easy to administer, analyse and feedback.
Given the difficulties described by the service users,
we chose the scale preferred by them to increase
the likelihood of retaining service users in the
full study.
Duffield (1993) proposed that using stability
of scores across rounds of Delphi research is
more rigorous than using solely percentages. This
requires the identification of an acceptable level
of change between Delphi rounds and Dajani et al
(1979) suggested that this decision should not be
arbitary. There are a number of ways to define
acceptable levels of change by using the level of
dispersion or applying an empirical rule (Walters
2009). Fiander and Burns (2000) for example,
applied a limit to changes in the measure of
Figure 3 Example of the different types of scale and response styles
Visual analogue scale
Benefits Disadvantages
Ratio level
statistics.
Quick and simple
to complete.
Difficult to produce.
Time consuming to analyse.
Not often used in Delphi research.
Seven-point ordinal scale
Benefits Disadvantages
Easy to produce
and interpret.
Well-understood.
Limited to descriptive statistics.
For example: ‘Nurses should prioritise making time to spend with clients.’
For example: ‘Nurses should prioritise making time to spend with clients.’
Not at all
important
Not at all
important
Extremely
important
Extremely
important
X
1 2 3 4 5 6 7
√
Despite the nurses’ preference for the visual
analogue scale it was rejected because it
was complex and time consuming
43
© RCN PUBLISHING / NURSE RESEARCHER January 2012 | Volume 19 | Number 2
Nurse Researcher
Delphi method
Ashmore R (2008) Nurses’ accounts of
locked ward doors: ghosts of the asylum or
acute care in the 21st century? Journal of
Psychiatric and Mental Health Nursing. 15,
3, 175-185.
Baker J, Lovell K, Harris N (2006) How expert
are the experts? An exploration of the concept
of ‘expert’ within Delphi panel techniques.
Nurse Researcher. 14, 1, 59-70.
Baker JA, Lovell K, Harris N (2007) Mental
health professionals’ pro re nata (p.r.n)
medication practices in acute inpatient mental
health care: a qualitative study. General Hospital
Psychiatry. 29, 2, 163-168.
Beech B (1997) Studying the future: a Delphi
survey of how multi-disciplinary clinical staff
view the likely development of two community
mental health centres over the course of the
next two years. Journal of Advanced Nursing.
25, 2, 331-338.
Beech B (1999) Go the extra mile – use
the Delphi Technique. Journal of Nursing
Management. 7, 5, 281-288.
Biondo PD, Nekolaichuk CL, Stiles C et al
(2008) Applying the Delphi process to palliative
care tool development: lessons learned.
Supportive Care in Cancer. 16, 8, 935-942.
Boje D, Murnighan K (1982) Group confidence
pressures in iterative decisions. Management
Science. 28, 10, 1187-1196.
Buchanan-Barker P, Barker P (2005)
Observation: the original sin of mental health
nursing? Journal of Psychiatric and Mental
Health Nursing. 12, 5, 541-549.
Butterworth T (1994) Working In Partnership: A
Collaborative Approach To Care. The Review of
Mental Health Nursing. HMSO, London.
Butterworth T, Bishop V (1995) Identifying the
characteristics of optimum practice: findings
from a survey of practice experts in nursing,
midwifery and health visiting. Journal of
Advanced Nursing. 22, 1, 24-32.
Campbell P (1999) The future of the mental
health system: a survivor’s perspective. Mental
Health Practice. 3, 1, 12-17.
Care Quality Commission (2009) Mental Health
Acute Inpatient Service Users Survey. National
Centre for Social Research, London.
Cramer CK, Klasser GD, Epstein JB et al (2008)
The Delphi process in dental research. The
Journal of Evidence-Based Dental Practice. 8, 4,
211-220.
Crawford I, Mackway-Jones K, Russell D et al
(2004) Delphi based consensus study into
planning for chemical incidents. Emergency
Medicine Journal. 21, 1, 24-28.
Crisp J, Pelletier D, Duffield C et al (1997)
The Delphi method? Nursing Research. 46, 2,
116-118.
Dajani JS, Sincoff MZ, Talley WK (1979) Stability
and agreement criteria for the termination of
Delphi studies. Technological Forecasting and
Social Change. 13, 1, 83-90.
Department of Health (2001) The Expert
Patient: A New Approach To Chronic Disease
Management For The 21st Century. HMSO,
London.
Department of Health (2002) Mental Health
Policy Implementation Guide: Adult Acute
Inpatient Care Provision. HMSO, London.
Department of Health (2006) From Values To
Action: The Chief Nursing Officer’s Review of
Mental Health Nursing. HMSO, London.
Dodds P, Bowles N (2001) Dismantling formal
observation and refocusing nursing activity in
acute inpatient psychiatry: a case study. Journal
of Psychiatric and Mental Health Nursing. 8, 2,
183-188.
Duffield C (1993) The Delphi technique: a
comparison of results obtained using two expert
panels. International Journal of Nursing Studies.
30, 3, 227-237.
Dyck MJ (2008) Weight loss prevention in
nursing home residents: a pilot study to
determine administrative strategies. Journal of
Gerontological Nursing. 34, 1, 28-35.
Faulkner A (1997) Knowing Our Own Minds: A
Survey of How People In Emotional Distress Take
Control of Their Lives. Mental Health Foundation,
London.
Fiander M, Burns T (2000) A Delphi approach
to describing service models of community
mental health practice. Psychiatric Services. 51,
5, 656-658.
Table 4 Consensus thresholds – pilot study
Seven-point ordinal scale 100mm visual analogue scale
Consensus is
achieved when:
Inter-quartile range (IQR) =
1.0 or less.
IQR=15 or less.
Stability is
achieved when:
The IQR has changed
between pilot rounds two
(PR2) and three (PR3) by
less than or equal to 0.5 in
either direction.
IQR changes between PR2
and PR3 by less than or equal
to 7.0 in either direction.
Convergence is
observed when:
The IQR has changed
between PR2 and PR3
towards zero by more than
0.5.
The IQR has changed between
PR2 and PR3 towards zero by
more than 7.0.
Divergence is
observed when:
The IQR has changed
between PR2 and PR3 away
from zero by more than 0.5.
The IQR has changed between
PR2 and PR3 away from zero
by more than 7.0.
Importance is
defined as:
Very high
importance
Median 7.0 Median 90-100
High importance Median 6.5-6.9 Median 80-89
Moderate
importance
Median 6.0-6.4 Median 65-79
Low importance Median 5.0-5.9 Median 50-64
Very low
importance
Median below 5.0 Median below 50
dispersion (IQR) of below 2 (on a five point scale).
A small IQR relative to the length of the scale used
to measure the outcome data would indicate that
most of the responses are clustered around a small
number of points on the scale. Observing the extent
of change in the IQR across rounds is therefore
one way to define stability and was adopted here
(Table 4).
Following the pilot study analysis, we noted
that we retained six statements in round
three when they did not meet the criteria (Table 4).
Of these statements, three converged towards
consensus and higher importance and the remaining
three were stable across rounds in not reaching
consensus. On reflection, it appeared that the
reduction of statements before iteration removed
the opportunity for the experts to reconsider
their scores in light of the controlled feedback.
Given the importance of observing consensus
(Greatorex and Dexter 2000, Hasson et al 2000)
and stability across rounds (Dajani et al 1979), we
decided not to carry out reduction of statements
in the second round of the full study to ensure
that all statements were subjected to iteration.
This could, however, have resulted in the iterative
questionnaire being large thereby risking sample
attrition so we approached it with caution (Mead
and Moseley 2001).
Conclusion
Pilot Delphi studies are rarely reported in the
academic literature, making it difficult to establish
best practice in this area. Here, all three rounds of
the Delphi were piloted in advance of undertaking
References
January 2012 | Volume 19 | Number 2 © RCN PUBLISHING / NURSE RESEARCHER
44
Nurse Researcher
Conflict of interest
None declared
Peer review
This article has been subject
to double-blind review and
has been checked using
antiplagiarism software
Forchuck C, Reynolds W (2001) Clients’
reflections on relationships with nurses:
comparisons from Canada and Scotland.
Journal of Psychiatric and Mental Health
Nursing. 8, 1, 45-51.
Gabb J, Balen R, Gibbs GR et al (2006) Victoria
Climbié inquiry Data Corpus project: using
the Delphi method in multidisciplinary child
protection. British Journal of Social Work. 36,
4, 577-596.
Garcia I, Kennett C, Quraishi M et al (2005)
Acute Care 2004: A National Survey Of Adult
Psychiatric Wards In England. Sainsbury Centre
for Mental Health CMH London.
Gibson J (1998) Using the Delphi Technique
to identify the content and context of nurses’
continuing professional development needs.
Journal of Clinical Nursing . 7, 5, 451-459.
Gilburt H, Rose D, Slade M (2008) The
importance of relationships in mental health
care: a qualitative study of service users’
experiences of psychiatric hospital admission
in the UK. BMC Health Services Research.
8, 92-103.
Gijbels H (1995) Mental health nursing skills in
an acute admission environment: perceptions
of mental health nurses and other mental
health professionals. Journal of Advanced
Nursing. 21, 3, 460-465.
Greatorex J, Dexter T (2000) An accessible
analytical approach for investigating what
happens between the rounds of a Delphi study.
Journal of Advanced Nursing. 32, 4, 1016-1024.
Hardy DJ, O’Brien AP, Gaskin CJ et al (2004)
Practical application of the Delphi technique
in a bicultural mental health nursing study in
New Zealand. Journal of Advanced Nursing. 46,
1, 95-109.
Hassan TB, Barnett DB (2002) Delphi type
methodology to develop consensus on the
future design of EMS systems in the United
Kingdom. Emergency Medicine Journal. 19, 2,
155-159.
Hasson F, Keeney S, McKenna H (2000)
Research guidelines for the Delphi survey
technique. Journal of Advanced Nursing. 32, 4,
1008-1015.
Hennessy D, Hicks C (2003) The ideal
attributes of chief nurses in Europe: a Delphi
study. Journal of Advanced Nursing. 43, 5,
441-448.
Hui A, Stickley T (2007) Mental health policy
and mental health service user perspectives on
involvement: a discourse analysis. Journal of
Advanced Nursing. 59, 4, 416-426.
Hung HL, Altschuld JW, Lee YF (2008)
Methodological and conceptual issues
confronting a cross-country Delphi study of
educational program evaluation. Evaluation and
Program Planning. 31, 2, 191-198.
Involve (2009) A Guide To Reimbursing and
Paying Members of The Public Who Are Actively
Involved In Research: For Researchers And
Research Commissioners, (Who May Also Be
People Who Use Services). Involve, London.
Jairath N, Weinstein J (1994) The Delphi
methodology: a useful administrative approach.
Canadian Journal of Nursing Administration.
7, 29-42.
Keeney S, Hasson F, McKenna HP (2001) A
critical review of the Delphi technique as a
research methodology for nursing. International
Journal of Nursing Studies. 38, 2, 195-200.
Kennedy HP (2000) A model of exemplary
midwifery practice: results of a Delphi study.
Journal of Midwifery & Women’s Health. 45,
1, 4-19.
Linstone HA, Turoff M (Eds) (1975) The Delphi
Method: Techniques and Applications.
Addison-Wesley, London.
Maxwell JA (2002) Understanding and validity
in qualitative research. In Huberman AM,
Miles MB (Eds) The Qualitative Researcher’s
Companion. Sage Publications, Thousand Oaks
CA.
McCabe C, Stevens K (2004) Visual
Analogue Scales: Do They Have A Role In The
Measurement of Preferences For Health States?
http://tinyurl.com/d5prgga (Last accessed:
December 15 2011.)
McKenna HP (1994) The Delphi technique: a
worthwhile research approach for nursing?
Journal of Advanced Nursing. 19, 6, 1221-1225.
Mead D, Moseley L (2001) The use
of the Delphi as a research approach.
Nurse Researcher. 8, 4, 2-23.
MIND (2004) Ward Watch. MIND Publications,
London.
Moseley L, Mead D (2001) Considerations in
using the Delphi approach: design, questions
and answers. Nurse Researcher. 8, 4, 24-37.
Okoli C, Pawlowski S (2004) The Delphi
method as a research tool: an example, design
considerations and applications. Information &
Management. 42, 1, 15-29.
Oppenheim AN (1992) Questionnaire Design,
Interviewing and Attitude Measurement. Second
edition. Continuum International Publishing,
London.
Pelletier D, Duffield C, Adams A et al (1997)
The cardiac nurse’s role: an Australian Delphi
study perspective. Clinical Nurse Specialist.
11, 6, 255-263.
Powell C (2003) The Delphi Technique: myths
and realities. Journal of Advanced Nursing. 41,
4, 376-382.
Quinn B, Sullivan SJ (2000) The identification
by physiotherapists of the physical problems
resulting from a mild traumatic brain injury.
Brain Injury. 14, 12, 1063-1076.
Rae M (2007) Review of Open Doors In Acute
In-patient wards. Care Services Improvement
Partnership and National Institute for Mental
Health in England, London.
Rowe G, Wright G, Bolger F (1991) Delphi
a re-evaluation of research and theory.
Technological Forecasting and Social Change.
39, 3, 235-251.
Rowe G, Wright G, McColl A (2005) Judgement
change during Delphi-like procedures: the
role of majority influence, expertise, and
confidence. Technological Forecasting and
Social Change. 72, 377-399.
Sainsbury Centre for Mental Health (1998)
Acute Problems: A Survey of The Quality
of Care In Acute Psychiatric Wards. SCMH,
London.
Sainsbury Centre for Mental Health (2006)
The Search For Acute Solutions: Improving The
Quality of Care In Acute Psychiatric Wards.
SCMH, London.
Standing Nursing and Midwifery Committee
(1999) Mental health nursing – Addressing
Acute Concerns: Report by the Standing Nursing
and Midwifery Committee. HMSO, London.
Star Wards (2005) Star Wards – Inspiring
Inpatient Care. www.starwards.org.uk (Last
accessed: December 17 2011.)
Streiner DL, Norman GR (2008) Health
Measurement Scales: A Practical Guide To Their
Development And Use. Fourth edition. Oxford
University Press, New York NY.
Thompson M (2009) Considering the
implication of variations within Delphi
research. Family Practice. 26, 5, 420-424.
Valdez AM (2009) So much to learn, so little
time: educational priorities for the future of
emergency nursing. Advanced Emergency
Nursing Journal. 31, 4, 337-353.
Walker L, Barker P, Pearson P (2000) The
required role of the psychiatric-mental health
nurse in primary health-care: an augmented
Delphi study. Nursing Inquiry. 7, 2, 91-102.
Walters S (2009) Quality of Life Outcomes In
Clinical Trials and Health-Care Evaluation: A
Practical Guide To Analysis And Interpretation.
Wiley, Chichester.
Wengstrom Y, Häggmark C (1998) Assessing
nursing problems of importance for the
development of nursing care in a radiation
therapy department. Cancer Nursing. 21, 1,
50-56.
Ziglio E (1996) The Delphi method and its
contribution to decision making. In Adler M,
Ziglio E (Eds) Gazing into the Oracle: The Delphi
Method and Its Application To Social Policy and
Public Health. Jessica Kingsley, London.
Online archive
For related information, visit our online archive of more
than 6,000 articles and search using the keywords
References
the full study. This approach enabled measurement
approaches across the Delphi process to be
trialled but it delayed the recruitment to the
full study, which may not always be practical. The
sampling and recruitment process should reflect
the full study in the pilot to identify potential
recruitment difficulties.
Where pilot tests have been conducted and
reported, they are most often limited to the first
round. While the importance of testing the first
round’s questions cannot be underestimated, our
experiences support the use of pilot tests beyond
the first round because of the complex nature of the
subsequent rounds in the Delphi method.
Piloting the second- and third-round measures
in this study enabled the experts to ‘vote’ for
a preferred measurement approach. This was
particularly important for the service users, who
reported difficulty completing VAS scales. One
limitation of the pilot tests conducted here was
the inability to make direct statistical comparisons
between measurements.
The pilot of the analysis in rounds two and
three enabled consensus thresholds to be developed
and tested in advance of the full study. This process
highlighted the importance of controlled feedback
and the iterative round in establishing consensus
and stability of response.
