Content uploaded by Dr Sharon Mickan
Author content
All content in this area was uploaded by Dr Sharon Mickan on Sep 04, 2018
Content may be subject to copyright.
Reviews: Systematic Reviews & Meta-analyses
Validation of descriptive clinical pathway
criteria in the systematic identification
of publications in emergency medicine
Raffi Adjemian
1,2
, Atbin Moradi Zirkohi
2
, Robin Coombs
2
,
Sharon Mickan
1,3
and Christian Vaillancourt
4
Abstract
Background: Heterogeneity in both the definition and terminology of clinical pathways presents a challenge to the
systematic identification of primary studies for review purposes. Recently developed clinical pathway identification
criteria may facilitate both the identification and assessment of clinical pathway studies. The goal of this publication is
the validation of these five criteria in a descriptive systematic review of actively implemented clinical pathway studies in
the emergency department setting. The main outcome measure is the inter-rater agreement of investigators using the
clinical pathway criteria.
Methods: We performed a systematic literature search from 2006 to 2015 using MEDLINE, EMBASE, CENTRAL,
and CINAHL. All types of prospective trial designs were eligible. We identified relevant publications using the
above-mentioned clinical pathway criteria. Two reviewers independently collected data using a piloted data
abstraction tool.
Results: We identified 5947 publications, with 472 potentially relevant full text publications retrieved. Of these, 357 did
not meet preliminary study inclusion criteria, leaving 115 publications where the clinical pathway criteria were applied.
Ultimately, 44 publications were included. The inter-rater agreement of the criteria was very good (k¼0.81, 95%
Confidence Interval ¼0.70–0.92). The vast majority of studies were excluded because the intervention did not meet
the criterion of being multidisciplinary in nature.
Conclusion: These criteria are a useful instrument to reliably identify clinical pathway publications for systematic review
purposes in an emergency department setting. Future modification of these criteria may improve their usefulness.
Particular attention should be placed on clarifying what is meant by multidisciplinary involvement within the context of
clinical pathway interventions, with specific emphasis placed on delineating the level of involvement of each discipline and
their decision-making responsibility.
Keywords
Clinical pathways, critical pathways, complex interventions, reporting guidelines, methodology
Introduction
Clinical pathways (CPWs) are a type of complex inter-
vention. The term ‘‘clinical pathway’’ was first coined in
1985 at the New England Medical Center (Boston, MA,
USA), primarily as a cost containment strategy.
1,2
By the late 1990s, more than 80% of hospitals in the
USA used CPWs as care strategies for at least some
groups of patients.
3
Since then they have become a
widely used care strategy in the UK, Europe,
Australia, Canada, and Asia.
1,2,4,5
They have the poten-
tial to improve and streamline patient care, especially
in the complicated environment of the emergency
department (ED).
1,6,7
Two main features of CPWs set
them apart from other complex interventions.
1,8–15
1
University of Oxford, UK
2
McGill University, Canada
3
Griffith University, Australia
4
University of Ottawa, Canada
Corresponding author:
Raffi Adjemian, Department of Family Medicine, McGill University,
1110 Pine Avenue West, Room 105, Montreal, QC, Canada H3A 1A3.
Email: raffi.adjemian@mcgill.ca
International Journal of Care
Coordination
2017, Vol. 20(1–2) 45–56
!The Author(s) 2017
Reprints and permissions:
sagepub.co.uk/journalsPermissions.nav
DOI: 10.1177/2053434517707971
journals.sagepub.com/home/icp
First, they take evidence from general recommenda-
tions, such as clinical practice guidelines, and they
transform these into structured local clinical practices.
Second, these interventions utilize a multidisciplinary
approach to patient care.
Although the term clinical pathway may be one of
the most commonly used terms to describe the concept
of CPWs, at least 17 separate terms have been identified
as synonymous with CPW. These include terms such as
critical pathway, care map, and integrated care path-
way.
1,14
While developing the protocol for a systematic
review, Kinsman et al.
12
observed that the existing
CPW definitions were not specific enough to guide a
systematic literature search strategy, and lacked the
precision to maximize the inter-rater agreement of
CPW study selection.
16
For example, the European
Pathway Association (EPA) definition may be con-
sidered adequate for use in health care service delivery,
but its use in research applications has been ques-
tioned.
12,13,16
For this reason, Kinsman et al.
12
devel-
oped a set of criteria derived from an assessment of the
CPW literature and communication with the EPA. The
criteria were validated in a systematic review that exam-
ined the effects of CPWs on professional practices,
patient outcomes, lengths of stay, and hospital
costs.
16
Included studies were restricted to controlled
clinical trials (CCTs) and interrupted time series
(ITS). The group found that the use of these criteria
facilitated both the identification and assessment of
CPW studies despite the heterogeneity of terms used
to denote a CPW.
Although these criteria are an important step in
improving our ability to identify primary CPW publi-
cations for systematic review purposes, there were a
few potential limitations. The derivation phase of the
criteria included pilot testing of 10 publications to
refine the criteria, ultimately resulting in a set of
weighted criteria.
12
However, these publications
were not referenced, and it is unclear if they were
part of the full text publication set of the validation
phase.
16
Furthermore, the authors state that there
was a ‘‘high level of agreement’’ between reviewers
applying the criteria in the validation phase.
However, there is some ambiguity as to whether this
agreement pertained to the initial coding of reviewers,
or to unresolved disagreements that needed a third
reviewer to resolve.
The goal of this publication is to validate these new
CPW criteria in a descriptive systematic review of
actively implemented CPW studies in the ED setting,
including both controlled and uncontrolled study
designs, and to quantitatively assess their inter-rater
reliability.
Methods
Study design
We identified CPW interventions using criteria devel-
oped by Kinsman et al.
12
The criteria are as follows:
1. The intervention was a structured multidisciplinary
plan of care.
2. The intervention was used to channel the translation
of guidelines or evidence into local structures.
3. The intervention detailed the steps in a course of
treatment or care in a plan, pathway, algorithm,
guideline, protocol, or other inventory of actions.
4. The intervention had time frames of criteria-based
progression (i.e., steps were taken if designated cri-
teria were met).
5. The intervention aimed to standardize care for a spe-
cific clinical problem, procedure or episode of care in
a specific population.
These criteria have been weighted; for an interven-
tion to be considered a CPW, criterion 1 and any three
of the four remaining criteria have to be met. The cri-
teria did not precisely detail what was meant by multi-
disciplinary. Rotter et al.
16
indicated that the question
of what constitutes multidisciplinary involvement was
frequently raised during their review process (T Rotter,
personal communication, 2013). They placed more
emphasis on the inter-personal level of involvement in
the organization of care processes, an approach that
was also adopted by our group.
We considered interventions to be multidisciplinary
if members of at least two disciplines were intimately
involved in the CPW. This was determined by carefully
scrutinizing publications for indications of active
involvement of health care staff, including any pre-
implementation teaching that was received. Health
care workers solely performing passive tasks such as a
nurse carrying out a medication or test order, a radi-
ology technician performing a radiology request, a
respiratory technician carrying out a medication
order, or any other such actions were not considered
to be active involvement. We did not consider the
involvement of a consultant at the end-point of a path-
way to constitute an active role. We only considered the
involvement of imaging specialists (radiologist, nuclear
medicine specialist) as having an active role if they had
decision-making abilities that could alter the flow of a
CPW, and not if solely reporting the results of an ima-
ging test.
We included the following trial designs: randomized
controlled trials (RCTs), CCTs, controlled before-after
studies (CBA), ITS, prospective cohort studies, and
46 International Journal of Care Coordination 20(1–2)
prospective simple before-after studies (SBA). We
included both adult and pediatric patient groups, and
CPW interventions of any disease entity. Publications
where the CPW was part of a multifaceted intervention
were included only if the CPW was the main focus of
the study. We excluded the following: SBA studies
where both the before arm and after arm were retro-
spective, conference abstracts, doctoral theses. We also
excluded studies where the continuum of CPW care
involved the ED setting, but where the ED-related
CPW care was not the focus of the publication.
Search strategy
We searched MEDLINE, EMBASE, CENTRAL,
and CINAHL from 2006 to 2015. We used a broad
range of keywords in our search strategy, with free
text in addition to the controlled vocabulary terms of
individual databases. This is especially important
for CPWs where the terminology is inconsistent. The
search strategy was developed with the assistance of an
information specialist. Bibliographies of CPW system-
atic reviews were also searched for missed publications.
We restricted the language of publication to either
English or French.
17
Outcome measures
The main outcome measure was the inter-rater agree-
ment of investigators using the CPW criteria. Our sec-
ondary outcome was the terminology used by authors
to describe the CPW intervention.
Data extraction
Titles and abstracts of articles identified by the litera-
ture search were independently screened for eligibility
by two sets of trained reviewers (RA/RC and RA/AM),
who then applied CPW criteria to eligible full text pub-
lications. Data abstraction was done using a piloted
standardized data extraction tool. Disagreements
between reviewers were resolved by consensus, and
when necessary by a meeting of all investigators.
Data analysis
We used a descriptive approach to synthesize the results
of the systematic review. Almost all of the data points
were discrete quantities and are presented as raw num-
bers, proportions, or percentages. Inter-rater variation
was calculated using Cohen’s kappa coefficient with
95% confidence interval.
18
Statistical analysis was per-
formed using SPSS (version 20, IBM
TM
).
Assessment of bias
We assessed RCTs, CCTs, ITS, and CBA studies with
the Cochrane Effective Practice and Organization of
Care Group Risk of Bias Tool.
19
However, since SBA
studies are essentially a comparison of two cohort
groups, we used the Newcastle-Ottawa Quality
Assessment Scale.
20
Results
Our search strategy identified 5947 publications. Of
these, we retrieved 472 potentially relevant full text
publications. The study flow diagram is presented in
Figure 1. Of the full text publications, 357 did not
meet basic study inclusion criteria. The CPW criteria
were applied to 115 studies with 44 publications being
included and 71 excluded.
21–64
Of the 71 publications excluded by using the CPW
criteria, all were excluded because they did not fulfill
criterion 1 (the intervention was a structured multidis-
ciplinary plan of care). There was never an instance
where a publication met inclusion for criterion 1 but
failed to meet three of the four remaining criteria
(Table 1(a)).
We initially examined 10% of the publication data
set in order to pilot test the screening criteria and to
develop a data abstraction tool. During this phase, it
became apparent that most full-text publications that
failed to meet criterion 1 did so because the interven-
tion lacked multidisciplinary involvement. The data
abstraction tool in the pilot phase was not structured
to distinguish between interventions having multidis-
ciplinary involvement and having a structured plan.
In the subsequent 90% of publications, we divided cri-
teria 1 into structured and multidisciplinary components
(Table 1(b)). The findings confirmed that the vast
majority of publications were excluded because the
intervention was not multidisciplinary in nature. Only
five studies were excluded because the study was both
not multidisciplinary and not structured, and none of
the studies were excluded solely because they were
unstructured. Of note, assessment of criteria 2–5
revealed that only criteria 2 caused disagreement
between reviewers (occurring in only three publications,
not shown in table).
Of the 115 studies evaluated by the CPW criteria,
reviewers noted that they found it more difficult to
apply criterion 1 in 13 studies (11%). None of these
publications reproduced the study intervention and
included relatively few details about the intervention.
Table 2 shows the inter-rater agreement of the CPW
in the 115 articles where they were applied. The
strength of agreement was very good.
Adjemian et al. 47
Table 3 describes the general characteristics of
included studies. The five most common clinical entities
studied were sepsis (30%), asthma (16%), stroke/tran-
sient ischemic attacks (14%), myocardial infarction
(7%), and head injury (5%). In total, 10 separate
terms were used to describe the CPW intervention.
The term protocol was the most commonly used term.
It was used as a unique term in 17 studies (39%) and as
a mixed term in 5 studies (11%). Notably, the term
clinical pathway was used infrequently. It was used as
a unique term in four studies and as a mixed term in
two studies. Two different terms were used to describe
the CPW intervention in eight studies. The authors
often interchanged these terms at random between the
title, abstract, figures, tables, and text. Most studies
were performed in adults (59%). The majority of
included studies were SBA (30 of 44, 68%). The
before group of these studies were retrospective in 17
and prospective in 13 studies. A prospective cohort
design was used in 20% of studies. There were only
four RCTs and one CBA study.
Tables 4 and 5 outline the assessment of risk of bias
for SBA and prospective cohort studies, respectively.
Since there were few RCTs and only one CBA study,
their risk of bias assessment is not shown. Overall, there
was a low to moderate risk of selection bias in both
types of studies. The comparability of cohorts’ section
in SBA studies was inadequate with very few groups
attempting to match groups and control confounding.
The risk of bias was comparatively higher in the out-
come assessment section in SBA studies with less than
one-third reporting data from medical charts in a
blinded fashion.
Discussion
The specific objective of this study was to validate a
recently developed set of CPW identification criteria
12
Figure 1. Review process flow diagram.
CPW = clinical pathway, ED = emergency department.
48 International Journal of Care Coordination 20(1–2)
for the systematic identification of CPW interventions
in the ED setting. We found the inter-rater agreement
between reviewers applying the criteria to be very good,
adding to the evidence of their efficacy.
18
Yet, from our
findings we offer several recommendations for future
refinement of these criteria.
We observed that once an ED intervention is deemed
multidisciplinary, the other CPW criteria are almost
always fulfilled. It is possible that the often-algorithmic
nature of emergency medicine is the reason for this.
Indeed 90% of the ED publications in the systematic
review of Rotter et al.
16
were excluded because they did
not fulfill criterion 1. However, even in their non-ED
publication set, the majority was also excluded because
of this reason. These observations corroborate the lit-
erature showing the paramount importance of multidis-
ciplinary involvement as a key distinguishing feature of
CPW interventions.
1,9–12
There is general acceptance that multidisciplinary
involvement in an intervention comprises of two or
more disciplines. Notably, it was only after we applied
the CPW criteria to the pilot set of publications that we
remarked that a member of a health care discipline
might participate in the intervention, yet their involve-
ment may be limited, not active, or not possess deci-
sion-making ability. However, we were unable to find a
reference that has systematically attempted to define or
clarify the meaning of multidisciplinary involvement in
the context of CPW interventions, particularly with
respect to the degree of involvement or decision-
making ability of disciplines. This is also true of the
‘‘sentinel’’ articles that were used to derive the CPW
criteria.
9–12
For example, De Bleser et al.
10
performed
a literature review in an attempt to define the concept of
a CPW. They organized the characteristics of CPWs
into 16 categories, and then attributed descriptive
terms to each category. The category of ‘‘multidiscip-
linary team’’ had the second most terms attributed to it
(36 different terms). However, no attempt was made to
amalgamate these terms into a unifying definition of
what was meant by the phrase within the context of
CPWs. Similarly, multidisciplinary involvement was
an important element in all seven of the CPW audit
tools evaluated in a systematic review by Vanhaecht
et al.,
11
but none explored the exact meaning of the
term. We had contacted the authors of the first system-
atic review to use these criteria, to detail what was
meant by multidisciplinary.
16
They indicated that the
question of what constitutes multidisciplinary involve-
ment was frequently raised during their review process
(T Rotter, personal communication, 2013). They stated
that they placed more emphasis on the inter-personal
level of involvement in the organization of care
processes.
The lack of clarity of multidisciplinary involvement
also extends to the concept of multidisciplinary teams
in other types of research. In a recent systematic review
of multidisciplinary team meetings in cancer and non-
cancer care, a major inclusion criteria specified that
multidisciplinary team meetings required a minimum
of two disciplines, and implied that each discipline
should have a decision-making ability
8
; an approach
similar to the one we used (see Introduction section).
Although a reference was not provided to support their
inclusion criteria, they stated that a weakness of the
literature they examined was the lack of a commonly
accepted definition of the term multidisciplinary team.
As a consequence of the discussion above, we feel
that what is meant by multidisciplinary in criterion 1
Table 1. Inclusion and exclusion of full-text articles by clinical
pathway (CPW) criteria.
a. All publications (N ¼115)
Studies included by CPW criteria 44
Fails to meet criterion 1 71
Meets criterion 1 but
fails to meet three of
four other criteria
0
b. Data separated into pilot and post pilot phases
Pilot phase
(N ¼20)
Post pilot phase
(N ¼95)
Included by CPW criteria 10 34
Fails to meet criterion 1 10 61
Not multidisciplinary N/A 56
Not a structured plan N/A 0
Neither multidisciplinary
or structured plan
NA 5
Meets criterion 1 but fails
to meet three of four
other criteria
00
Table 2. Inter-rater agreement of clinical pathway (CPW)
criteria.
Reviewer 1
Reviewer 2
Include
based on
criterion 1
Exclude
based on
criterion 1
Include based on
criterion 1
37 7
Exclude based
on criterion 1
368
115
k¼0.81 (CI 95%: 0.70–0.92).
Adjemian et al. 49
Table 3. Characteristics of clinical pathway publications in the emergency department setting (N ¼44).
Author Country Year
Study
design
Study
population Clinical entity
Terminology used to
describe intervention
Andrews et al.
56
Zambia 2014 RCT Adult Sepsis Protocol
Bekmezian et al.
64
USA 2015 SBA-rp Both Asthma Clinical pathway
Callegro et al.
21
Multinational 2009 SBA-rp Pediatric Febrile seizures Algorithm/care pathway
a
Calver et al.
22
Australia 2010 SBA-rp Adult Acute behavioral
disturbance
Protocol
Chen et al.
57
Taiwan 2014 SBA-pp Adult Stroke Protocol
Chern et al.
23
USA 2010 SBA-rp Pediatric CSF shunt malfunction Clinical pathway/protocol
a
Correia et al.
52
Brazil 2013 SBA-pp Adult Myocardial infarction Protocol
Crowe et al.
24
USA 2010 SBA-rp Adult Sepsis Protocol
Cruz et al.
25
USA 2011 SBA-rp Pediatric Sepsis Protocol
Dalcin et al.
26
Brazil 2007 SBA-pp Both Asthma Clinical pathway/protocol
a
Decostered et al.
27
Switzerland 2007 SBA-pp Adult Acute pain Guideline
Dexheimer et al.
58
USA 2014 RCT Pediatric Asthma Protocol
Ender et al.
59
USA 2014 SBA-pp Pediatric Sickle cell disease Clinical pathway
Fong et al.
28
Australia 2008 SBA-rp Adult Head injury Guideline
Geurts et al.
60
Netherlands 2014 SBA-pp Pediatric Urinary tract infection Guideline
Guse et al.
61
USA 2014 SBA-rp Both Syncope Guideline
Hoegerl et al.
29
USA 2011 SBA-rp Adult Stoke Protocol
Hyden and Fields
30
USA 2010 SBA-pp Adult Myocardial infarction Pathway
Jones et al.
31
USA 2007 SBA-pp Adult Sepsis Algorithm/protocol
a
Kim et al.
32
Korea 2008 SBA-rp Both Pyelonephritis Protocol
Lau et al.
47
Hong Kong 2010 Prospective
cohort
Adult Stroke Pathway
Li et al.
33
Hong Kong 2009 Prospective
cohort
Adult Urinary retention Protocol
Lougheed et al.
34
Canada 2009 CBA Adult Asthma Care pathway/care map
a
Mackey et al.
35
Canada 2007 SBA-pp Both Asthma Care map/protocol
a
MacRedmond et al.
36
Canada 2010 SBA-rp Adult Sepsis Protocol/algorithm
a
McCarthy et al.
53
Ireland 2013 SBA-rp Adult Chronic obstructive
pulmonary disease
Bundle
Mohd et al.
48
Malaysia 2010 Prospective
cohort
Adult Sepsis Protocol
Munoz et al.
37
USA 2011 SBA-rp Adult Hyperglycemia Protocol
Na et al.
49
Multinational 2012 Prospective
cohort
Adult Sepsis Bundle
Nguyen et al.
38
USA 2007 Prospective
cohort
Adult Sepsis Bundle
Norton et al.
39
Canada 2007 SBA-pp Pediatric Asthma Clinical pathway
Paul et al.
62
USA 2014 SBA-rp Pediatric Sepsis Bundle
Plambech et al.
50
Denmark 2012 SBA-rp Both Sepsis Guideline
Ratanalert et al.
40
Thailand 2007 Prospective
cohort
Both Head injury Guideline
Ross et al.
41
USA 2007 RCT Adult TIA Protocol
Russell et al.
54
USA 2013 SBA-rp Pediatric Appendicitis Pathway/guideline
a
Sairanen et al.
42
Finland 2011 Prospective
cohort
Adult Stoke Consultation
Santillanes et al.
51
USA 2012 Prospective
cohort
Pediatric Appendicitis Clinical practice
guideline
(continued)
50 International Journal of Care Coordination 20(1–2)
needs to be clarified, with specific emphasis placed on
delineating the level of involvement of each discipline,
and their decision-making responsibility.
We found that only criterion 2 (the intervention was
used to channel the translation of guidelines or evidence
into local structures) resulted in quantifiable disagree-
ment between reviewers. We believe this is due to ambi-
guity of whether the ‘‘guidelines or evidence’’ used by
the publication in question were required to be expli-
citly referenced or not. We propose to clarify whether
‘‘evidence’’ needs to be clearly referenced in the publi-
cation being evaluated.
We found that reviewer’s had some difficulty with
the interpretation of criterion 4, especially the phrase
‘‘time frames of criteria-based progression.’’ It was
unclear whether time was actually a criterion that
should require a clinician to progress forward in the
CPW. The original CPW criteria validation article did
not elaborate on this issue.
12
We propose that criterion
4 be rewritten as follows: the intervention had criteria-
based progression (criteria can be a clinical event,
diagnostic test, time, etc.).
Poor reporting of study interventions made it rela-
tively more difficult to apply the CPW criteria in
one-tenth of eligible studies. Once again, much of this
surrounded the concept of multidisciplinary involve-
ment. Some studies explicitly stated the multidisciplin-
ary nature of the intervention in the reproduced CPW.
For example, in an acute pain study, the fully repro-
duced CPW incorporated color-coded roles of phys-
icians and nurses.
27
However, in most studies the
roles of participants were not clear within the fully
reproduced CPW. For example, in a study of the diag-
nosis of myocardial infarction, the intimate involve-
ment of physicians was clear in the reproduced CPW,
but the involvement of nurses was only made clear in
the text explanation.
30
Certain studies explicitly stated
that the study was not multidisciplinary and thus
allowed the publication to be confidently excluded.
One such study of a physician only pharyngitis man-
agement intervention clearly stated, ‘‘nursing staff was
not targeted in our intervention.’’
65
It appears that
poor reporting of study interventions can be an impedi-
ment to the meaningful application of these criteria.
Adherence to the most recent reporting guidelines
may improve this.
66
The vast majority of our included studies were of
SBA or prospective cohort design, with an overall
risk of bias that was low to moderate. However, despite
using appropriate scales for assessing the risk of bias,
these scales do not specifically measure the adequacy of
reporting about the intervention itself.
19,20
Although
assessing the risk of bias of studies often focuses on
their design and conduct, poor or inadequate reporting
may represent an uncertain risk of bias.
67,68
Selection
bias may occur when intervention details, potential
inclusion and exclusion criteria for a systematic
review, are poorly reported.
69
In our case, missing
information about the roles and responsibilities of
the different disciplines involved in study inter-
ventions may have lead to studies being inadvert-
ently excluded and therefore contributing to selection
bias.
66
Our results corroborate that CPW terminology is
inconsistently used in the existing literature, and may
cause confusion when evaluating a study.
1,10,14,16
Among the 44 included publications, 10 separate
terms were used to describe the intervention, including
eight publications where two different terms were used
simultaneously and/or interchangeably. Furthermore
and perhaps compounding the confusion, the authors
often interchanged these two different terms at random
between the title, abstract, figures, tables, and text. Our
broad search strategy maximized the identification of
studies where the primary intervention could be con-
sidered a CPW, despite the wide variety of terms used
Table 3. Continued
Author Country Year
Study
design
Study
population Clinical entity
Terminology used to
describe intervention
Stead et al.
43
USA 2009 Prospective
cohort
Adult TIA Protocol
To et al.
44
Canada 2010 SBA-rp Pediatric Asthma Guideline
Tromp et al.
45
Netherlands 2010 SBA-pp Adult Sepsis Protocol
Wang et al.
55
China 2013 SBA-pp Adult Sepsis Protocol
Wilson et al.
46
Australia 2010 SBA-pp Adult Myocardial infarction Pathway
Yealy et al.
63
USA 2014 RCT Adult Sepsis Protocol
SBA (RP): simple before-and-after (SBA) studies where the intervention is prospective and the control is retrospective; SBA (PP): SBA studies where
both the intervention and the control phases are prospective; CBA: controlled before-and-after study; RCT: randomized control trial; CSF: cerebro-
spinal fluid; TIA: transient ischemic attacks.
a
These studies used two terms each.
Adjemian et al. 51
in the literature. However, as Rotter et al.
16
point out,
this increases the time and effort taken to identify rele-
vant studies for systematic review and makes it difficult
for clinicians and healthcare managers to appraise
CPW publications. For this reason, we feel that it is
essential to work towards having a standardized inter-
nationally accepted definition of what a clinical path-
way actually constitutes, and consensus on its
terminology. As mentioned above, we feel that particu-
lar attention needs to be given to what is meant by
multidisciplinary involvement.
Limitations
There are a few limitations to the study. First, at present,
a lack of definite consensus of what a clinical pathway
actually constitutes, and the heterogeneity of terms used
to denote CPWs, makes the systematic identification of
primary studies for review purposes challenging.
1,14
For
example, the strict multidisciplinary inclusion criteria
inherent in using these novel criteria resulted in many
ED publications being excluded. We took multidisciplin-
ary to mean intimate involvement at the inter-personal
Table 4. Assessment of bias in simple before-after studies (SBA) using the Newcastle-Ottawa scale (N ¼30).
Selection Comparability Outcome
Publication
Representativeness
of the exposed
cohort
Selection
of the non-
exposed
cohort
Ascertainment
of exposure
Demonstration
that outcome
of interest was
not present at
start of study
Comparability
of cohorts on
the basis of
the design
or analysis
Assessment
of outcome
Was
follow-up
long enough
for outcomes
to occur
Adequacy of
follow-up of
cohorts Total
Bekmezian et al.
64
***N/A*****8
Callegro et al.
21
***N/A–––**5
Calver et al.
22
***N/A–––**5
Chen et al.
57
***N/A–––**5
Chern et al.
23
***N/A–––**5
Correia et al.
52
***N/A–––**5
Crowe et al.
24
***N/A*–***7
Cruz et al.
25
––*N/A–––*–2
Dalcin et al.
26
***N/A–––**5
Decostered et al.
27
***N/A–––**5
Ender et al.
59
***N/A–––**5
Fong et al.
28
***N/A–––**5
Geurts et al.
60
***N/A–––**5
Guse et al.
61
***N/A––**–5
Hoegerl et al.
29
***N/A–––**5
Hyden and Fields
30
––*N/A–––*–2
Jones et al.
31
***N/A–––**5
Kim et al.
32
***N/A–––**5
Mackey et al.
35
***N/A–––**5
MacRedmond et al.
36
***N/A*––**6
McCarthy et al.
53
***N/A––***6
Munoz et al.
37
***N/A*––**6
Norton et al.
39
***N/A*****8
Paul et al.
62
***N/A–––**5
Plambech et al.
50
***N/A––***6
Russell et al.
54
***N/A––***6
To et al.
44
***N/A**–**7
Tromp et al.
45
***N/A–––**5
Wang et al.
55
***N/A–––**5
Wilson et al.
46
***N/A––***6
The scale had to be modified to fit the study type. ‘‘Demonstration that outcome of interest was not present at start of study’’ in the selection category
was not included (N/A) in the assessment of bias, because it is meant for epidemiological studies, and is not applicable to acute care illness studies.
20
Therefore, the maximum number of starred items is ¼8.
*It is used to allow a semi-quantitative assessment of study quality, such that the highest quality studies are awarded a maximum of one star for each
item with the exception of the item related to comparability that allows the assignment of two stars.
52 International Journal of Care Coordination 20(1–2)
level in the organization of care processes, in line with
the interpretation of Rotter et al.
16
(T Rotter, personal
communication, 2013). However, it is quite possible that
other investigators feel that the mere involvement of
healthcare workers in non-decision-making roles still
constitute multidisciplinary involvement. Furthermore,
most reporting guidelines of complex interventions do
not provide detailed direction on this matter.
66,70,71
Second, poor reporting of study interventions made it
relatively more difficult to apply the CPW criteria in 13 of
115 eligible publications (11%). For example, in an
excluded study on cellulitis, the authors stated that the
development of the intervention was multidisciplinary,
making it clear that physicians were intimately involved.
However, the authors provided no mention of the role of
nursing, and did not reproduce the intervention at all,
making it difficult to have a good idea of whether nurses
were intimately involved.
72
Difficulties in intervention
description were a problem encountered by Kinsman
et al.
12
during the initial validation of the CPW criteria.
19
They found that the main obstacle to agreement on all five
criteria was the poor reporting of the intervention compo-
nents, requiring adjustment by weighting the criteria.
As with any systematic review, our study was sus-
ceptible to publication bias. In the case of our review,
we did not examine the gray literature, particularly con-
ference abstracts and dissertations, and we only
included publications that were either in the English
or French language, both of which may have contrib-
uted to publication bias.
Conclusions
Our study suggests that these CPW criteria are a
useful instrument to reliably identify CPW interven-
tion publications in an ED setting. Poor reporting of
study interventions can be an impediment to the
meaningful application of these criteria, and adher-
ence to the most recent reporting guidelines may
improve this. We believe that future research and
further modification of these criteria should focus
on clarifying the meaning of multidisciplinary
involvement within the context of CPW interven-
tions, with specific emphasis placed on delineating
the level of involvement of each discipline and their
decision-making responsibility.
Acknowledgements
This publication was part of an MSc in Evidence Based
Health Care at the University of Oxford, Oxford, UK.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with
respect to the research, authorship, and/or publication of this
article.
Funding
The author(s) received no financial support for the research,
authorship, and/or publication of this article.
Table 5. Assessment of bias in prospective cohort studies using the Newcastle-Ottawa scale (N ¼9).
Selection Comparability Outcome
Publication
Representativeness
of the exposed
cohort
Selection
of the non-
exposed
cohort
Ascertainment
of exposure
Demonstration
that outcome
of interest
was not
present at
start of study
Comparability
of cohorts on
the basis of
the design
or analysis
Assessment
of outcome
Was
follow-up
long enough
for outcomes
to occur
Adequacy of
follow-up
of cohorts Total
Lau et al.
47
* N/A * N/A N/A N/A – * – 3
Li et al.
33
* N/A * N/A N/A N/A * * – 4
Mohd et al.
48
* N/A – N/A N/A N/A – * * 3
Na et al.
49
* N/A * N/A N/A N/A – * * 4
Nguyen et al.
38
* N/A * N/A N/A N/A * * * 5
Ratanalert et al.
40
* N/A * N/A N/A N/A – * * 3
Sairanen et al.
42
* N/A * N/A N/A N/A – * * 4
Santillanes et al.
51
* N/A * N/A N/A N/A * * – 4
Stead et al.
43
* N/A * N/A N/A N/A * * * 5
The scale had to be modified to fit the study type. ‘‘Demonstration that outcome of interest was not present at start of study’’ in the selection category
was not included (N/A) in the assessment of bias, because it is meant for epidemiological studies, and is not applicable to acute care illness studies.
20
Furthermore, since these are prospective studies without a control group, items related to a control group were marked as non-applicable (N/A).
Therefore, the maximum number of starred items is ¼5.
*It is used to allow a semi-quantitative assessment of study quality, such that the highest quality studies are awarded a maximum of one star for each
item with the exception of the item related to comparability that allows the assignment of two stars.
Adjemian et al. 53
References
1. Vanhaecht K. The impact of clinical pathways on the
organisation of care processes. Leuven: Katholieke
Universiteit Leuven, 2007, p.154.
2. Zander K, Etheredge ML and Bower KA. Nursing case
management: blueprints for transformation. Boston:
New England Medical Center Hospitals, 1987.
3. Pearson SD. Et tu, critical pathways? Am J Med 1999;
107: 397–398.
4. Vanhaecht K, Panella M, van Zelm R, et al. An overview
on the history and concept of care pathways as complex
interventions. Int J Care Pathw 2010; 14: 117–123.
5. Hindle D and Yazbeck AM. Clinical pathways in 17
European Union countries: a purposive survey. Aust
Health Rev 2005; 29: 94–104.
6. Jabbour M, Curran J, Scott SD, et al. Best strategies to
implement clinical pathways in an emergency department
setting: study protocol for a cluster randomized con-
trolled trial. Implement Sci 2013; 8: 55.
7. Yetter D. Critical pathways in the emergency depart-
ment. Nurs Manage 1995; 26: 60.
8. Ke KM, Blazeby JM, Strong S, et al. Are multidiscip-
linary teams in secondary care cost-effective? A system-
atic review of the literature. Cost Eff Resour Alloc 2013;
11: 7.
9. Campbell H, Hotchkiss R, Bradshaw N, et al. Integrated
care pathways. BMJ 1998; 316: 133–137.
10. De Bleser L, Depreitere R, De Waele K, et al. Defining
pathways. J Nurs Manag 2006; 14: 553–563.
11. Vanhaecht K, De Witte K, Depreitere R, et al. Clinical
pathway audit tools: a systematic review. J Nurs Manag
2006; 14: 529–537.
12. Kinsman L, Rotter T, James E, et al. What is a clinical
pathway? Development of a definition to inform the
debate. BMC Med 2010; 8: 31.
13. European Pathway Association. EPA, http://e-p-a.org/
(2008, accessed 29 July 2012).
14. de Luc K, Kitchiner D, Layton A, et al. Developing care
pathways – the handbook. Oxford: Radcliffe Medical
Press Ltd, 2001.
15. Roeder N, Hensen P, Hindle D, et al. Clinical pathways:
effective and efficient inpatient treatment. Chirurg 2003;
74: 1149–1155.
16. Rotter T, Kinsman L, James E, et al. Clinical pathways:
effects on professional practice, patient outcomes, length
of stay and hospital costs. Cochrane Database Syst Rev
2010; (3).
17. Morrison A, Polisena J, Husereau D, et al. The effect of
English-language restriction on systematic review-based
meta-analyses: a systematic review of empirical studies.
Int J Technol Assess Health Care 2012; 28: 138–144.
18. Landis JR and Koch GG. The measurement of observer
agreement for categorical data. Biometrics 1977; 33:
159–174.
19. Cochrane Effective Practice and Organization of Care
Group (EPOC). Suggested risk of bias criteria for
EPOC reviews, http://epoc.cochrane.org/sites/epoc.
cochrane.org/files/uploads/Suggested risk of bias criteria
for EPOC reviews.pdf (2009, accessed 20 March 2017).
20. Wells GA, Shea B, O’Connell D, et al. The Newcastle-
Ottawa Scale (NOS) for assessing the quality of nonran-
domised studies in meta-analyses, www.ohri.ca/pro-
grams/clinical_epidemiology/oxford.asp (2008, accessed
20 March 2017).
21. Callegaro S, Titomanlio L, Donega S, et al.
Implementation of a febrile seizure guideline in two pedi-
atric emergency departments. Pediatr Neurol 2009; 40:
78–83.
22. Calver LA, Downes MA, Page CB, et al. The impact of a
standardised intramuscular sedation protocol for acute
behavioural disturbance in the emergency department.
BMC Emerg Med 2010; 10: 14.
23. Chern JJ, Macias CG, Jea A, et al. Effectiveness of a
clinical pathway for patients with cerebrospinal fluid
shunt malfunction. J Neurosurg Pediatr 2010; 6: 318–324.
24. Crowe CA, Mistry CD, Rzechula K, et al. Evaluation of
a modified early goal-directed therapy protocol. Am J
Emerg Med 2010; 28: 689–693.
25. Cruz AT, Perry AM, Williams EA, et al. Implementation
of goal-directed therapy for children with suspected sepsis
in the emergency department. Pediatrics 2011; 127:
e758–e766.
26. Dalcin PdTR, da Rocha PM, Franciscatto E, et al. Effect
of clinical pathways on the management of acute asthma
in the emergency department: five years of evaluation.
J Asthma 2007; 44: 273–279.
27. Decosterd I, Hugli O, Tamches E, et al. Oligoanalgesia in
the emergency department: short-term beneficial effects
of an education program on acute pain. Ann Emerg
Med 2007; 50: 462–471.
28. Fong C, Chong W, Villaneuva E, et al. Implementation
of a guideline for computed tomography head imaging in
head injury: a prospective study. Emerg Med Australasia
2008; 20: 410–419.
29. Hoegerl C, Goldstein FJ and Sartorius J. Implementation
of a stroke alert protocol in the emergency department: a
pilot study. J Am Osteopath Assoc 2011; 111: 21–27.
30. Hyden R and Fields W. Improving the acute myocardial
infarction rapid rule out process. J Nurs Care Qual 2010;
25: 313–319.
31. Jones AE, Focht A, Horton JM, et al. Prospective exter-
nal validation of the clinical effectiveness of an emergency
department-based early goal-directed therapy protocol
for severe sepsis and septic shock. Chest 2007; 132:
425–432.
32. Kim K, Lee CC, Joong ER, et al. The effects of an insti-
tutional care map on the admission rates and medical
costs in women with acute pyelonephritis. Acad Emerg
Med 2008; 15: 319–323.
33. Li YK, Leung CS, Hui TL, et al. Acute urinary retention:
how useful is an ambulatory care protocol? Hong Kong J
Emerg Med 2009; 16: 134–140.
34. Lougheed MD, Olajos-Clow J, Szpiro K, et al.
Multicentre evaluation of an emergency department
asthma care pathway for adults. Can J Emerg Med
2009; 11: 214–229.
35. Mackey D, Myles M, Spooner CH, et al. Changing the
process of care and practice in acute asthma in the
54 International Journal of Care Coordination 20(1–2)
emergency department: experience with an asthma care
map in a regional hospital. Can J Emerg Med 2007; 9:
353–365.
36. MacRedmond R, Hollohan K, Stenstrom R, et al.
Introduction of a comprehensive management protocol
for severe sepsis is associated with sustained improve-
ments in timeliness of care and survival. Qual Saf
Health Care 2010; 19: e46.
37. Munoz C, Villanueva G, Fogg L, et al. Impact of a sub-
cutaneous insulin protocol in the emergency department:
Rush Emergency Department Hyperglycemia
Intervention (REDHI). J Emerg Med 2011; 40: 493–498.
38. Nguyen HB, Corbett SW, Steele R, et al. Implementation
of a bundle of quality indicators for the early manage-
ment of severe sepsis and septic shock is associated with
decreased mortality. Crit Care Med 2007; 35: 1105–1112.
39. Norton SP, Pusic MV, Taha F, et al. Effect of a clinical
pathway on the hospitalisation rates of children with
asthma: a prospective study. Arch Dis Child 2007; 92:
60–66.
40. Ratanalert S, Kornsilp T, Chintragoolpradub N, et al.
The impacts and outcomes of implementing head injury
guidelines: clinical experience in Thailand. Emerg Med J
2007; 24: 25–30.
41. Ross MA, Compton S, Medado P, et al. An emergency
department diagnostic protocol for patients with transi-
ent ischemic attack: a randomized controlled trial. Ann
Emerg Med 2007; 50: 109–119.
42. Sairanen T, Soinila S, Nikkanen M, et al. Two years of
Finnish Telestroke: thrombolysis at spokes equal to that
at the hub. Neurology 2011; 76: 1145–1152.
43. Stead LG, Bellolio MF, Suravaram S, et al. Evaluation of
transient ischemic attack in an emergency department
observation unit. Neurocrit Care 2009; 10: 204–208.
44. To T, Wang C, Dell SD, et al. Can an evidence-based
guideline reminder card improve asthma management in
the emergency department? Respir Med 2010; 104:
1263–1270.
45. Tromp M, Hulscher M, Bleeker-Rovers CP, et al. The
role of nurses in the recognition and treatment of patients
with sepsis in the emergency department: a prospective
before-and-after intervention study. Int J Nurs Stud 2010;
47: 1464–1473.
46. Willson AB, Mountain D, Jeffers JM, et al. Door-
to-balloon times are reduced in ST-elevation myocardial
infarction by emergency physician activation of the car-
diac catheterisation laboratory and immediate patient
transfer. Med J Aust 2010; 193: 207–212.
47. Lau AYL, Soo YOY, Graham CA, et al. An expedited
stroke triage pathway: the key to shortening the door-to-
needle time in delivery of thrombolysis. Hong Kong Med
J2010; 16: 455–462.
48. Mohd. Salleh F, Mohd. Fathil S, Ahmad Z, et al. Early
goal-directed therapy in the management of severe sepsis/
septic shock in an academic Emergency Department in
Malaysia. Crit Care Shock 2010; 13: 91–97.
49. Na S, Kuan WS, Mahadevan M, et al. Implementation of
early goal-directed therapy and the surviving sepsis cam-
paign resuscitation bundle in Asia. Int J Qual Health
Care 2012; 24: 452–462.
50. Plambech MZ, Lurie AI and Ipsen HL. Initial, successful
implementation of sepsis guidelines in an emergency
department. Dan Med J 2012; 59: A4545.
51. Santillanes G, Simms S, Gausche-Hill M, et al.
Prospective evaluation of a clinical practice guideline
for diagnosis of appendicitis in children. Acad Emerg
Med 2012; 19: 886–893.
52. Correia LC, Brito M, Kalil F, et al. Effectiveness of a
myocardial infarction protocol in reducing door-
to-ballon time. Arq Bras de Cardiol 2013; 101: 26–34.
53. McCarthy C, Brennan JR, Brown L, et al. Use of a care
bundle in the emergency department for acute exacerba-
tions of chronic obstructive pulmonary disease: a feasi-
bility study. Int J Chron Obstruct Pulmon Dis 2013; 8:
605–611.
54. Russell WS, Schuh AM, Hill JG, et al. Clinical practice
guidelines for pediatric appendicitis evaluation can
decrease computed tomography utilization while main-
taining diagnostic accuracy. Pediatr Emerg Care 2013;
29: 568–573.
55. Wang Z, Xiong Y, Schorr C, et al. Impact of sepsis
bundle strategy on outcomes of patients suffering from
severe sepsis and septic shock in China. J Emerg Med
2013; 44: 735–741.
56. Andrews B, Muchemwa L, Kelly P, et al. Simplified
severe sepsis protocol: a randomized controlled trial of
modified early goal-directed therapy in Zambia. Crit
Care Med 2014; 42: 2315–2324.
57. Chen CH, Tang SC, Tsai LK, et al. Stroke code improves
intravenous thrombolysis administration in acute ische-
mic stroke. PLoS One 2014; 9: e104862.
58. Dexheimer JW, Abramo TJ, Arnold DH, et al.
Implementation and evaluation of an integrated compu-
terized asthma management system in a pediatric emer-
gency department: a randomized clinical trial. Int J Med
Inform 2014; 83: 805–813.
59. Ender KL, Krajewski JA, Babineau J, et al. Use of a
clinical pathway to improve the acute management of
vaso-occlusive crisis pain in pediatric sickle cell disease.
Pediatr Blood Cancer 2014; 61: 693–696.
60. Geurts DH, Vos W, Moll HA, et al. Impact analysis of an
evidence-based guideline on diagnosis of urinary tract
infection in infants and young children with unexplained
fever. Eur J Pediatr 2014; 173: 463–468.
61. Guse SE, Neuman MI, O’Brien M, et al. Implementing a
guideline to improve management of syncope in the emer-
gency department. Pediatrics 2014; 134: e1413–e1421.
62. Paul R, Melendez E, Stack A, et al. Improving adherence
to PALS septic shock guidelines. Pediatrics 2014; 133:
e1358–e1366.
63. Yealy DM, Kellum JA, Huang DT, et al. A randomized
trial of protocol-based care for early septic shock. New
Engl J Med 2014; 370: 1683–1693.
64. Bekmezian A, Fee C and Weber E. Clinical pathway
improves pediatrics asthma management in the emer-
gency department and reduces admissions. J Asthma
2015; 52: 806–814.
65. Fakih MG, Berschback J, Juzych NS, et al. Compliance
of resident and staff physicians with IDSA guidelines for
Adjemian et al. 55
the diagnosis and treatment of streptococcal pharyngitis.
Infect Dis Clin Pract 2006; 14: 84–88.
66. Adjemian R. Are complex interventions adequately
described in the literature? Examining the quality of report-
ing of clinical pathway publications in emergency medicine.
Oxford: University of Oxford, 2014, p.125.
67. Higgins JPT and Green S (eds). Cochrane handbook for
systematic reviews of interventions, www.cochrane-hand-
book.org/ (2009, accessed 30 March 2017).
68. Viswanathan M, Ansari MT, Berkman ND, et al.
Assessing the Risk of Bias of Individual Studies in
Systematic Reviews of Health Care Interventions.
Agency for Healthcare Research and Quality Methods
Guide for Comparative Effectiveness Reviews. March
2012. AHRQ Publication No. 12-EHC047-EF.
Available at: www.effectivehealthcare.ahrq.gov/.
69. El Baz N, Middel B, van Dijk JP, et al. Are the outcomes
of clinical pathways evidence-based? A critical appraisal
of clinical pathway evaluation research. J Eval Clin Pract
2007; 13: 920–929.
70. Campbell M, Fitzpatrick R, Haines A, et al. Framework
for design and evaluation of complex interventions to
improve health. BMJ 2000; 321: 694–696.
71. Craig P, Dieppe P, Macintyre S, et al. Developing and
evaluating complex interventions: the new Medical
Research Council guidance. BMJ 2008; 337: a1655.
72. Campbell SG, Burton-MacLeod R and Howlett T. A cel-
lulitis guideline at a community hospital – we can reduce
costs by standardizing care. J Emerg Prim Health Care
2009; 7: 12.
56 International Journal of Care Coordination 20(1–2)
