The Cancer Patient Experiences Questionnaire (CPEQ): Reliability and construct validity following a national survey to assess hospital cancer care from the patient perspective

Abstract

Patient experience questionnaires have been criticised owing to the lack of supporting psychometric evidence. The objective of this study was to describe the development and psychometric evaluation of the Cancer Patient Experiences Questionnaire (CPEQ) in Norway.
Questionnaire development was based on a literature review of existing questionnaires, patient interviews, expert-group consultations, pretesting of questionnaire items and a national survey. Psychometric evaluation included exploratory and confirmatory factor analysis, and tests of internal consistency reliability and test-retest reliability.
Data were collected using a postal survey of cancer patients attending 54 hospitals in all 4 health regions. The subjects were 14 227 adult cancer patients who had attended an outpatient clinic or who had been discharged from an inpatient ward. Patients with all types of cancer were included. Data quality, internal consistency reliability and construct validity were assessed.
Of the 13 846 patients who received the CPEQ, 7212 (52%) responded. Exploratory factor analysis identified six scales of outpatient experiences relating to nurse contact, doctor contact, information, organisation, patient safety and contact with next of kin, and seven scales of inpatient experiences, with the addition of hospital standard to the aforementioned scales. All but two of the scales met the criterion of 0.70 for Cronbach's α testing, and test-retest correlations ranged from 0.57 to 0.85. Confirmatory factor analysis supported the interpretation of six and seven scales for outpatients and inpatients, respectively. Statistically significant associations based on explicit hypotheses provided evidence for the construct validity of the scales. One additional scale measuring the hospital level was identified (α=0.85).
The CPEQ is a self-report instrument that includes the most important aspects of patient experiences with cancer care at hospitals. The instrument was tested following a national survey in Norway; good evidence is provided herein for the internal consistency reliability, test-retest reliability and construct validity.

Full text

The Cancer Patient Experiences
Questionnaire (CPEQ): reliability and
construct validity following a national
survey to assess hospital cancer care
from the patient perspective
Hilde Hestad Iversen, Olaf Holmboe, Øyvind Andresen Bjertnæs
To cite: Iversen HH,
Holmboe O, Bjertnæs ØA.
The Cancer Patient
Experiences Questionnaire
(CPEQ): reliability and
construct validity following a
national survey to assess
hospital cancer care from the
patient perspective. BMJ
Open 2012;2:e001437.
doi:10.1136/bmjopen-2012-
001437
▸Prepublication history and
additional material for this
paper are available online. To
view these files please visit
the journal online (http://
dx.doi.org/10.1136/bmjopen-
2012-001437).
Received 7 May 2012
Accepted 28 August 2012
This final article is available
for use under the terms of
the Creative Commons
Attribution Non-Commercial
2.0 Licence; see
http://bmjopen.bmj.com
Department for Quality
Measurement and Patient
Safety, Norwegian Knowledge
Centre for the Health
Services, Oslo, Norway
Correspondence to
Dr Hilde Hestad Iversen;
hii@nokc.no
ABSTRACT
Objectives: Patient experience questionnaires have
been criticised owing to the lack of supporting
psychometric evidence. The objective of this study was
to describe the development and psychometric
evaluation of the Cancer Patient Experiences
Questionnaire (CPEQ) in Norway.
Design: Questionnaire development was based on a
literature review of existing questionnaires, patient
interviews, expert-group consultations, pretesting of
questionnaire items and a national survey.
Psychometric evaluation included exploratory and
confirmatory factor analysis, and tests of internal
consistency reliability and test–retest reliability.
Setting: Data were collected using a postal survey of
cancer patients attending 54 hospitals in all 4 health
regions. The subjects were 14 227 adult cancer patients
who had attended an outpatient clinic or who had been
discharged from an inpatient ward. Patients with all types
of cancer were included. Data quality, internal consistency
reliability and construct validity were assessed.
Results: Of the 13 846 patients who received the CPEQ,
7212 (52%) responded. Exploratory factor analysis
identified six scales of outpatient experiences relating to
nurse contact, doctor contact, information, organisation,
patient safety and contact with next of kin, and seven
scales of inpatient experiences, with the addition of
hospital standard to the aforementioned scales. All but
two of the scales met the criterion of 0.70 for Cronbach’s
αtesting, and test–retest correlations ranged from 0.57 to
0.85. Confirmatory factor analysis supported the
interpretation of six and seven scales for outpatients and
inpatients, respectively. Statistically significant
associations based on explicit hypotheses provided
evidence for the construct validity of the scales. One
additional scale measuring the hospital level was
identified (α=0.85).
Conclusions: The CPEQ is a self-report instrument that
includes the most important aspects of patient
experiences with cancer care at hospitals. The instrument
was tested following a national survey in Norway; good
evidence is provided herein for the internal consistency
reliability, test–retest reliability and construct validity.
INTRODUCTION
The measurement of patient experiences is
now recognised as an important part of
healthcare performance evaluation. Patient
experiences are included as one of three
core quality dimensions in the Organisation
for Economic Cooperation and Development
quality indicator framework,
1
and are central
ARTICLE SUMMARY
Article focus
▪To describe the development and evaluation of the
Cancer Patient Experiences Questionnaire (CPEQ).
▪Data quality, internal consistency reliability and
construct validity were assessed.
Key messages
▪Patient satisfaction questionnaires have generally
been criticised due to the lack of supporting psy-
chometric evidence, including reliability and val-
idity data.
▪Valid and reliable measures for the measurement
of cancer patients’experiences with hospitals are
crucial if the data are to be used in quality
improvement, business control and by both
patients and the general public.
Strengths and limitations of this study
▪The strengths of the study include the psycho-
metric assessment of the CPEQ following a
national survey, including data quality, dimen-
sionality, internal consistency and construct
validity.
▪The scale could prove useful for evaluating
cancer patients’experiences with hospitals in
Norway and in similar settings in other countries,
and includes the most important aspects regard-
ing both inpatient and outpatient hospital care
from the patient perspective.
▪Further investigation of the explanatory charac-
teristics of variations in cancer patients’experi-
ence with healthcare is warranted.
Iversen HH, Holmboe O, Bjertnæs ØA. BMJ Open 2012;2:e001437. doi:10.1136/bmjopen-2012-001437 1
Open Access Research

to the framework of the WHO for assessing the perform-
ance of health systems.
2
Several studies have shown that
patient experiences are positively related to patient satis-
faction
34
and that patient satisfaction is positively asso-
ciated with compliance
5
and health outcomes.
4
Eliciting
feedback from patients helps healthcare providers to
identify potential areas for improvement, which in turn
can increase the quality of healthcare.
The acquisition of valid information requires patient
experiences to be measured using rigorously developed
and validated tools. Patient satisfaction questionnaires
have generally been criticised because of the lack of sup-
porting psychometric evidence, including reliability and
validity data. A review of 195 patient satisfaction papers
published in 1994 found that the satisfaction instru-
ments presented little evidence of reliability or validity,
6
casting doubt on the credibility of the resultant findings.
Furthermore, questionnaires that ask patients to rate
their care in terms of how satisfied they are tend to elicit
very positive ratings.
4
A more valid approach is to ask
patients to report in detail on their experiences by
asking them specific questions about certain processes
and events; this will provide results that can be easily
interpreted and acted upon.
7
Norway has a national patient experience survey pro-
gramme that is run by the Norwegian Knowledge Centre
for the Health Services. The purpose of the programme
is to systematically measure user experiences with health-
care, to provide data for quality improvement, business
control, hospital choice and public accountability.
Because of the aforementioned problems that arise
when attempts are made to measure satisfaction, the
instruments focus on experiences of the healthcare
delivery, not satisfaction. In 2007, the Ministry of Health
in Norway decided to measure cancer patients’experi-
ences with hospitals. The Norwegian Knowledge Centre
for the Health Services has developed and validated
several generic instruments to be used in national
patient experience surveys,
8–19
but none specifically for
cancer patients. Ensuring the sensitivity of the instru-
ment and detecting changes that are significant for the
patient requires instruments specific to the study object-
ive as well as the population of interest to be devel-
oped.
20
A review of the literature was conducted to
determine whether there was an instrument that
included multidimensional scales including different
aspects of patient with experiences. Two national surveys
of patients’experiences of cancer services have been
carried out in the UK: the first in 1999–2000 by the
National Patient Survey Programme
21
and the second in
2004 by the National Audit Office.
22
However, these
surveys included only six types of cancer: breast, colorec-
tal, lung, ovarian, prostate and non-Hodgkin’s lymph-
oma. Other questionnaires identified in the review
typically addressed quality of life, or involved specific
cancer types, treatments, services or clinical staff; no
measures appropriate for all cancer types, including
assessment of inpatient and outpatient cancer care, were
identified. Consequently, the literature review identified
shortcomings with existing measures, and it was decided
to develop a new instrument to measure the experiences
of hospital care of both inpatients and outpatients that
could be used in a future national survey in Norway.
This paper describes the development and evaluation
of the Cancer Patient Experiences Questionnaire
(CPEQ). This tool was designed for application in a
national survey of adult cancer patients, whereby the
results are published in the form of reports for the
public, the government and individual healthcare units
as a basis for national surveillance, quality improvement
and hospital choice.
METHODS
Questionnaire development
The questionnaire was developed based on the findings
of a literature review of questionnaires aimed to deter-
mine cancer patients’experiences with hospitals, semi-
structured interviews with 13 cancer patients, and focus
groups of experts including clinicians and representa-
tives for patient with cancer organisations. This process
was designed to ensure content validity and that the
questionnaire addresses important aspects of patient
with cancer experiences of care, and to establish a
model for including participants in the survey. The
development of the questionnaire also followed previous
work in the identification of domains and items of rele-
vance for the patients.
8–19
We tested the questionnaire
by performing cognitive interviews with 12 cancer
patients and a pilot study of 953 cancer patients at a
single hospital. The cognitive interviews and the pilot
study suggested minor changes to the questionnaire,
which were discussed within the group of experts before
finalisation.
We asked the patients about their overall experiences
with a specific hospital. The questionnaire was divided
into separate sections for inpatients and outpatients,
consisting mainly of the same questions, except for six
questions included in the inpatient section about hos-
pital standards and waiting time. The questionnaire also
included items concerning aspects of care irrespective of
outpatient or inpatient settings, as well as sociodemo-
graphic and health-status questions. The final question-
naire comprised 127 items. Items relating to experiences
of care had a five-point scale that range from ‘not at all’
(1) to ‘a very large extent’(5). Negative items were
reverse-coded, and a higher score represent a better
experience for all items. Scales were transformed to
scores ranging from 0 to 100 where 100 is the best pos-
sible. In this study, we enquired about experiences that
do not apply to all patients but which are considered
important to the relevant patients. Consequently, the
levels of missing data were expected to reflect patients
being at different stages of cancer treatment and having
varying levels of experience with hospital care.
2Iversen HH, Holmboe O, Bjertnæs ØA. BMJ Open 2012;2:e001437. doi:10.1136/bmjopen-2012-001437
CPEQ: reliability and construct

Data collection
The questionnaire was mailed to 14 227 patients with all
types of cancer aged 16 years or older who had at least
one outpatient appointment or a hospital stay at a
Norwegian hospital from 20 September to 10 December
2009. A maximum of 400 patients from each of the 54
hospitals was included; 400 patients were selected ran-
domly from hospitals with more than 400 eligible
patients. Power calculations have been conducted at the
hospital level, resulting in an appropriate sample size of
400 for each hospital. When presenting population
estimates, individual weights are used to adjust for non-
response. These are also related to the size of the hos-
pital populations so that results at higher levels (health
enterprise, regions and the nation) take into account
differences in hospital size. The hospitals had the oppor-
tunity to exclude patients who were not considered
applicable to participate in a survey, and the patients
themselves could withdraw from the survey at any time
by contacting the hospital. Checks were undertaken to
ensure that patients were not sent more than one ques-
tionnaire, and the first hospital episode with a cancer
registration was taken as the episode to use in the survey
sample for patients who appeared on the lists more than
once. Units for palliative care and birth and maternity
departments were excluded from the survey. The ques-
tionnaires were mailed to the patients by the end of
January to mid-May 2010. Non-responders were sent one
reminder after 3 weeks. We also mailed a retest question-
naire to 291 consenting patients approximately 1 week
after their first reply for the purpose of assessing test–
retest reliability. After the completion of data collection,
the patient contact information was deleted and the hos-
pitals transferred the demographic, clinical and adminis-
trative data, which were merged with the patients’
responses.
The survey was approved by the Norwegian Regional
Committee for Medical Research Ethics, the Data
Inspectorate and the Ministry of Health and Care Services.
Statistical analysis
Most statistical analyses were carried out using SPSS
V.15.0. Confirmatory factor analyses (CFAs) were con-
ducted using the linear structural relation (LISREL)
analysis programme. STREAMS (Structural Equation
Modelling Made Simple) offers an interface to the
LISREL programme and was used as a support.
23
Separate exploratory factor analyses (EFAs) were
carried out for outpatients and inpatients using principal
axis factoring. We expected some correlations among the
factors and chose oblique rotation (promax). The criter-
ion for the number of factors to be rotated was eigenva-
lues greater than 1, and items with factor loadings lower
than 0.4 were excluded.
Cronbach’sαcoefficient was calculated to determine
the internal consistency of the scales. An αvalue greater
than 0.70 is considered satisfactory.
24 25
Internal consist-
ency was also assessed by item-total correlation,
measuring the strength of the association between an
individual item and the total score of the remainder of
its scale. A correlation coefficient of 0.30 or higher has
been recommended previously.
26
The intraclass correl-
ation coefficient was used as an estimate of the test–
retest reliability, and was used to assess the correlation
between scores obtained at different times for each
scale; the estimated coefficients should exceed 0.7.
24
The item structure from EFAs was used to construct
theoretically derived scales. CFA was applied to further
test the relationship between the observed variables and
their underlying latent constructs identified from the
EFAs. The LISREL analysis programme was used to test
the goodness of fit of the models
27
using various fit
indexes, including the root mean square error of
approximation (RMSEA), the goodness-of-fit index
(GFI), the comparative fit index (CFI) and the incre-
mental fit index (IFI). An RMSEA of 0.05 or less, and a
GFI and CFI of 0.90 or above are generally taken to indi-
cate a good fit. The IFI values range from 0 to 1, with
larger values indicating a better goodness of fit.
Construct validity refers to whether a scale measures
or is correlated with the theorised scientific construct,
and was assessed by exploring the associations of scale
scores as well as their associations with additional items
not included in the scales. A systematic review identified
139 articles that provided evidence about determinants
of satisfaction,
4
and concluded that satisfaction was
linked to prior satisfaction, predisposition, utilisation,
granting of desires, health status, health outcomes and
age. Evidence about the effects of gender, ethnicity,
socioeconomic status and expectations was equivocal.
Following the literature review and previous findings,
4–17
it was hypothesised that scale scores would be correlated
with global satisfaction, age, gender and health status.
Time since cancer diagnosis and whether the treatment
provided was considered the best possible were also
hypothesised as potential determinants.
RESULTS
Data collection
Of the 14 227 questionnaires that were mailed to cancer
patients, 137 were returned because of erroneous
addresses, and 244 of the patients had died. Accordingly,
the adjusted total sample was 13 846 patients, of which
7212 responded (52%), 6642 patients with experiences
from outpatient wards and 4856 patients from inpatient
wards; 4460 patients had attended both inpatient and
outpatient wards at the hospital. Table 1 lists the charac-
teristics of the respondents.
Data quality and psychometric evaluation
The levels of missing data and descriptive statistics for
the items are presented in table 2. The levels of missing
data ranged from 2% to 12%, which can be considered
as acceptable for most items. Responses in the category
‘not applicable’ranged from 2% to 33%. However, since
Iversen HH, Holmboe O, Bjertnæs ØA. BMJ Open 2012;2:e001437. doi:10.1136/bmjopen-2012-001437 3
CPEQ: reliability and construct

the questionnaire was intended to apply to all cancer
patients, rates of not applicable or missing data higher
than 10% were accepted. Mean scores for single items
were generally skewed towards a positive rating (table 2).
The mean scores were lowest (at 3.2 for both outpatients
and inpatients) for information about pain and pain
alleviation. The mean scores were highest for items
about patient safety, and ranged from 4.5 to 4.7.
Table 1 Patient characteristics
Variable Outpatients (N=6642) Inpatients (N=4856) All respondents (N=7212) n (%)
Gender
Male 3733 (56) 2590 (53) 4049 (56)
Female 2908 (44) 2265 (47) 3162 (44)
Age in years (mean±SD) 66±12.2 66±12.2 66±12.1
Education level
Primary school 1742 (27) 1319 (28) 1947 (28)
High school 2408 (38) 1831 (39) 2608 (38)
University undergraduate 1425 (22) 983 (21) 1519 (22)
University postgraduate 838 (13) 557 (12) 884 (13)
Native language
Norwegian 6324 (97) 4628 (97) 6858 (97)
Sami 7 (0) 6 (0) 7 (0)
Other Nordic 59 (1) 35 (1) 64 (1)
Other European 92 (1) 70 (2) 100 (1)
Non-European 35 (1) 30 (1) 39 (1)
Main activity
Work 1556 (24) 1128 (24) 1670 (24)
Sick leave 1270 (20) 994 (21) 1369 (19)
Retired 3537 (54) 2516 (53) 3860 (55)
Education 33 (1) 27 (1) 35 (1)
Home worker 52 (1) 43 (1) 56 (1)
Unemployed 24 (0) 18 (0) 28 (0)
Other 50 (1) 43 (1) 56 (1)
Marital status
Married 4559 (71) 3302 (70) 4941 (71)
Cohabitant 529 (8) 409 (9) 579 (8)
Neither married nor cohabitant 1347 (21) 991 (21) 1457 (21)
Type of cancer (ICD-10 codes)
Breast 884 (13) 669 (14) 932 (13)
Female genitalia 452 (7) 363 (8) 489 (7)
Male genitalia 1449 (22) 822 (17) 1591 (22)
Skin 493 (7) 119 (3) 512 (7)
Respiratory organs 347 (5) 273 (6) 386 (5)
Urinary tract 697 (11) 641 (13) 751 (10)
Digestive organs 866 (13) 860 (18) 984 (14)
Blood 1031 (16) 728 (15) 1103 (15)
Other 422 (6) 380 (8) 463 (6)
Time since diagnosis
Less than 3 months 171 (3) 132 (3) 209 (3)
3–6 months 807 (12) 562 (12) 929 (13)
6–12 months 1192 (18) 840 (18) 1295 (18)
1–2 years 1205 (18) 883 (19) 1286 (18)
2–5 years 1740 (27) 1286 (27) 1864 (26)
More than 5 years 1424 (22) 1070 (22) 1498 (21)
Type of contact
Examination 3876 (58) 2747 (57) –
Surgery 1928 (29) 3189 (66) –
Radiotherapy 1082 (16) 510 (11) –
Chemotherapy 2345 (35) 1181 (24) –
Hormone therapy 663 (10) 220 (5) –
Control/follow-up 4456 (67) 1960 (40) –
Other 492 (7) 570 (12) –
Except where stated otherwise, data are n (%) values.
ICD-10, International Classification of Diseases, 10th Revision; n, number of responses received; N, total number.
4Iversen HH, Holmboe O, Bjertnæs ØA. BMJ Open 2012;2:e001437. doi:10.1136/bmjopen-2012-001437
CPEQ: reliability and construct

Table 2 Descriptive statistics,* factor loadings and internal consistency for outpatients and inpatients
Scale/item
Outpatients (N=6642) Inpatients (N=4856)
Missing
(%)
Does
not
apply
(%) Mean* SD
Factor
loadings
Cronbach’sα/
item-total
correlation
coefficient
Test–retest
reliability†
(n=229)
Missing
(%)
Does
not
apply
(%) Mean* SD
Factor
loadings
Cronbach’sα/
item-total
correlation
coefficient
Test–retest
reliability†
(n=229)
Nurse contact 79.8 0.92 0.73 76.8 0.93 0.83
1. Updated about
your treatment
5 9 4.2 0.74 0.63 0.72 5 5 4.1 0.77 0.66 0.77
2. Followed up on
side effects
6 29 4.0 0.94 0.64 0.73 6 24 4.0 0.85 0.75 0.78
3. Took your
concerns seriously
6 19 4.2 0.79 0.80 0.81 6 13 4.1 0.81 0.93 0.84
4. Cared for you 7 –4.4 0.70 0.88 0.79 5 –4.3 0.73 0.85 0.81
5. Provided enough
time for dialogue
8–3.9 0.96 0.70 0.75 5 –3.6 0.99 0.63 0.75
6. Was
understandable
7–4.3 0.70 0.72 0.73 5 –4.2 0.70 0.62 0.72
7. Was competent 8 –4.3 0.68 0.67 0.71 5 –4.2 0.72 0.60 0.74
Doctor contact 77.6 0.92 0.81 75.8 0.93 0.85
1.Updated about
your treatment
5 5 4.3 0.77 0.65 0.73 6 4 4.2 0.76 0.77 0.76
2. Followed up on
side effects
7 30 3.8 0.98 0.62 0.72 8 28 3.8 0.93 0.55 0.75
3. Took your
concerns seriously
6 15 4.0 0.87 0.83 0.83 7 14 4.0 0.85 0.82 0.84
4. Cared for you 6 –4.1 0.85 0.93 0.83 6 –4.0 0.86 0.94 0.84
5. Provided enough
time for dialogue
5–3.8 0.98 0.85 0.79 7 –3.6 1.01 0.82 0.79
6. Was
understandable
6–4.2 0.77 0.72 0.75 6 –4.1 0.77 0.74 0.73
7. Was competent 5 –4.4 0.72 0.66 0.72 6 –4.4 0.71 0.73 0.74
Information 67.0 0.92 0.79 65.3 0.94 0.78
1. Your illness 2 3 3.9 0.87 0.47 0.73 4 5 3.8 0.96 0.47 0.76
2. Treatment options 3 8 3.9 0.95 0.51 0.72 4 11 3.8 1.03 0.55 0.76
3. Examination and
test results
3 2 4.0 0.91 0.41 0.67 4 4 3.9 0.96 0.44 0.72
4. Side effects of
treatment
4 14 3.5 1.13 0.85 0.82 5 18 3.5 1.14 0.91 0.85
5. Effects of
treatment
5 13 3.7 1.03 0.82 0.83 6 14 3.6 1.05 0.86 0.86
6. Pain to be
expected
4 18 3.2 1.24 0.96 0.80 5 17 3.2 1.20 0.96 0.82
7. Pain relief 5 21 3.2 1.26 0.92 0.77 6 19 3.3 1.18 0.91 0.80
Organisation 71.4 0.78 0.77 69.0 0.82 0.85
1. Co-operation
between hospital
departments
7 21 3.8 0.91 –0.59 7 22 3.8 0.95 0.64
Continued
Iversen HH, Holmboe O, Bjertnæs ØA. BMJ Open 2012;2:e001437. doi:10.1136/bmjopen-2012-001437 5
CPEQ: reliability and construct

Table 2 Continued
Scale/item
Outpatients (N=6642) Inpatients (N=4856)
Missing
(%)
Does
not
apply
(%) Mean* SD
Factor
loadings
Cronbach’sα/
item-total
correlation
coefficient
Test–retest
reliability†
(n=229)
Missing
(%)
Does
not
apply
(%) Mean* SD
Factor
loadings
Cronbach’sα/
item-total
correlation
coefficient
Test–retest
reliability†
(n=229)
2. Staff collaboration 6 10 4.1 0.77 –0.64 6 8 4.0 0.78 0.71
3. Information
provided to the
correct person
8–4.0 0.90 –0.52 8 –3.9 0.88 0.65
4. Same group of
nurses
10 –3.6 1.14 –0.49 5 –3.4 1.04 0.55
5. Multiple doctors
involved was a
problem
6 26 3.8 1.18 –0.47 7 18 3.8 1.14 0.49
6. One doctor
responsible
6–3.8 1.24 –0.51 7 –3.7 1.23 0.55
Patient safety 90.6 0.61 0.57 90.5 0.67 0.62
1. Information
deliberately held
back
4–4.5 0.87 –0.57 0.44 5 –4.6 0.80 –0.64 0.49
2. Contradictory
information about
illness
6–4.7 0.74 –0.55 0.43 6 –4.6 0.79 –0.70 0.48
3. Perceived
incorrect treatment
5 11 4.7 0.81 –0.52 0.41 6 7 4.6 0.82 –0.66 0.47
Contact with next of
kin
80.9 0.87 0.65 79.9 0.85 0.81
1. Received next of
kin
6 33 4.3 0.80 –0.77 7 17 4.2 0.77 0.75 0.76
2. Arranged for next
of kin to be present
6 33 4.2 0.91 –0.77 7 22 4.1 0.87 0.88 0.78
3. Arranged for
interaction with
visitors
––––– – 6 10 4.2 0.75 0.67 0.64
Hospital standard ––– 71.3 0.74 0.75
1. Arranged for
interaction with
other patients
––––– – 6–3.9 0.85 0.45 0.50
2. Physical
environment
––––– – 12 –3.6 0.94 0.66 0.58
3. Food ––––– – 7–4.0 0.88 0.65 0.53
4. Cleanliness ––––– – 7–3.9 0.89 0.62 0.54
*Items are scored 1–5 and scales are scored 0–100, where a higher score represents a better experience.
†Intraclass correlation coefficient.
6Iversen HH, Holmboe O, Bjertnæs ØA. BMJ Open 2012;2:e001437. doi:10.1136/bmjopen-2012-001437
CPEQ: reliability and construct

Separate EFAs were conducted for outpatients and
inpatients. Results from the first factor analyses for both
outpatients and inpatients showed that most of the items
addressing organisation loaded on the first factor, which
related to nurse contact. For outpatients, this was also
the case for items about contact with next of kin. On
the basis of both empirical and theoretical assumptions,
summed rating scales were constructed for organisation
(outpatients and inpatients) and contact with next of
kin (outpatients).
Further analyses where these items were omitted pro-
duced four factors for outpatients and six for inpatients
(see table 2). All item–total correlation coefficients
exceeded the 0.4 criterion (range 0.41–0.86). The four
factors related to outpatients’experiences were nurse
contact, doctor contact, information and patient safety;
the αvalues ranged from 0.61 to 0.92. The four-factor
solution accounted for 62% of the total variance. The
same four factors were produced for inpatients, compris-
ing the equivalent items, as well as two other factors
(contact with next of kin and hospital standard); the α
values ranged from 0.67 to 0.94. These six factors
explained 63% of the total variance in the scores for
inpatients’experiences. Table 2 lists the factor loadings.
All factor loadings for the items exceeded 0.4, and no
item had a cross-loading ≥0.40.
αValues for 11 of the 13 scales satisfied the criterion
of 0.70 or higher (table 2). Cronbach’sαis sensitive to
the number of test items, and the two patient safety
scales that did not meet the criterion of 0.70 (0.614 and
0.667) had three test items each. Item–total correlation
coefficients showed that all items satisfied the stated cri-
terion of 0.40, and it was decided to keep the six scales
for measuring outpatient experiences and the seven
scales for measuring inpatient experiences. Of the 291
patients mailed a retest questionnaire, 229 responded
(79%). Table 2 lists the results of the test–retest correla-
tions for the 13 scales. The test–retest correlation coeffi-
cients ranged from 0.57 to 0.81 for outpatients and from
0.62 to 0.85 for inpatients.
CFAs were applied to examine the hypothesised factor
structures. The measurement part specifies the relation-
ship of the latent to the observed variables, and the
structural part of the models specifies the relationship
among the latent variables. First, the six-factor solution
of the outpatient experiences was tested, which revealed
that there was a satisfactory model fit to the data
(χ
2
=23559.96, p<0.001, df=458, RMSEA=0.091, GFI=0.81,
CFI=0.96 and IFI=0.96). The results are shown in figure 1.
The exogenous, latent variables were the six factors and
the endogenous, latent variable was outpatient experi-
ences, introducing a second-order analysis examining
the correlations among the first-order factors to
examine predictions on the endogenous variable.
Organisation was the strongest predictor (β=0.95), but
nurse contact (β=0.89), doctor contact (β=0.88) and
information (β=0.76) were also strongly associated with
the endogenous, latent variable. This was also the case
for contact with next of kin (β=0.61) and patient safety
(β=0.60). Second, the seven-factor solution of the
inpatient experiences was tested, which indicated a
slightly better fit(χ
2
=26697.60, p<0.001, df=621,
RMSEA=0.083, GFI=0.81, CFI=0.97 and IFI=0.97; figure 2).
The results were consistent with those of the previous
CFA; the inpatient experiences factor was strongly
affected by organisation (β=0.93), nurse contact
(β=0.89), doctor contact (β=0.89), information (β=0.77),
contact with next of kin (β=0.70) and hospital standard
(β=0.70). The prediction from patient safety was slightly
lower (β=0.56).
Table 3 gives the results of construct validity testing.
The correlations between the scale scores were all sig-
nificant at the p<0.001 level (range of correlation coeffi-
cients, 0.27–0.77). The correlations were weakest for the
patient safety scales and the other scales, and strongest
for organisation and doctor contact. The correlations
between scale scores and other single items were stron-
gest for the patients’confidence in that the treatment
received was the best possible, with the correlation coef-
ficients ranging from 0.43 to 0.70. Overall satisfaction
and overall experiences were moderately to strongly cor-
related to the scales (range of correlation coefficients,
0.24–0.59), with all correlations significant at the
p<0.001 level. A significant correlation was also found
between all of the scales and the patients’health status,
indicating that patients with poorer health scored sub-
stantially lower on all scales. Age and time since cancer
diagnosis were more weakly correlated with the scales.
Results from t tests showed significant but small differ-
ences between men and women on four scales. Relative
to women, male outpatients had significantly lower
scores on the contact with next of kin scale (t=−2.03,
p<0.05); however, male inpatients had higher scores for
nurse contact (t=2.79, p<0.01), organisation (t=3.78,
p<0.001) and hospital standard (t=6.84, p<0.001).
The questionnaire also included items about hospital
care not directly related to either outpatient or inpatient
wards, addressing the hospitals’extra-medical services,
co-operation with the primary doctor and other commu-
nity services, and information on how to manage future
problems. The items had high levels of responses in the
category ‘not applicable’(range 12–63) indicating that
they are not relevant to a high proportion of the respon-
dents. Table 4 indicates that the items have lower mean
scores than those included in the other factors, ranging
from 2.5 to 3.4. Factor analysis revealed a one-factor
solution accounting for 57% of the variance of the
scores (table 4). The factor loadings ranged from 0.56
to 0.79, the item–total correlation coefficients ranged
from 0.52 to 0.71 and the results from reliability analysis
yielded a Cronbach’sαof 0.85. The test–retest correl-
ation coefficient for this scale was 0.78.
The scale was not correlated significantly with time
since cancer diagnosis, but patients with poorer health
reported significantly more negative experiences, as did
older patients and women (see table 5). Results from
Iversen HH, Holmboe O, Bjertnæs ØA. BMJ Open 2012;2:e001437. doi:10.1136/bmjopen-2012-001437 7
CPEQ: reliability and construct

t tests showed that male patients reported significantly
more positive experiences than did female patients
(t=3.54, p<0.001).
DISCUSSION
This study was part of the first national survey under-
taken in Norway to assess cancer patients’experiences
Figure 1 Confirmatory factor
analysis model for outpatients.
8Iversen HH, Holmboe O, Bjertnæs ØA. BMJ Open 2012;2:e001437. doi:10.1136/bmjopen-2012-001437
CPEQ: reliability and construct

Figure 2 Confirmatory factor
analysis model for inpatients.
Iversen HH, Holmboe O, Bjertnæs ØA. BMJ Open 2012;2:e001437. doi:10.1136/bmjopen-2012-001437 9
CPEQ: reliability and construct

Table 3 Correlations
†
between scale scores and individual items for outpatients and inpatients
Scale/item
Outpatients (N=6642) Inpatients (N=4856)
Nurse
contact
Doctor
contact Information Organisation
Patient
safety
Contact
with next
of kin
Nurse
contact
Doctor
contact Information Organisation
Patient
safety
Contact
with next
of kin
Hospital
standard
Doctor contact 0.73 0.75
Information 0.64 0.68 0.68 0.71
Organisation 0.68 0.73 0.61 0.73 0.77 0.68
Patient safety 0.38 0.44 0.38 0.41 0.39 0.44 0.39 0.40
Contact with
next of kind
0.63 0.58 0.47 0.53 0.29 0.65 0.61 0.51 0.55 0.29
Hospital
standard
––– – –– 0.55 0.50 0.42 0.48 0.27 0.55
Overall
satisfaction with
service
0.51 0.51 0.51 0.48 0.39 0.36 0.59 0.56 0.52 0.55 0.40 0.44 0.43
Overall
experience with
service
0.44 0.43 0.45 0.44 0.24 0.35 0.52 0.45 0.46 0.48 0.26 0.41 0.39
Confident that
treatment
provided was
the best
possible
0.64 0.69 0.57 0.61 0.43 0.54 0.70 0.69 0.58 0.65 0.43 0.57 0.48
Health status −0.17 −0.21 −0.22 −0.16 −0.18 −0.10 −0.23 −0.22 −0.24 −0.22 −0.18 −0.13 −0.17
Age −0.04** 0.02
ns
−0.11 −0.01
ns
0.10 −0.00
ns
0.01
ns
0.02
ns
−0.09 −0.01
ns
0.11 0.01
ns
0.08
Time since
cancer
diagnosis
−0.02
ns
−0.01
ns
−0.01
ns
−0.04** −0.03
ns
−0.06** −0.04** −0.03* 0.01
ns
−0.05** −0.01
ns
−0.05** −0.00
ns
All correlations are significant at p<0.001 except for **p<0.01, *p<0.05 and
ns
p>0.05.
†Data are Spearman’s rank correlations (r).
10 Iversen HH, Holmboe O, Bjertnæs ØA. BMJ Open 2012;2:e001437. doi:10.1136/bmjopen-2012-001437
CPEQ: reliability and construct

with somatic hospitals. The development of the CPEQ
followed a review of the literature, interviews with cancer
patients and consultation with an expert group of pro-
fessionals and researchers. The resulting questionnaire
underwent a thorough process of piloting and testing
for data quality, reliability and construct validity, as
recommended for evaluating such questionnaires. The
CPEQ addresses broad domains of cancer-related care at
somatic hospitals, rather than focusing on specifictreat-
ments, cancer types or specific professionals involved in
care of the patients.
The results from the survey can be used as national
quality indicators in Norway and were designed to
inform patient choice and for quality improvement. The
CPEQ was designed specifically for use with cancer
patients attending somatic hospitals, and was assumed to
increase content validity from a patient perspective as
well as allowing hospital staff to investigate in detail the
extent to which their service meets the needs of their
patients. Questionnaires that assess specific aspects of
care allow the domains where patients have poorer
experiences to be identified and potentially improved.
Satisfactory evidence of internal consistency, test–retest
reliability and construct validity was obtained, indicating
that the CPEQ can be considered a high-quality instru-
ment. The results of the EFAs and tests of internal con-
sistency provided empirical support for the scales, and
confirmed that both outpatient and inpatient experi-
ences are multidimensional concepts. CFAs were sup-
portive of the structures suggested by EFAs. There is
evidence for construct validity of the questionnaire fol-
lowing the application of hypotheses based on previous
research findings and theory.
78–19
The results also
provide support for the longitudinal temporal stability of
the measure. High agreement between scores adminis-
tered approximately 1 week apart provided good evi-
dence of the test–retest reliability of the CPEQ.
Some limitations of the study should be considered. The
levels of missing data suggest that the measure is accept-
able to patients. However, some of the included items were
only relevant for some of the respondents. It may be pos-
sible to extract a shorter version of the CPEQ with fewer
questions without sacrificing the psychometric qualities of
the measure, but this task was beyond the scope of this
study. Another potential limitation is the response rate. In
general, postal surveys have lower response rates than
other data-collection modes.
4
Non-response bias occurs
when the main variables differ systematically between
respondents and non-respondents.
28
The response rate
(52%) means that almost half of the patients failed to
respond; however, it was relatively high compared with pre-
vious user-experience surveys carried out in Norway.
8–19
Findings from some of these surveys have shown that the
low response rates have not caused serious bias.
15 29–32
Table 4 Descriptive statistics,* factor loadings and internal consistency for all patients
Scale/item
All patients (N=7212)
Missing
(%)
Does not
apply (%) Mean* SD
Factor
loadings
Cronbach’sα/
item-total correlation
coefficient
Test–retest
reliability†
(n=229)
Extra-medical services,
information, co-operation
49.2 0.85 0.78
Hospital extra-medical
services
6 45 2.5 1.35 0.65 0.60
Hospital co-operation
with primary doctor
6 16 3.4 1.15 0.56 0.52
Hospital co-operation
with community services
7 63 2.8 1.37 0.64 0.60
Information about future
problems
5 12 3.2 1.22 0.74 0.65
Information about
managing potential
relapse
6 19 2.9 1.30 0.79 0.69
Information about
rehabilitation
6 40 2.5 1.31 0.79 0.71
*Items are scored 1–5 and scales are scored 0–100, where a higher score represents a better experience.
†Intraclass correlation coefficient.
Table 5 Correlations* between scale score and individual
items for all patients
Scale/item
All patients (N=7212)
Extra-medical services,
information, co-operation
Health status −0.23
Age −0.11
Time since cancer
diagnosis
−0.02
ns
*All correlations are significant at p<0.001 except for ns (p>0.05).
Iversen HH, Holmboe O, Bjertnæs ØA. BMJ Open 2012;2:e001437. doi:10.1136/bmjopen-2012-001437 11
CPEQ: reliability and construct

The findings from a Norwegian follow-up study involving a
hospital population showed that postal respondents and
non-respondents had almost the same scores.
32
These
studies indicate that non-response might be of less
concern, but uncertainty related to external validity means
that more research is needed on the effect of non-
response in patient experience surveys on cancer care and
that the main findings in this study should be replicated in
future studies.
Consistent with previous findings,
33–35
some skewing
towards positive assessment was identified. Whether this
reflects truly positive experiences or low expectations is
unknown.
36
As for any study based on self-reports, social
desirability bias and recall bias may also have affected
the results. Respondents may introduce bias in several
ways, for example, by giving socially desirable responses
as a result of cognitive consistency pressure (making
ratings congruent with their continuing use of the
service) and through acquiescent response sets (a ten-
dency always to agree or reply positively).
4
However,
respondents have been shown to give more positive and
socially desirable responses in interview surveys than in
self-administered surveys.
37
Moreover, it is assumed that
recall bias is less likely when asking about the overall
experience rather than about a specific visit or
hospitalisation.
Instead of developing a cancer-specific questionnaire,
one of the existing generic questionnaires could poten-
tially have been used in the national survey, such as the
Patient Experience Questionnaire.
8
This would have
reduced the resource requirements, and also has some
empirical support. One study compared the measure-
ment properties and the patients’evaluation of one
generic and two psychiatric-specific patient satisfaction
questionnaires in a sample of psychiatric patients. The
results indicated that no single instrument was superior
in either respect.
38
Another study identified 10 generic
core items covering major dimensions of experiences
that patients across a range of specialist healthcare ser-
vices report to be important.
39
A short, generic question-
naire might be expected to give a higher response rate
and better comparability than the CPEQ, but would not
be suited for the purpose of a national survey in Norway.
The purpose of the present study was a broad assess-
ment of hospital cancer care. Furthermore, content val-
idity is better for a cancer-specific questionnaire, since
all activities are directed against securing validity for
cancer patients, rather than to patients in general.
Naturally, a national survey with a narrower focus could
have used a generic and perhaps shorter questionnaire.
Results from the national patient experience survey
programme in Norway are used to develop quality indi-
cators presented both to the public and to the respon-
sible institutions. Public use includes an Internet site for
free hospital choice in Norway. Research has shown that
patients have difficulty in understanding quality informa-
tion,
40
and that ‘less is more’in this respect.
41
Therefore, an aggregated and overall measure of
experiences with the hospitals seems appropriate in the
context of presenting information to patients. Further
research is needed to determine how to construct a
composite score, including how to weight each of the
underlying subdimensions. More specific results are
called for when reporting information to health provi-
ders with the aim of evaluating and improving the
quality of care.
33
Consequently, aggregated scores on the
13 CPEQ subdimensions might be a useful supplement
when reporting results to the responsible hospitals.
Comparing the contribution at the organisational and
individual levels is relevant for comparisons of hospitals
based on patient evaluations. The approach for institu-
tional benchmarking in the national survey programme
involves developing an appropriate case-mix model and
correct for multiple comparisons in statistical testing.
Another emerging approach is to use multilevel analysis
to estimate the amount of variation in scores that can be
explained by levels above the individual level.
42
A previ-
ous study of patient experiences found that only a small
part of the variation is attributable to the organisational
level.
43
Future studies based on the CPEQ should
explore this topic further in order to elucidate the use-
fulness of the CPEQ as a basis for quality indicators at
the hospital level. This also includes research on
hospital-level reliability, which is based on the theory
that patients who are treated at the same hospital should
agree regarding their assessments of that hospital. The
larger the ratio of between-hospital to within-hospital
variation in the scores, and the larger the number of
respondents, the more precise will be the measurement
of differences between hospitals, and thus the greater
the reliability of the scores.
44
Patient-satisfaction questionnaires have been criticised
for insufficient knowledge of their reliability and validity
in psychometric testing.
3
The strengths of the present
study include the psychometric assessment of the CPEQ
following a national survey, including data quality,
dimensionality, internal consistency and construct valid-
ity. The scale should prove useful for evaluating cancer
patients’experiences with hospitals in Norway and in
similar settings in other countries, and includes the most
important aspects regarding both inpatient and out-
patient hospital care from the patient perspective.
CONCLUSIONS
Valid and reliable measures for the measurement of
cancer patients’experiences with hospitals are crucial if
the data are to be used in quality improvement, business
control and by both patients and the general public.
The CPEQ includes the most important aspects of
cancer patients’experiences with somatic hospitals,
from the perspective of the cancer patients themselves.
The present study has provided strong evidence for the
high data quality, internal consistency, content and con-
struct validity of this questionnaire. The questionnaire is
recommended for future applications designed to assess
12 Iversen HH, Holmboe O, Bjertnæs ØA. BMJ Open 2012;2:e001437. doi:10.1136/bmjopen-2012-001437
CPEQ: reliability and construct

cancer patients’experiences of both inpatient and out-
patient care in Norway and similar settings in other
countries. Further investigation of the explanatory char-
acteristics of variations in cancer patients’experience
with healthcare is warranted.
Acknowledgements The authors thank Tomislav Dimoski at the Norwegian
Knowledge Centre for Health Services, Oslo, Norway, for developing the
software necessary for gathering data from the hospitals, and for conducting
the data collection and quality assurance of data. We also thank Saga
Høgheim and Marit Skarpaas, who carried out most of the practical tasks
relating to data collection, as well as members of the expert group that
contributed to the development of the questionnaire and survey design.
Contributors All authors contributed to the design of the questionnaire and
survey. HHI conducted the analysis and drafted the manuscript. HHI and OH
were involved in the data acquisition. All authors have made significant
contributions by critically reviewing the manuscript and have read and
approved the final version.
Funding The survey was funded by the Norwegian Ministry of Health.
Competing interests None.
Ethics approval The Norwegian Regional Committee for Medical Research
Ethics, the Data Inspectorate, and the Ministry of Health and Care Services.
Provenance and peer review Not commissioned; externally peer reviewed.
Data Sharing Statement The data from the current study are available to
specific researchers at the Norwegian Knowledge Centre for Health Services.
REFERENCES
1. Kelly E, Hurst J. Health care quality indicators project: conceptual
framework paper. OECD Health Working Papers 23-2006. Paris:
OECD, 2006.
2. Murray CJ, Frenk J. A framework for assessing the performance of
health systems. Bull World Health Organ 2000;78:717–31.
3. Bjertnæs ØA, Sjetne IS, Iversen HH. Overall patient satisfaction with
hospitals: effects of patient-reported experiences and fulfilment of
expectations. BMJ Qual Saf 2012;21:39–46.
4. Crow R, Gage H, Hampson S, et al. The measurement of
satisfaction with healthcare: implications for practice from a
systematic review of the literature. Health Technol Assess
2002;6:1–244.
5. Barbosa CD, Balp MM, Kulich K, et al. A literature review to explore
the link between treatment satisfaction and adherence, compliance,
and persistence. Patient Prefer Adherence 2012;6:39–48.
6. Sitzia J. How valid and reliable are patient satisfaction data? An
analysis of 195 studies. Int J Qual Health Care 1999;11:319–28.
7. Cleary P, Edgman-Levitan S, McMullen W, et al. The relationship
between reported problems and summary evaluations of hospital
care. QRB Qual Rev Bull 1992;18:53–9.
8. Pettersen KI, Veenstra M, Guldvog B, et al. The Patient Experiences
Questionnaire: development, validity and reliability. Int J Qual Health
Care 2004;16:453–63.
9. Garratt AM, Bjertnæs ØA, Krogstad U, et al. The Out Patient
Experiences Questionnaire (OPEQ): data quality, reliability and
validity in patients attending 52 Norwegian hospitals. Qual Saf
Health Care 2005;14:433–7.
10. Garratt AM, Danielsen K, Bjertnæs ØA, et al. PIPEQ—a method for
measurement of user satisfaction in mental health services. Tidsskr
Nor Laegeforen 2006;126:1478–80.
11. Garratt AM, Bjørngaard JH, Dahle KA, et al. The Psychiatric
Out-Patient Experiences Questionnaire (POPEQ): data quality,
reliability, and validity in patients attending 90 Norwegian clinics.
Nord J Psychiatry 2006;60:89–96.
12. Garratt AM, Bjertnæs ØA, Barlinn J. Parent experiences of
paediatric care (PEPC) questionnaire: reliability and validity following
a national survey. Acta Paediatr 2007;96:246–52.
13. Bjertnæs ØA, Garratt AM, Nessa J. The GPs’Experiences
Questionnaire (GPEQ): reliability and validity following a national
survey to assess GPs’views of district psychiatric services. Fam
Pract 2007;24:336–42.
14. Bjertnæs ØA, Garratt A, Ruud T, et al. The General Practitioner
Experiences Questionnaire (GPEQ): validity and reliability
following the inclusion of new accessibility items. Fa m Pract
2010;27:513–19.
15. Garratt AG, Bjertnæs ØA, Holmboe O, et al. Parent Experiences
Questionnaire for Child and Adolescent Mental Health Services
(PEQ-CAMHS): reliability and validity following a national survey.
Child Adolesc Psychiatry Ment Health 2011;5:18.
16. Garratt AM, Danielsen K, Forland O, et al. The Patient Experiences
Questionnaire for Out-of-Hours Care (PEQ-OHC): data quality,
reliability, and validity. Scand J Prim Health Care 2010;28:95–101.
17. Oltedal S, Garratt A, Bjertnæs ØA, et al. The NORPEQ patient
experiences questionnaire: data quality, internal consistency and
validity following a Norwegian inpatient survey. Scand J Public
Health 2007;35:540–7.
18. Bjertnæs ØA, Lyngstad I, Malterud K, et al. The Norwegian
EUROPEP questionnaire for patient evaluation of general practice:
data quality, reliability and construct validity. Fam Pract
2011;28:342–9.
19. Olsen RV, Garratt AM, Iversen HH, et al. Rasch analysis of the
Psychiatric Out-Patient Experiences Questionnaire (POPEQ). BMC
Health Serv Res 2010;10:282.
20. Brédart A, Morvan E, Savignoni A, et al. Respiratory Gated
Radiotherapy Study Group Stic-Rar 2003. Patient’s perception of
care quality during radiotherapy sessions using respiratory gating
techniques: validation of a specific questionnaire. Cancer Invest
2011;29:145–52.
21. Department of Health (2002) National Survey: Cancer Patients.
http://www.dh.gov.uk/en/Publicationsandstatistics/PublishedSurvey/
NationalsurveyofNHSpatients/Nationalsurveycancer/index.htm
(accessed 19 Dec 2011).
22. National Audit Office. Tackling cancer: improving the patient journey.
London: Department of Health, The Stationery Office, 2005. http://
www.nao.org.uk/publications/0405/tackling_cancer.aspx (accessed
19 Dec 2011).
23. Gustafsson JE, Stahl PA. STREAMS User’s Guide. Version 2.5 for
Windows. Mölndal, Sweden: Multivariate Ware, 2000.
24. Nunnally JC, Bernstein IH. Psychometric theory, 3rd edn. New York:
McGraw-Hill, 1994.
25. Kline P. The handbook of psychological testing. London: Routledge,
1993.
26. Hair J, Anderson RE, Tatham RL, et al.Multivariate data NALYSIS
with readings. 4th edn. Englewood Cliffs, NJ: Prentice-Hall
International, 1995.
27. Jöreskog KG, Sörbom D. LISREL 8. Structural equation modelling
with the SIMPLIS command language. Hillsdale, NJ: Erlbaum, 1993.
28. Groves RM, Fowler FJ, Couper MP, et al.Survey methodology, 2nd
edn. Hoboken, NJ: John Wiley & Sons; 2009.
29. Bjertnæs ØA, Garratt AM, Johannessen JO. Data collection
methods and results in users surveys in mental health care. Tidsskr
Nor Laegeforen 2006;126:1481–3.
30. Bjertnæs ØA, Garratt A, Botten G. Nonresponse bias and
cost-effectiveness in a Norwegian survey of family physicians. Eval
Health Prof 2008;31:65–80.
31. Bjertnæs ØA, Garratt A, Iversen H, et al. The association between
GP and patient ratings of quality of care at outpatient clinics. Fam
Pract 2009;26:384–90.
32. Guldvog B, Hofoss D, Pettersen KI, et al. PS-RESKVA (Patient
Satisfaction, Results and Quality)—patient satisfaction in hospitals.
Tidsskr Nor Laegeforen 1998;118:386–91.
33. Jenkinson C, Coulter A, Bruster S, et al. Patients’experiences and
satisfaction with health care: results of a questionnaire study of
specific aspects of care. Qual Saf Health Care 2002;11:335–9.
34. Ware JE Jr, Hays RD. Methods for measuring patient satisfaction
with specific medical encounters. Med Care 1988;26:393–402.
35. Williams B, Coyle J, Healy D. The meaning of patient satisfaction:
an explanation of high reported levels. Soc Sci Med
1998;47:1351–9.
36. Sitzia J, Wood N. Patient satisfaction: a review of issues and
concepts. Soc Sci Med 1997;45:1829–43.
37. Tourangeau R, Smith TW. Asking sensitive questions: the impact of
data collection mode, question format, and question context. Pub
Opin Quart 1996;60:275–304.
38. Peytremann-Bridevaux I, Scherer F, Peer L, et al. Satisfaction of
patients hospitalised in psychiatric hospitals: a randomised
comparison of two psychiatric-specific and one generic satisfaction
questionnaires. BMC Health Serv Res 2006;6:108.
39. Sjetne IS, Bjertnæs ØA, Olsen RV, et al. The Generic Short Patient
Experiences Questionnaire (GSPEQ): identification of core items
from a survey in Norway. BMC Health Serv Res 2011;11:88.
40. Faber M, Bosch M, Wollersheim H, et al. Public reporting in health
care: how do consumers use quality-of-care information? A
systematic review. Med Care 2009;47:1–8.
Iversen HH, Holmboe O, Bjertnæs ØA. BMJ Open 2012;2:e001437. doi:10.1136/bmjopen-2012-001437 13
CPEQ: reliability and construct

41. Peters E, Dieckmann N, Dixon A, et al. Less is more in presenting
quality information to consumers. Med Care Res Rev
2007;64:169–90.
42. Bjertnæs ØA, Garratt A, Ruud T. Family physicians’experiences
with community mental health centers: a multilevel analysis.
Psychiatr Serv 2008;59:864–70.
43. Sjetne IS, Veenstra M, Stavem K. The effect of hospital size and
teaching status on patient experiences with hospital care: a
multilevel analysis. Med Care 2007;45:252–8.
44. Keller S, O’Malley AJ, Hays RD, et al. Methods used to streamline
the CAHPS Hospital Survey. Health Serv Res 2005;40(6 Pt 2):
2057–77.
14 Iversen HH, Holmboe O, Bjertnæs ØA. BMJ Open 2012;2:e001437. doi:10.1136/bmjopen-2012-001437
CPEQ: reliability and construct