Content uploaded by Bertrand Lombard
Author content
All content in this area was uploaded by Bertrand Lombard on Jul 10, 2019
Content may be subject to copyright.
Analysing collaborative trials for qualitative microbiological
methods: accordance and concordance
S.D. Langton
a,
*, R. Chevennement
b
, N. Nagelkerke
c
, B. Lombard
d
a
DEFRA Central Science Laboratory, Sand Hutton, York, YO41 1LZ, UK
b
Ecole Nationale d’Industrie Laitie
`re et des Biotechnologies (ENILBIO), Poligny, BP: 49, 39800, Poligny, France
c
National Institute of Public Health and the Environment (RIVM), P.O. Box 1, 3720 BA Bilthoven, The Netherlands
d
Agence Francßaise de Securite
´Sanitaire des Aliments, Laboratoire d’Etudes et de Recherches en Hygiene et Qualite des Aliments,
41 Rue du 11 Novembre 1918, F-94 700 Maisons-Alfort, France
Received 5 March 2001; received in revised form 25 January 2002; accepted 26 February 2002
Abstract
In qualitative (detection) food microbiology, the usual measures of repeatability and reproducibility are inapplicable. For
such studies, we introduce two new measures: accordance for within laboratory agreement and concordance for between
laboratory agreement, and discuss their properties. These measures are based on the probability of finding the same test results
for identical test materials within and between laboratories, respectively. The concordance odds ratio is introduced to present
their relationship. A method to test whether accordance differs from concordance is discussed. D2002 Elsevier Science B.V.
All rights reserved.
Keywords: Collaborative trials; Qualitative microbiological methods; Accordance; Concordance
1. Introduction
The concepts of repeatability (r) and reproducibility
(R)(Anonymous, 1994; IDF, 1991) are widely used in
the analysis of data from collaborative trials in quanti-
tative microbiology (see for example, Szita et al., 1998;
Schulten et al., 2000). Repeatability provides a meas-
ure of the variability between analyses conducted on
identical test materials by the same technician in the
same laboratory, under conditions as similar as possible
(e.g. by using the same apparatus and reagents within
the shortest possible interval of time), whilst the
reproducibility measures the variability when the anal-
yses are conducted by different technicians at different
laboratories. The concepts were originally borrowed
from the literature relating to chemical analysis, but
they are equally applicable to microbiological data,
provided the counts are transformed to logarithms
before analysis. Formally, there is a 95% probability
that the absolute difference between two test results
should be less than rand Runder repeatability and
reproducibility conditions, respectively.
These measures of repeatability and reproducibility
do not measure departure from the ‘‘true value’’, i.e.
accuracy. They only indicate to what extent results can
be repeated or reproduced—either correctly or not. In
microbiology, where the objective is to count the
0168-1605/02/$ - see front matter D2002 Elsevier Science B.V. All rights reserved.
PII: S0168-1605(02)00107-1
*
Corresponding author. Tel.: +44-1904-455100; fax: +44-
1904-455254.
E-mail address: s.langton@csl.gov.uk (S.D. Langton).
www.elsevier.com/locate/ijfoodmicro
International Journal of Food Microbiology 79 (2002) 175 – 181
number of colony forming units (cfu), accuracy may
depend on many factors, such as the (food) matrix, the
strain of organism or the presence and type of compet-
ing flora. A particular quantification method may
perform differently under different circumstances.
However, it should not perform randomly. Random
behaviour indicates a lack of ‘‘control’’, e.g. due to
ambiguities in the definition of the method. It is this
random behaviour that rand Rare designed to meas-
ure. In addition, (substantial) differences between rand
Rmay give clues about the origin of randomness. From
the wide use that is being made of these concepts, one
may infer that they must be useful and aid micro-
biologists in decision making and in improving and
standardizing their procedures. The increasing need to
standardize microbiological methods (Lahellec, 1998;
Leclercq et al., 2000) provides an additional argument
for the development and use of (statistical) methods
and the organization of collaborative trials (e.g. Scotter
et al., 2001; Schulten et al., 1998; De Buyser and
Lombard, 2000) that promote further standardization.
A major limitation of repeatability and reproduci-
bility is that they are only applicable to quantitative
data and cannot be applied to qualitative analyses, i.e.
detection methods. This is because both parameters are
expressed in terms of the likely difference in log units
between two test results; clearly this can have no
meaning for qualitative data where two results are
either the same or are different. In this paper, we
describe two new measures, accordance and concord-
ance, which can be regarded as analogous quantities to
repeatability and reproducibility for qualitative data.
These new parameters have been developed as part
of European project SMT 4-CT 96-2098 funded by
European Commission, which aimed to validate the
six main reference methods (ISO/CEN Standards) in
food microbiology by means of interlaboratory trials.
They have been in particular applied to the validation
of the reference method EN ISO 11290-1 for the
detection of Listeria monocytogenes in foods (Scotter
et al., 2001).
2. Example data
The methods will be illustrated by means of the
data in Table 1, which is taken from a collaborative
study on L. monocytogenes. These data are just for
one level of one food type using one particular
medium. In practice, a collaborative trial would usu-
ally aim to involve at least 20 laboratories (in this
case, there was a similar number using a different
medium), but a smaller data set makes it easier to
explain the calculations.
3. Statistics for qualitative data
3.1. Accuracy
In a collaborative trial with quantitative data,
whenever possible, results will be compared with
the ‘‘true’’ contents of the sample(s) in order to
demonstrate the accuracy of the method. The equiv-
alent statistics for qualitative data are straightforward.
For positive samples this is known as sensitivity and is
the percentage of samples correctly identified as
positives; in the example this is 46 (eight laboratories
with five correct plus two laboratories with three
correct) out of 50 or 92.0%. For the purposes of this
calculation it must be assumed that all supposedly
positive samples do in fact contain the organism; in
some cases with very low levels, this may not be the
case in reality. As the sensitivity can depend on
circumstances such as the food matrix, a reported
sensitivity only applies to the set of circumstances
under which it was measured.
For blank samples, the percentage of samples
correctly identified as being negative is recorded; this
is known as the specificity.
Approximate standard errors and confidence limits
for these figures can be based on standard methods for
Table 1
Example data taken from a collaborative study on L. monocytogenes
Laboratory Replicate number Number positive
12345
(out of 5)
1 +++++5
2 +++++5
3 +++++5
4 +++++5
5+++3
6 +++++5
7+++3
8 +++++5
9 +++++5
10 +++++5
S.D. Langton et al. / International Journal of Food Microbiology 79 (2002) 175–181176
binomial (positive/negative) data (Armitage and Ber-
ry, 1987). For example, using their methods to calcu-
late an exact 95% confidence interval for the sensi-
tivity, we obtain 80.8– 97.8%. However, such figures
may be misleading if there is any heterogeneity
among laboratories and should not be used where this
is clearly the case. Also, it is assumed that all samples
are in fact positive and that negative results are
therefore incorrect.
3.2. Accordance
We will define the qualitative equivalent of repeat-
ability as accordance; this is the (percentage) chance
that two identical test materials analysed by the same
laboratory under standard repeatability conditions will
both be given the same result (i.e. both found positive
or both found negative). To calculate the accordance,
we take each laboratory in turn and calculate the
probability that two samples will give the same result
and then average this probability over all laboratories.
For those laboratories (such as laboratory 1) where all
samples were found positive, the best estimate of the
probability of getting the same result is clearly 1.00 or
100%; all 10 out of 10 possible pairs are both positive.
For the others (in the example, laboratories 5 and 7)
we must count the number of pairs that are either both
negative (1 pair) or both positive (3 pairs). Thus a
total of 4 out of 10 pairs (40%) give the same result. In
general, when a laboratory has nresults and kof these
are positive, then the accordance for that laboratory is
estimated as
accordance for laboratory
¼fkðk1ÞþðnkÞðnk1Þg=nðn1Þ
The accordance for the collaborative trial as a
whole is the average (mean) of these probabilities
for each laboratory, which is 88% in this case, as can
be seen from Table 2. The calculation of the standard
errors and confidence intervals is discussed in Appen-
dix A.
3.3. Concordance
The equivalent of reproducibility is concordance,
which is the percentage chance that two identical test
materials sent to different laboratories will both be
given the same result (i.e. both found positive or both
found negative result). The most intuitive way to
calculate concordance is simply to enumerate all
possible between laboratory pairings in the data.
To enumerate all pairings, each observation in each
laboratory is considered in turn, starting with the first
observation of laboratory 1, which is positive. This
can be paired with any of the 45 observations from
other laboratories, and all but four of these pairings
(those with samples 1 and 2 of laboratories 5 and 7)
match (i.e. give a pair with both positive); there are
thus 41 pairs giving the same result. Similarly, the
other four samples from laboratory 1 also each have
41 out of 45 pairings giving the same result, and so
there are a total of 225 (5 45) between laboratory
pairings involving laboratory 1, of which 205
(5 41) give the same result. The same applies to
all other laboratories with all samples positive. For
laboratory 5, with three out of five positives, the two
negative samples each match with just two other
negative samples at other laboratories, whilst the three
positive samples each match with 43 positive samples.
Thus the total number of pairs with the same result for
laboratory 5 (and also laboratory 7) is 133
(2 2+343). These calculations are summarised
in Table 3.
The concordance is the percentage of all pairings
giving the same result; in this example this is 84.7%
(1906/2250 100). (The alert reader will have
noticed that there are actually only 1125 pairings as
we have double counted them all by this procedure;
this does not affect the results as it applies to all pairs
Table 2
Calculating the accordance for the example data; the accordance is
the average of the probabilities in the final column
Laboratory Number
positive
Positive
pairs
Negative
pairs
Total pairs
the same
Proportion
the same
1 5 10 0 10 100%
2 5 10 0 10 100%
3 5 10 0 10 100%
4 5 10 0 10 100%
5 3 3 1 4 40%
6 5 10 0 10 100%
7 3 3 1 4 40%
8 5 10 0 10 100%
9 5 10 0 10 100%
10 5 10 0 10 100%
Total = 88 Mean = 88%
S.D. Langton et al. / International Journal of Food Microbiology 79 (2002) 175–181 177
and is easier than allowing for it during the calcula-
tions.)
An alternative way to calculate concordance is to
first calculate all pairs with the same results from all
laboratories irrespective whether the result is from the
same laboratory or not. As there are four negative re-
sults and 46 positive results, we have 1041 (1035 + 6)
pairs with the same result (note that the number of pair-
ings possible between nitems is given by n(n1)/2,
for example for the 46 positives 46 45/2 = 1035).
From the calculation of accordance, we know that 88
are from the same laboratory (Table 2), and so 953
(1041 88) pairs with the same result are from differ-
ent laboratories (counting each pair only once). Sim-
ilarly, it is easily found that there are a total of 1125
(1225 100) pairs from different laboratories. Again,
we find 84.7% as the value of concordance.
1
The
calculation of the standard errors and confidence
intervals is discussed in Appendix A, and an Excel
spreadsheet to perform these calculations is available
by email from the authors.
3.4. Measuring and testing between laboratory
variation
With quantitative data, variation between laborato-
ries can be tested for statistical significance and its
magnitude can be estimated by examining the labo-
ratory component of variance or, alternatively, simply
by comparing the repeatability with the reproducibil-
ity. A similar comparison between accordance and
concordance can be used to assess the magnitude of
between laboratory variation; if the concordance is
smaller than the accordance (i.e. samples analysed at
the same laboratory are more likely to give the same
result than ones sent to different laboratories) between
laboratory variation is present. This may be due to, for
example, variations in media quality across laborato-
ries, differences in the likelihood of cross-contamina-
tion or different interpretations of guidelines by
different laboratory technicians.
Unfortunately, the magnitude of the concordance
and accordance is strongly dependent on the sensitiv-
ity, making it difficult to assess easily the degree of
between laboratory variation. It is therefore helpful to
calculate the concordance odds ratio (COR)
COR ¼accordanceð100 concordanceÞ
concordanceð100 accordanceÞ
where concordance and accordance are expressed as
percentages. This ratio is less sensitive to the level of
sensitivity than either accordance, concordance, or a
simple ratio between the two, and therefore provides a
convenient measure of the degree to which results
vary between laboratories. The magnitude of the ratio
can be interpreted as the relative chance (‘odds’ in
betting terms) of getting the same result when two
samples are analysed in the same laboratory compared
to if they are sent to the different laboratories. Thus an
odds ratio of 1.3 indicates that the samples are 1.3
times more likely to produce the same result (both
positive or both negative) if they are analysed in the
same laboratory than if they are analysed in different
laboratories. Ideally an odds ratio should be close to
1.0, indicating that results are just as likely to be the
same irrespective of whether the two samples are
analysed at the same or different laboratories. How-
ever, in practice, the variation between laboratories
will generally be larger than within laboratories. The
1
Concordance can also be calculated from accordance using the
formula
ðestimatedÞconcordance ¼f2rðrnLÞþnLðnL 1Þ
AnLðn1Þg=fðn2ÞLðL1Þg
where r= total number of positives, L= number of laboratories,
n= replications per laboratory, and A= accordance, expressed as a
proportion.
Table 3
Calculating the concordance for the example data
Laboratory Number
positive
Between-
laboratory pairings
with same results
Total
between-
laboratory
pairings
1 5 205 225
2 5 205 225
3 5 205 225
4 5 205 225
5 3 133 225
6 5 205 225
7 3 133 225
8 5 205 225
9 5 205 225
10 5 205 225
Total 1906 2250
S.D. Langton et al. / International Journal of Food Microbiology 79 (2002) 175–181178
larger the odds ratio, the greater is the level of inter-
laboratory variation in the data.
The COR equals infinity in the case when accord-
ance is 100% (i.e. when all laboratories report either
all positives or all negatives), indicating serious differ-
ences between laboratories. When both accordance
and concordance are 100%, there is no variability
whatsoever in the data and the COR can be considered
to equal the ideal value of 1.0.
Values of the concordance odds ratio above 1.00
may occur by chance variation, and so a statistical
significance test should be used to confirm whether
the evidence for extra variation between laboratories
is convincing. Such tests are based on the idea that if
accordance equals concordance, then each laboratory
has the same probability of finding a positive test. A
cross tabulation of laboratory by result (number pos-
itive, number negative) should then be homogeneous.
The best test to use is an ‘exact test’, which can be
obtained using statistical packages such as SAS or
StatXact. The philosophy behind such tests is that the
probabilities of occurrence are calculated for all sets
of results that could have produced the overall num-
bers of positives and negatives. In the example, with
46 positives and four negatives the arrangements
shown in Table 4 are possible.
The laboratories are not labelled in this table, as
any permutation would give the same overall result
(e.g. for the right hand column, the 1 could be for any
of the laboratories). The test adds up the probabilities
of all those possible arrangements that show at least as
much evidence for between laboratory variation as the
real result—here that means all permutations of the
three columns on the right. If this probability is less
than the conventional value of 0.05 or 5%, it is
unlikely that this degree of between laboratory varia-
tion could have occurred by chance and hence we
conclude that there is significant variation in perform-
ance between laboratories. In the example above
P= 0.039 indicating that variation between laborato-
ries is significant at the 5% level. Except in very
simple examples like this one, specialist software is
needed to calculate exact tests.
Where software for exact tests is not available, an
ordinary chi-squared analysis for contingency tables
(see e.g. Armitage and Berry, 1987) can be used as an
alternative. The results of this test will be less reliable
than the exact test with the number of replicates
usually used in collaborative trials, but simulations
suggest that the results provide a reasonable guide to
the significance of between laboratory differences.
With either test, it must be remembered that the
ability to detect between laboratory differences is
dependent on the number of laboratories and the
number of replicate samples analysed at each labo-
ratory. A non-significant test result should not be
taken to mean that performance does not vary between
laboratories, but rather that such differences have not
been proved; this is particularly true where the P-
value is only just above 0.05. The ideal solution is to
quote the odds ratio with a standard error or con-
fidence limits, but the distribution of the odds ratio is
highly skewed making it very difficult to produce
reliable limits.
4. Discussion
We have presented two new statistics, accordance
and concordance, that are aimed to assume the role of
repeatability and reproducibility for qualitative (detec-
tion) studies. Like the latter two measures, accordance
and concordance do not measure departure from the
‘‘true’’ value, i.e. presence or absence of an organism.
They only indicate whether the particular procedure
used is sufficiently ‘‘standardised’’. Departure from
the true value can be measured using concepts such as
sensitivity (probability of giving a positive result
Table 4
Possible arrangements of the example data with 46 positives and
four negatives
43321
44345
44555
45555
55555
55555
55555
55555
55555
55555
Accordance (%) 84.0 86.0 88.0 90.0 96.0
Concordance (%) 85.1 84.9 84.7 84.5 84.0
COR 0.92 1.09 1.32 1.65 4.57
The observed arrangement is shown in bold. For convenience, the
laboratories are shown in ascending order of positives.
S.D. Langton et al. / International Journal of Food Microbiology 79 (2002) 175–181 179
when the organism is present) and specificity (prob-
ability of giving a negative result when the organism
is absent). These concepts are different: for example,
specificity may be low (e.g. due to the presence of a
cross-reacting competitive organism, or to cross-con-
tamination) while both accordance and concordance
are high.
Although this is strictly speaking not part of the
description of accordance and concordance, we feel it
is appropriate to say a few words on which data
should be included in the calculation of these meas-
ures. As with the validation of alternative methods
(Anonymous, 2000), data should not be excluded
solely due to a statistical test for outliers (e.g. Dixon,
1951; Grubbs, 1969). This is because routine removal
of ‘outliers’ may give an unrealistic picture of the
method’s performance by removing the extremes of
variation in the data. Data should only be rejected
where there is a clear protocol violation or where a
laboratory is clearly not producing acceptable results.
The latter criterion might include instances where a
laboratory reports a very high proportion of positives
amongst blank samples. However, if such data would
have been deemed ‘‘acceptable’’ outside the context
of the trial (i.e. under routine circumstances), then
they should be retained in the analysis. In any case,
exclusion of data, and on what grounds, should
always be clearly reported.
One option where there are results which are
considered unusual (perhaps identified using Fisher’s
Exact Test), but cannot be ascribed to a protocol
violation, is to analyse the data with and without the
data from that laboratory. The value including all
laboratories should still be regarded as the definitive
figure, but comparison of the two figures will indicate
whether the one doubtful observation is having a
major influence on the overall statistics and what
scope there is for improvement in accordance and
concordance.
Acknowledgements
This study was supported by the European Union’s
Standards, Measurements and Testing Fourth Frame-
work Programme Project SMT4 CT 96 2098
(Coordinator C. Lahellec, Agence Francßaise de
Securite
´Sanitaire des Aliments).
Appendix A. Standard errors and confidence
intervals
The standard error of both the accordance and the
concordance depend on the way the data has been
collected and the interpretation we want to give to the
data.
The most common situation is that laboratories
should be considered a representative sample of a
larger ‘‘population’’ of laboratories, e.g. all laborato-
ries in Europe recognised under a certain programme.
Similarly, they may represent all laboratories in a
certain region or represent, for example, high quality
laboratories. For example, in order to assess the
variability in performance of a test and to certify its
performances, a certification body might randomly
select a small sample of the laboratories it has
approved and ask them to take part in a collaborative
trial.
When laboratories are considered representative,
results from a trial have implications for all laborato-
ries in the ‘‘population’’ of laboratories, not just the
participating ones. In the above example of a certif-
ication body examining the performance of a test, the
results of the trial have implications for all laborato-
ries which it has approved and therefore could have
been included in the trial. Standard errors then provide
an indication of how different the accordance and
concordance values might have been if, instead of the
actual sample, another sample of laboratories had
been taken.
Less frequently, the participating laboratories may
be ‘‘fixed’’ in the sense that they are not considered a
random sample of a population of laboratories. They
only represent themselves. For example, a group of
laboratories may have joined forces to examine (and
improve) the performance of certain tests in their
laboratories. The results of the trial have no implica-
tions for non-participating laboratories. Standard
errors then refer to how variable the data would have
been under replication of the trial in the same labo-
ratories.
The easiest way to calculate standard errors is by
using a statistical device called the ‘‘bootstrap’’ (Davi-
son and Hinkley, 1997). From the actual observations
a number M(usually a number of between 20 and 100
suffices) of bootstrap samples is created. The standard
deviation of accordance, concordance or COR (or any
S.D. Langton et al. / International Journal of Food Microbiology 79 (2002) 175–181180
other measure) among bootstrap samples then esti-
mates the standard error of these measures. A boot-
strap sample is constructed by sampling with re-
placement from the actual observations. The sample
size should be exactly equal to the original data set.
Where the Llaboratories can be considered as a
random sample of a larger population, we proceed
as follows to obtain a single bootstrap sample. We
give laboratories consecutive numbers 1, 2, 3,...,L.
We write these numbers on pieces of paper and put
them in a box. We then draw Ltimes a number, write
this number down, and after each draw we put the
piece of paper back in the box. Some laboratories are
thus ‘‘sampled’’ more than once, while others are not
included at all. Of course, randomly drawing from a
box can be done efficiently by simulating it on a
computer using a random number generator in, for
example, Excel. Standard errors then have the inter-
pretation of showing how the value of accordance and
concordance might fluctuate under replication of the
same collaborative trial in different samples of labo-
ratories from the same ‘‘population’’.
For fixed laboratories, it makes little sense to
resample laboratories, as these are considered fixed.
Instead, each bootstrap sample is obtained by boot-
strapping within each laboratory. The idea behind this
is that while laboratories are fixed, results obtained
within each laboratory are not. All laboratories are
present (once) in each bootstrap sample, but observa-
tions from within a laboratory are bootstrapped and
thus may be either absent, occur once, or more than
once in each bootstrap sample.
Once bootstrap standard errors have been esti-
mated, approximate 95% confidence intervals can be
obtained by taking the actual number plus or minus
two standard deviations.
An Excel application to calculate all the statistics
described in this paper, including the bootstrapped
standard errors, can be obtained by sending an email
to s.langton@csl.gov.uk.
References
Anonymous, 1994. ISO 5725 Accuracy (Trueness and Precision) of
Measurement Methods and Results International Organization
for Standardization, Geneva, Switzerland.
Anonymous, 2000. prEN ISO 16 140 Microbiology of Food and
Animal Feeding Stuffs - Protocol for the Validation of Alterna-
tive Methods International Organization for Standardization,
Geneva, Switzerland.
Armitage, P., Berry, G., 1987. Statistical Methods in Medical Re-
search, 2nd edn. Blackwell, Oxford.
Davison, A.C., Hinkley, D.V., 1997. Bootstrap Methods and Their
Application Cambridge Univ. Press, Cambridge.
De Buyser, M.L., Lombard, B., 2000. Validation of ISO Microbio-
logical Methods: Enumeration of Coagulase-Positive Staphylo-
cocci According to EN ISO 6888 Part 1 and Part 2: 1999. Agence
Francßaise de Securite
´Sanitaire des Aliments, Maisons-Alfort.
Dixon, W.J., 1951. Ratios involving extreme values. Ann. Math.
Stat. 22, 68 – 78.
Grubbs, F.E., 1969. Procedures for detecting outlying observations
in samples. Technometrics 11, 1 – 21.
IDF, 1991, Precision characteristics of analytical methods—outline
of collaborative study procedure. International IDF Standard
135B.
Lahellec, C., 1998. Development of standard methods with special
reference to Europe. Int. J. Food Microbiol. 45, 13 – 16.
Leclercq, A., Lombard, B., Mossel, D.A., 2000. Standardisation of
food microbiological analysis methods: an asset or a constraint.
Sci. Aliments 20, 179 – 202.
Schulten, S.M., vd Lustgraaf, B.E.B., Nagelkerke, N.J.D., In’t Veld,
P.H., 1998. Validation of Microbiological Methods: Enumera-
tion of Bacillus cereus According to ISO 7932 (2nd ed., 1993).
RIVM, Bilthoven.
Schulten, S.M., in’t Veld, P.H., Nagelkerke, N.J.D., Scotter, S., de
Buyser, M.L., Rollier, P., Lahellec, C., 2000. Evaluation of the
ISO 7932 standard for the enumeration of Bacillus cereus in
foods. Int. J. Food Microbiol. 57, 53 – 61.
Scotter, S.L., Langton, S., Lombard, B., Lahellec, C., Schulten, S.,
Nagelkerke, N., In’t Veld, P.H., Rollier, P., de Buyser, M.L.,
2001. Validation of ISO method 11290: Part 1. Detection of
Listeria monocytogenes in foods. Int. J. Food Microbiol., 64,
295 – 306.
Szita, G., Tabajdi, V., Fabian, A., Biro, G., Reichart, O., Kormoczy,
P.S., 1998. A novel, selective synthetic acetamide containing
culture medium for isolating Pseudomonas aeruginosa from
milk. Int. J. Food Microbiol. 43, 123 – 127.
S.D. Langton et al. / International Journal of Food Microbiology 79 (2002) 175–181 181
