A framework for the design, implementation, and evaluation of output-based surveillance systems against zoonotic threats

Abstract

Output-based standards set a prescribed target to be achieved by a surveillance system, but they leave the selection of surveillance parameters, such as test type and population to be sampled, to the responsible party in the surveillance area. This allows proportionate legislative surveillance specifications to be imposed over a range of unique geographies. This flexibility makes output-based standards useful in the context of zoonotic threat surveillance, particularly where animal pathogens act as risk indicators for human health or where multiple surveillance streams cover human, animal, and food safety sectors. Yet, these systems are also heavily reliant on the appropriate choice of surveillance options to fit the disease context and the constraints of the organization implementing the surveillance system. Here we describe a framework to assist with designing, implementing, and evaluating output-based surveillance systems showing the effectiveness of a diverse range of activities through a case study example. Despite not all activities being relevant to practitioners in every context, this framework aims to provide a useful toolbox to encourage holistic and stakeholder-focused approaches to the establishment and maintenance of productive output-based surveillance systems.

Full text

Frontiers in Public Health 01 frontiersin.org
A framework for the design,
implementation, and evaluation of
output-based surveillance systems
against zoonotic threats
SamanthaRivers
1
*, MaciejKochanowski
2, AgnieszkaStolarek
2,
AnnaZiętek-Barszcz
2, VerityHorigan
1, AlexanderJ.Kent
3 and
RobDewar
1
1 Animal and Plant Health Agency, Addlestone, United Kingdom, 2 Department of Swine Diseases,
National Veterinary Research Institute, Puławy, Poland, 3 National Wildlife Management Centre, Animal
and Plant Health Agency, York, United Kingdom
Output-based standards set a prescribed target to be achieved by a surveillance
system, but they leave the selection of surveillance parameters, such as test type
and population to besampled, to the responsible party in the surveillance area. This
allows proportionate legislative surveillance specifications to be imposed over a
range of unique geographies. This flexibility makes output-based standards useful
in the context of zoonotic threat surveillance, particularly where animal pathogens
act as risk indicators for human health or where multiple surveillance streams cover
human, animal, and food safety sectors. Yet, these systems are also heavily reliant
on the appropriate choice of surveillance options to fit the disease context and
the constraints of the organization implementing the surveillance system. Here
we describe a framework to assist with designing, implementing, and evaluating
output-based surveillance systems showing the eectiveness of a diverse range of
activities through a case study example. Despite not all activities being relevant to
practitioners in every context, this framework aims to provide a useful toolbox to
encourage holistic and stakeholder-focused approaches to the establishment and
maintenance of productive output-based surveillance systems.
KEYWORDS
output-based, surveillance, framework, zoonotic, design, implementation, evaluation
1. Introduction
e concept of One Health (OH) promotes the decompartmentalization of human, animal,
and environmental health for more ecient and sustainable governance of complex health issues
(1). is article details a framework developed as part of the MATRIX project, part of the OH
European Joint Programme (OHEJP). e OHEJP is a partnership of 44 food, veterinary and
medical laboratories and institutes across Europe and the Med-Vet-Net Association. MATRIX
aims to build on existing resources within OH Surveillance by creating synergies along the whole
surveillance pathway including the animal health, human health, and food safety sectors. is
work aims to describe the design, implementation, and evaluation of surveillance systems
against zoonotic threats using output-based standards (OBS).
An OBS does not strictly dene the surveillance activity that must take place in a geographical
area, e.g., to randomly collect and test X samples per year from Y location. Instead OBS is dened
by what the surveillance system must achieve, e.g., to detect a set prevalence of a hazard with a set
OPEN ACCESS
EDITED BY
Rene Hendriksen,
Technical University of Denmark, Denmark
REVIEWED BY
Paolo Tizzani,
World Organisation for Animal Health, France
Alasdair James Charles Cook,
University of Surrey, UnitedKingdom
*CORRESPONDENCE
Samantha Rivers
Samantha.Rivers@apha.gov.uk
SPECIALTY SECTION
This article was submitted to
Infectious Diseases: Epidemiology and
Prevention,
a section of the journal
Frontiers in Public Health
RECEIVED 22 December 2022
ACCEPTED 10 March 2023
PUBLISHED 20 April 2023
CITATION
Rivers S, Kochanowski M, Stolarek A,
Ziętek-Barszcz A, Horigan V, Kent AJ and
Dewar R (2023) A framework for the design,
implementation, and evaluation of output-
based surveillance systems against zoonotic
threats.
Front. Public Health 11:1129776.
doi: 10.3389/fpubh.2023.1129776
COPYRIGHT
© 2023 Rivers, Kochanowski, Stolarek, Ziętek-
Barszcz, Horigan, Kent and Dewar. This is an
open-access article distributed under the terms
of the Creative Commons Attribution License
(CC BY). The use, distribution or reproduction
in other forums is permitted, provided the
original author(s) and the copyright owner(s)
are credited and that the original publication in
this journal is cited, in accordance with
accepted academic practice. No use,
distribution or reproduction is permitted which
does not comply with these terms.
TYPE Methods
PUBLISHED 20 April 2023
DOI 10.3389/fpubh.2023.1129776

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 02 frontiersin.org
condence level (2). Output-based standards therefore allow for
variation in how surveillance is conducted, inuenced by a variety of
country/region specic factors including hazard prevalence,
performance of the tests used and mechanisms of infection. ese
standards can also enable the comparison of results from dierent
surveillance programs across dierent geographical contexts (3). Due
to this exibility, and ability to compare surveillance results across
countries and sectors, OBS are useful in the OH context where animal
pathogens may act as risk indicators for human health. In directing
eorts to minimize spread of zoonoses in the animal population with
robust surveillance, OBS may help to curtail the spread of disease at the
public health level. Surveillance systems implemented using OBS will
hereaer bereferred to as OBS systems.
e exibility of OBS systems also necessitates a far more involved
decision-making process when designing and evaluating them. While
passive surveillance can form part of the implementation of OBS, active
surveillance would also be needed to ensure that surveillance is
sucient to detect the design prevalence set out in the OBS. If
conducting active surveillance for a pathogen, practitioners
implementing OBS have the exibility but also the responsibility to
select the most appropriate host or medium to sample from, the test
type to use, and the geographical sampling distribution. ey must then
calculate the appropriate sample number to meet their OBS, and make
sure that each of these decisions works within the practical and
budgetary constraints of the existing organizational systems in their
surveillance area. Guidance has already been produced for analyzing
conventional surveillance systems in tools such as SERVAL (4),
RISKSUR (5), EpiTools (6), and OH-EpiCap (7). And while research
such as the SOUND control project is developing tools to encourage
and aid OBS implementation in Europe (8), there is currently no
broadly applicable, practical framework showing how OBS surveillance
systems can bedesigned, implemented, and evaluated. In this paper
weprovide a framework that aims to describe the surveillance format,
provide evidence-based decision-making on the best ways of applying
it, and showcase methodologies to evaluate these systems using
worked examples.
is framework is aimed at those who are considering OBS as a
solution to a surveillance need, whether they are looking to design and
implement a system from scratch, replace a conventional surveillance
system, or consider potential improvements to an existing OBS
system. Not all sections may berelevant to all users. us, while a
loose sequence exists throughout the framework, most sections can
beread out of order or in isolation. Depending upon your starting
point, the recommended route through this framework will dier; a
diagram showing these routes can befound in Figure1.
roughout this framework, wewill use the surveillance system
for Echinococcus multilocularis in Great Britain (GB) as a worked
example. Wehave chosen this pathogen because GB employs OBS for
Echinococcus multilocularis, it is a zoonotic pathogen with a wide
range of stakeholders that illustrate this process well.
2. Methods
2.1. Setting the scope of the framework
e goal of this work under the Matrix project was to develop
guidelines for the design, implementation and evaluation of ocial
controls, in this case active surveillance systems, which use OBS. is
needed to include methods for:
1. Identifying operational partners and stakeholders
2. Selecting appropriate output-based systems
3. Evaluating output-based methods.
Tools for evaluating surveillance systems have already been
produced such as SERVAL (4), RISKSUR (5), EpiTools (6), and
OH-EpiCap (7). However, these tools do not cover all essential aspects
of OBS. Hence, wewanted to produce a framework that would draw
from this past work, but would focus on the practical elements of
designing, implementing, and evaluating OBS systems.
2.2. Overarching approach
We sought to establish the essential attributes of OBS systems. For
the design section, we developed a series of activities that would
support the selection of appropriate design options for each attribute.
e implementation section provides activities and general practical
advice to assist with the roll-out of the nal OBS system design. e
evaluation section of the framework includes methods to assess the
ecacy of the implemented design against the current context.
Applying these methods would provide recommendations for
improving existing OBS systems.
2.3. Identification of design attributes
To identify the essential design attributes of OBS systems, wedrew
from a literature search conducted by Horigan (9) which included a
search of Scopus
1
and PubMed
2
using the search string “output or risk
and based and surveillance or freedom” in the “title, keyword, or
abstract.” is provided articles on a range of OBS systems for
zoonotic and non-zoonotic hazards.
From these articles, several surveillance attributes were found to
beespecially important for the success of OBS systems:
1. A strong understanding of the life cycle of the target hazard.
Hazard life cycles inuence the selection of host species and/or
medium tested for the hazard (10–13).
2. An appropriate sample number and distribution. For example,
selection of risk-based, random or convenience sampling to
provide a statistically robust demonstration of the hazard
prevalence (10, 14, 15).
3. A suciently cost-eective testing approach. is inuences
the practical feasibility and sustainability of the system (13,
16, 17).
We then investigated the OBS system for E. multilocularis in GB to
validate these attributes and gain further insight into these systems.
Contact with the Animal and Plant Health Agency (APHA) Parasitology
1 www.scopus.com
2 www.ncbi.nlm.nih.gov/pubmed

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 03 frontiersin.org
discipline lead and laboratory coordinator for E. multilocularis
surveillance in GB raised three further aspects to consider:
4. e clear denition of OBS system objectives
5. e identication and engagement of key stakeholders within
the system
6. e appropriate communication and reporting of results.
2.4. Development of framework activities
In the design section wedeveloped activities to help ensure system
designs considered these six identied attributes. ese activities were
mainly documentation exercises, providing an outline of the
information that should begathered and the design choices that
should bemade.
In the implementation section wefollowed systems mapping work
conducted in the COHESIVE project, a partner project to MATRIX
in the OHEJP. eir approach eectively described the Q fever
reporting and testing system in GB (18). Recognizing the practical
challenges of implementing OBS systems, wealso explored project
management techniques applicable to the implementation of large,
complex systems, including project le-shi, integrated stakeholder
feedback, and operational risk analysis and risk management, drawing
practical advice from the eld of systems engineering (19).
e evaluation section included activities that would provide
recommendations to improve the performance of the OBS system.
ese were also grounded in the six OBS system attributes listed above
and based on a range of previously published work and practical
experience. Wedeveloped a stakeholder analysis based on work by
Mendelow (20), selected because of its inclusion in the COHESIVE
project (21). A methodology for cost-eectiveness analysis was also
developed based on COHESIVE project outputs (22), using
information gathered under a literature review of economic analysis
approaches. A bespoke method for a exibility analysis to assess how
easily recommended changes to the system could beimplemented was
developed based on published research in the systems thinking eld
FIGURE1
Showing the recommended route an analyst should take through this guidance if they either know they want to improve an existing surveillance
system, want to design and implement an output-based surveillance system from scratch, or want to assess the performance of an existing OBS
system.

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 04 frontiersin.org
(23). Methodologies were also set out for evaluating the minimum
required sample sizes and true prevalence of hazards in host
populations using EpiTools (6), based on practical experience from
the Polish E. multilocularis surveillance system.
3. Framework
3.1. Design of an OBS system
Primarily, design is about selecting the appropriate attributes of a
surveillance system to deliver on its dened objectives, this requires
information gathering, decision-making, and objective setting. Here
weset out methodologies to dene the:
• System objectives
• Key stakeholders
• Target hazard and surveillance stream(s)
• Sampling methods
• Testing methods and costs
• D ata reporting
3.1.1. System objectives
e objectives describe what the surveillance system aims to
achieve from a top-level perspective, for example, to ll a regulatory
requirement, to contribute to a national strategy, or to assist with
disease or hazard control at the local level. us, the objective of an
OBS system could beto demonstrate freedom from disease, or to show
disease or hazard prevalence in a population with a certain level of
condence. For an OBS system the important attributes which should
beconsidered when setting the objectives are:
• Design Prevalence: is is a xed prevalence used to determine
the hypothesis that disease/hazard is present in a population of
interest (24). It can bethought of as the minimum prevalence
that youwould expect to detect using a given surveillance system.
• Condence levels: is is the level of certainty that the result is
correct. at is, when compared to the true level in the
population, the result of surveillance would be‘correct’ X% of the
time, where X is the condence level. e range of values for
which that remains true (sample prevalence = population
prevalence in X% of cases), is known as the condence
interval (25).
• Surveillance streams: these refer to the supply chain of samples
from a particular host population or medium (with associated
risk level) to the laboratory in which they are tested. A single
hazard could have several surveillance streams. For example, the
hazard could betested for in both live animals and bulk milk
from those animals, making up two surveillance streams within
the one system.
• Probability of introduction: Likelihood of the disease or hazard
in question being introduced to at least the number of units (e.g.,
animals) that would beinfected given the design prevalence.
One method of compiling a list of objectives is to use a hierarchy
of objectives which divides objectives into three tiers: policy, strategic,
and project (26). e policy objective is the overarching reason for
implementing this system at the top level such as providing condence
in disease freedom. Below this, the strategic objectives outline what
needs to beachieved to attain the policy objective such as testing a
specic design prevalence. Below strategic objectives are project
objectives. ese are the practical constraints and drivers that need to
be worked within to achieve the strategic and policy objectives.
Objectives in a tier below can bethought of as the ‘how’ of objectives
in the tier above, while objectives in the tier above can bethought of
as the ‘why’ of objectives in the tier below.
e objectives can be dened and validated through
communication with the prospective system stakeholders.
Example: Great Britain must demonstrate freedom from
Echinococcus multilocularis by upholding surveillance in accordance
with an output-based scheme prescribed by the European Commission
(27). Although GB has le the European Union (EU), this surveillance
is still mandated by retained legislation. In this example, the policy
objective therefore is to provide evidence of freedom from
Echinococcus multilocularis. e strategic objectives describe how this
OBS system aims to achieve this policy objective by detecting a 1%
prevalence in a representative host population with 95% condence,
but also to do so cost-eectively. e project objectives include the
sampling from appropriate denitive host(s) across a representative
geographic spread, the testing using a test of appropriate sensitivity
and specicity, and to do all of these within the budgetary constraints
of the project.
3.1.2. Key stakeholders
Stakeholders, dened as “any parties who are aected by or who
can aect the surveillance system” (28), have oversight of the
surveillance system and are a useful resource for informing design
choices to optimize the surveillance system design.
Generally, stakeholders comprise of three distinct groups: rst,
governance stakeholders with the inuence to set the required output
of the surveillance system, e.g., a regulatory authority like the
European Food Safety Authority (EFSA); second, delivery stakeholders
who are actively involved in the delivery of the required outputs, such
as the collection of samples, laboratory analysis or planning and
strategy roles; and nally, beneciaries who directly or indirectly
benet from the system running well, and whose wellbeing would
be directly or indirectly aected by a change to the surveillance
system. e general public, for example, are beneciaries of
surveillance systems involving zoonotic pathogens.
e list of stakeholders should becreated based on the available
information about the hazard and the objectives of the system. Once
a list of stakeholders has been established, a strategy for engagement
should be devised. A simple strategy could be to reach out to
stakeholders using links within your network. For example, through
people in your institution who have worked with them in the past.
Once contact with at least one stakeholder has been established, these
may then beused to establish contact with other stakeholders in the
system. Following initial engagement, stakeholders can begood
sources for further information gathering. A structured interview
with a pre-planned series of questions is recommended.
Example: In GB, weidentied potential stakeholders for the
E. multilocularis surveillance system using literature research
(particularly previous EFSA reports) and known contacts. Wethen
contacted one of our known stakeholders to develop a wider
stakeholder list. e nal list, per stakeholder group, was as follows:

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 05 frontiersin.org
Governance:
• e World Organisation for Animal Health (WOAH); who
record the disease status of E. multilocularis following the
compilation of GB results.
• e GB Department for Food, Environment, and Rural Aairs
(DEFRA); who compile the results.
• Local councils, who play a role in maintaining good education on
the disease/hazard and responding to cases.
• e European Free Trade Association (EFTA); who advise on the
measures which should bein place to control E. multilocularis
given a change in GB’s status.
Delivery:
• e Animal and Plant Health Agency (APHA), who maintain the
surveillance system, collecting samples and running analysis.
• e national reference laboratory (NRL) for Echinococcus
• APHA wildlife management team
• APHA wildlife risk modeling team.
• Veterinary practitioners, who respond to cases in dogs and hold
a stake in maintaining their good health.
• UK Health Security Agency (UKHSA), who respond to and
detect human cases.
• Hunters and gamekeepers, who provide carcasses from across the
country for testing.
Beneciaries:
• e Wildlife Trust, who support the welfare and environmental
inuences of surveillance on fox populations and the general
ecology. ey have a voice in ensuring surveillance does not
severely, or unnecessarily, impact the wellbeing of foxes.
• Fera science, a wildlife science advice organization who receive
samples from foxes and other wildlife for rodenticide survey, and
who could benet from collection of foxes for this surveillance.
• Science Advice for Scottish Agriculture, who also receive samples
from foxes for rodenticide survey.
• Pet owners, who hold a stake in making sure their pets remain
healthy, and who are at risk of infection in the event of incursion.
• Media outlets, who have an interest in distributing information
on the quality of surveillance and in the event of case detection.
• e general public: good surveillance ensures that any incursion
of E. multilocularis reaches as few members of the public
as possible.
3.1.3. Target hazard and surveillance stream
Knowledge of the hazard both informs the choice of surveillance
stream, and heavily impacts the downstream practical decisions
around how the system will function. Structured interviews with
stakeholders along with literature research can provide knowledge
about the target hazard which can becompiled into a prole. Any
relevant information can beadded to this prole, but it should aim to
bea complete overview covering all OH aspects. If the hazard is a
zoonotic pathogen, particularly if it is foodborne, this should
be agged at this stage. As with the target hazard, the choice of
surveillance stream, including the target host population and/or
detection medium (e.g., red fox feces or bulk milk) is key to the system
design. Sampling is usually from the population considered most at
risk of infection or contamination and therefore the one in which
youare most likely to detect a positive case. e choice of population,
and the medium from which this population are sampled, has
implications on almost all areas of the workow, including the
applicable sampling types and methods, and the geographical area(s)
sampled.
Example: In the case of E. multilocularis, the red fox is the most
relevant to sample in GB as it is a denitive host for the hazard and is
also widely abundant. Additionally, sampling individual animals
rather than collecting environmental samples or sampling from
intermediate hosts is more compatible with the available testing
methods for the hazard, which require tissue samples. is also
ensures that positive detection relates to one animal, rather than
leaving potential for multiple sources of contamination as
environmental samples would. It ensures the species and approximate
location of death is known.
3.1.4. Sampling methods
e distribution of the target population and the sampling
strategy are essential for informing the type of test used, and how the
nal design proposal will beimplemented.
Samples may betaken using a risk-based framework or by taking
randomly from the entire population. While convenience sampling
could detect a case and thereby rule out disease freedom, it is not
recommended for output-based surveillance as it would beunlikely
to support representative sampling of the host population to prove
disease freedom. Delivery stakeholders can provide the contextual
knowledge to inform the type of sampling that is most appropriate and
feasible. Additional external information sources such as population
surveys could provide further information to support the chosen
sampling type.
Regardless of the sampling method chosen, we recommend
including all populations that are relevant to the probability of
introduction of the pathogen. For farmed or kept animals, this will
likely include multiple surveillance streams, for example, sampling
from slaughter animals, imported and moved animals. For wild
animals, relevant surveillance streams may include samples from
trapped or hunted animals, roadkill, resident populations, and
transient or migratory populations, particularly where they
cross borders.
e sampling methods link closely to the testing method chosen
because the number of samples required will vary based on the
sensitivity of the test used, and because certain tests will only
becompatible with certain sample media (e.g., serum, nasal swab, or
feces). In order to conrm the number of samples required, and to
validate condence in the test results, wesuggest using a sample size
calculator such as EpiTools (29).
Example: Using E. multilocularis in GB as an example, the red
fox population was 357,000 (30). e egg otation test can berun
on intestinal tissues of fox carcasses with an estimated test
sensitivity of 0.78 (31). With these inputs, EpiTools output was a
suggested sample size of 383 fox carcasses to detect the hazard at a
1% design prevalence with 95% condence, given a random
sampling distribution.

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 06 frontiersin.org
3.1.5. Testing methods and costs
When choosing a testing method, we suggest engaging
stakeholders and reviewing literature for an overview of the tests
available. From there, the most appropriate method can bechosen,
considering the budget and resources available, the sensitivity and
specicity of the testing method, the population available for testing
and the specic surveillance scheme chosen.
As part of test selection, understanding the costs of testing helps
determine whether surveillance is achievable within the budgetary
constraints of your system. is is also a useful precursor to
establishing which surveillance streams give the best value for money,
as described in the cost-eectiveness analysis guidance in the
evaluation section.
Generally, the cost of testing can be broken down into
the following:
• Consumables and reagents: is covers any routine consumables
costs such as reagents, PPE, laboratory, or eld consumables.
• Sta: is covers all costs relating to sta, e.g., cost of sta time
for sampling, testing, training and travel.
• Equipment: is covers the cost of all equipment used in the
system. is may, for example, include the cost of purchasing and
maintaining laboratory equipment.
• Other operational costs: is covers all other costs not accounted
for, such as sample transport and equipment maintenance.
Delivery stakeholders may beable to provide detailed cost data,
depending on which part of the system they are linked to. For example,
laboratory stakeholders may beable to provide the procurement costs
of reagents if they are already used for other tests. If further
information is needed, an average price per item can be sought
through the price lists of online retailers.
Example: For the GB E. multilocularis, we used the standard
operating procedure (SOP) of the egg otation method to generate a
list of consumables, reagents and equipment which were then assigned
hypothetical values detailed in Table1.
3.1.6. Data reporting
e types of data to report will depend on the surveillance
program. In general, a system should report the frequency of data
collection, the sampling strategy and testing method used, along with
sensitivity/specicity, target population, sampling period and volume,
methodology for results analysis, and results of testing. Commonly,
these data are provided in scientic reports to the
governance stakeholders.
Example: e full data reporting for GB E. multilocularis can
befound in the annual reports produced by EFSA prior to 2021 (32),
and are explored in this example.
From the 2019/2020 sampling year, GB reported results for 464
samples taken between March 2019 and January 2020, from locations
across GB (31).
e testing was conducted using the egg otation method (31)
with an overview of the methodology provided in the report (32).
Random sampling was used, with the sample size calculated by the
RIBESS tool (33) based on the test sensitivity, and the estimated
population size for detection at 1% prevalence with a 95% condence
interval. EFSA evaluated the data provided to determine whether it
fullled the legal requirements of the legislation and assigned a
disease-free status.
3.2. Implementation of an OBS system
To aid system implementation, it is important to outline how the
proposed OBS will function in a way that communicates its vision and
purpose to the system stakeholders. e stakeholders can then provide
feedback on the proposed system design and suggest improvements to
make it more practically or economically viable. Once the design has
been agreed, a strategy can bedevised for maintaining the continued
quality of the system through test validation and accreditation.
3.2.1. System mapping
System mapping provides a ow diagram showing all processes
from the point of sample collection to the reporting of results.
Visualizing the entire system in this way helps document the sequence
of the surveillance system and makes the function of the system
easily disseminated.
e simplest method for system mapping is constructing a ow
diagram with direct input from your stakeholders (18). is should
describe the steps from sample acquisition to result analysis. Most of
the system structure will already have been determined in the design
process. However, any remaining aspects of the system that are unclear
should be highlighted in this ow diagram and claried by the
stakeholders. e diagram should outline which stakeholders will
beinvolved at each step in the process.
e system structure map can also beused to represent any
synergistic systems linked to the surveillance, for example, if the same
samples could beused for other purposes. is helps document the
linkages of the surveillance system with other activities and highlights
opportunities to make sampling more practical, cost-eective and
mutually benecial. e surveillance system for E. multilocularis in
GB, for example, has multiple stakeholders each contributing to, and
benetting from, its various stages (Figure2).
TABLE1 Hypothetical data showing the cost breakdown per test of the
egg flotation test, and the data sources associated with these costs.
Parameter Value
Tes t Egg otation
Species sampled Fox
Test sensitivity 0.78
Test specicity 1
Parameter Unit Cost/Value
Consumables and reagents Per test €56.88
Sta time (testing) Per test €9.26
Operational costs (excluding
testing)
Annual cost €291,593.12
Equipment Annual cost €894.15
Tests required at 1%
prevalence
No. of tests 383
Cost of testing at 1%
prevalence
Total cost €165,823.53

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 07 frontiersin.org
3.2.2. Project management planning
Eective project management is required to coordinate the
implementation of your proposed surveillance design, especially if
operating to a deadline. Formal training in this eld is highly
recommended before undertaking the implementation of any large,
complex output-based surveillance systems. However, wesuggest
drawing ideas from systems engineering practices such as project “le
shi.” is focusses on shiing project funding and input to the start
of a project rather than the end of it. Early investment in a project
provides better value for money due to ination. Also, spending more
time on the early planning stages of the project can prevent mistakes
that may bechallenging or expensive to resolve later in the project (34).
In the implementation of output-based surveillance systems, le-
shi means investing heavily in building up the cohesion and
experience-base of the delivery stakeholders of the system. ese are
similarly highlighted as important factors in the RISKSUR framework
best practices (35). is could include investment in dedicated
training for sample collection, analysis, and result reporting, or a pilot,
where a small number of samples are collected and tested to ensure all
aspects of the system work well together before scaling up. Outreach
could be part of this early investment. For example, allowing
laboratory sta time to shadow sample collectors and vice versa. Such
activities will greatly improve cohesion along the sample analysis
pipeline, allowing stakeholders to form close working relationships,
facilitating a faster response to problems and potentially contributing
to eciency gains as stakeholders share experiences with one another.
Verication and validation stages with stakeholders during
implementation are also recommended. ese stages could test
whether each part of the system delivers on the original system
objectives and provides value to stakeholders as the systems are being
implemented (36). Verication, as with all stages of project
management, should bewell documented and werecommend having
a robust documentation process to make sure plans and activities are
transparent to the implementation team and wider stakeholders
(37–40).
Another recommendation is to conduct an operational risk
analysis. is can identify, assess, and derive actions against issues
which can occur during the implementation process. In this risk
analysis, the probability of each of these risks occurring and the
impact if these risks occur as either Low, Medium, or High. is
facilitates decision-making on the proportionate action to take to
either avoid these risks, mitigate their impacts, or accept them.
We recommend guidance in Lavanya and Malarvizhi (41) or the
textbook by the Institution of Civil Engineers (42) for further details
on the steps to follow for operational risk analysis. All changes made
to avoid a risk must bechecked against the prior design stages
and documented.
Stakeholders should agree with the outcomes of risk analysis, to
any resultant changes to the system design and any accepted risks.
Agreeing the nal system design and implementation strategy with
delivery stakeholders will improve the likelihood of successful
implementation (43).
3.3. Evaluation of an OBS system
is section provides a range of evaluation exercises to help direct
improvements to existing OBS systems.
3.3.1. Evaluation of system objectives
is evaluation determines whether the system objectives are still
relevant and complete. For example, the hazard prevalence may have
changed since the implementation of the OBS system, so is the design
prevalence for detection still appropriate? A new test may have been
developed for the target hazard, so how does this compare with the
test currently implemented?
Assessing the suitability of the system objectives requires analysis
of current research relevant to the OBS system. is can beconducted
through a combination of literature review and stakeholder
engagement, to explore the following questions:
• Has the level of detection changed since the rst implementation
of the surveillance system? Has prevalence of the hazard
increased/decreased or changed in its geographical distribution?
• Has new evidence come to light on the dynamics of the hazard
under surveillance? For example, have new competent hosts
been found?
• Have new tests been developed for the same hazard and host as
the original surveillance system? Do these new tests oer
improved sensitivity and/or specicity to the current option; do
they oer other advantages?
• Have any aspects of the surveillance system been recognized to
beoperating particularly well? For example, have other groups
taken inspiration from the current system and implemented the
same methods elsewhere?
• Have any issues or doubts about aspects of the surveillance
system been raised? Are any of these corroborated by data?
• Has the political or legislative context of surveillance changed?
Has the target hazard or population become higher or lower
priority to governing bodies? Is the need for surveillance brought
in to question by these changes?
3.3.2. Flexibility analysis
It is expected that every system will undergo changes throughout
its lifecycle. A good output-based surveillance system needs to
beadaptive to technological, practical, or political changes to continue
delivering value for its stakeholders. A exibility analysis determines
how changes to a system could aect its various stakeholders and its
ability to deliver on its core objectives.
Determining the exibility of the system requires systems
thinking so werecommend using causal loop diagrams to illustrate
links between system components and stakeholders. e system
components are any aspect of the system that aect its overall
function. e surveillance streams, test type, number of tests, design
prevalence, and even the method of result reporting and analysis can
all beconsidered system components. Causal loop diagrams illustrate
the dynamics of complex systems by showing the positive or negative
relationships system components have on one another and on the
stakeholders (23, 44). To produce these diagrams, the rst step is to
identify which system components aect each stakeholder. For
example, sample collectors will bedirectly impacted if they are asked
to collect more samples. e number of samples required is inuenced
by the sensitivity and specicity of the test chosen, and by the design
prevalence and required condence level set out in the system
objectives. Hence, these stakeholders are linked to the sampling
requirements, the test chosen, the design prevalence, and the required

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 08 frontiersin.org
condence in the results. When a link is demonstrated, it is essential
to show whether the relationship is positive or negative. For example,
higher test sensitivity has a negative eect on the number of tests
required since more sensitive tests are statistically more likely to detect
a hazard if it is present. Logically, the number of tests required
positively inuences the number of samples taken: more tests required
means more samples will need to betaken and consequently, these too
are linked. While making these links, it is likely that further
FIGURE2
Showing the system structure and chronology from carcass collection to result reporting. Rectangles represent steps in the system while circles
represent stakeholders involved in relevant steps.

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 09 frontiersin.org
interrelationships between dierent stakeholders and system
components will emerge. Documenting all relevant links will provide
a complete picture of the emergent impacts of design decisions on
each of the stakeholders.
Once the links between design decisions and stakeholders have
been established, engagement of stakeholders is required to determine
their tolerance to change. If stakeholders operate under xed
constraints these should beidentied and documented. For example,
delivery stakeholders may beworking within a budgetary range. If
they can agree to an increase in sampling rate, what is their maximum
sample number? Governance stakeholders may have some tolerance
in the design prevalence or testing condence they expect to see from
a surveillance system. What is this tolerance and to what extent could
the system adapt before those tolerances are exceeded?
Example: For E. multilocularis surveillance in GB, wedetermined
that changing the type of surveillance scheme, for example the test
used, would impact the required sample size, and thereby aect both
the workload of the delivery stakeholders and the condence in the
test results, altering the outcome for end beneciaries. By representing
the system using a causal loop diagram (Figure3), weidentied 5
distinct interrelationships to beaware of if any changes to the system
are considered. ese were:
• e chosen surveillance scheme will aect how many carcasses
are collected, and where they are collected from (for example, if
collected according to risk-based sampling rather than random
sampling). is has ripple eects on every other part of
the system.
• A higher sample requirement would mean more time and money
spent collecting those samples. It would also demand more from
farmers, hunters and gamekeepers to provide carcasses for
analysis. is could strengthen or damage relationships with
these stakeholders, depending on their appetite for collaboration,
and thereby increase or decrease their satisfaction with the
system and their willingness to supply samples (45). Hunters,
farmers and gamekeepers already deliver an excess of samples to
APHA, and it was estimated they would be receptive to an
increase in the number of carcasses asked of them if needed,
though their specic upper-bound tolerance was unknown.
• More carcasses collected means more of all sample types are
available for commercial collaborators.
• A higher sampling rate, or improvement in the geographical spread
of collected samples will increase the overall condence in the
surveillance system. It will increase the probability that cases in
wildlife will bedetected before the disease becomes established in
the wild population. is will reduce the number of human cases,
and therefore provide a higher benet to society at large.
• A change in the costs of maintaining the system, and the
downstream eects on the benet to stakeholders, will aect the
benet–cost ratio of the surveillance system. A higher benet–
cost ratio means the surveillance system generates greater value
for money.
3.3.3. Stakeholder analysis
is evaluation determines and depicts the level of interest and
inuence current stakeholders have in the system. Stakeholders have
diverse views and roles. us, to understand them, it is a useful
exercise to categorize them in order to identify the most inuential
stakeholders, or those who hold the largest stake in the system
achieving its objectives. As a result, it is then possible to establish
whether the position of individual stakeholders on the matrix is
appropriate. A modied Mendelow matrix is an eective way to
categorize stakeholders. is is a two-dimensional matrix plotting the
interest and inuence of stakeholders (20). It provides information
about which stakeholders are the most engaged, and which are
most inuential.
Structured interviews should beused to determine the level of
inuence and interest in the system. Direct questions are a good
starting point, for example ‘what is your perceived level of inuence
on the system?’. It can beuseful to follow up with more descriptive
questioning. A question which asks the stakeholders how they might
implement change to a system could return more tangible insights into
FIGURE3
Example causal loop diagram illustrating the positive and negative
interrelationships of dierent parts of the UK E. multilocularis
surveillance system and the perceived stakeholder benefits and
losses from changing aspects of the system.

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 10 frontiersin.org
the barriers stakeholders face when trying to implement change. A
stakeholder with high inuence will likely have a strong idea of how
to enact change to the system and may even have been directly
involved in making prior changes to the system.
e level of interest in the system involves how stakeholders
would beaected by changes to the system. When ascertaining the
interest of stakeholders, questions that explore hypothetical scenarios
may yield richer results, for example, asking how a stakeholder might
beaected by increasing or decreasing the sample numbers taken, or
by changing the objectives of the system. If their answers indicate they
would need to take immediate action because of these changes, this
illustrates a high level of interest in the system. For beneciaries of
output-based surveillance systems, such as the general public, who
may not beaware of the implications of changes to it on their own
health and wellbeing however, this can bea challenge. A judgment can
bemade in these cases based on the prior information compiled.
Another tool for collecting information from stakeholders could
be survey-based questions rating interest and inuence on a
quantitative scale, for example from 1 to 10. With interviews and
surveys, every eort should bemade to contact as many stakeholders
as possible from across the system. Where this is not possible, a proxy
can be used to evaluate/assess the inuence and interest these
stakeholders have. is could bebased on the perceptions of other
stakeholders in the system, taking care to get input about missing
stakeholders from as many other stakeholders as possible. Once the
bulk of information has been compiled, they can beplaced on the
Mendelow’s matrix. A completed matrix of all stakeholders should
then beveried by the stakeholders.
Finally, you should evaluate whether the position of the
stakeholders on the matrix is still appropriate, particularly regarding
the inuence they have on the system. is can beassessed by asking
stakeholders whether they think they should have more or less
inuence on the system in the future. A desire to change their level of
inuence can be represented on the matrix with arrows. Arrows
provide an indication of stakeholder satisfaction and suggest areas for
improving stakeholder involvement.
Example: For the E. multilocularis surveillance system in GB,
we reached out to stakeholders via email or through interviews,
assembling information to plot these stakeholders on a Mendelow
matrix. Weinterviewed the following stakeholders:
• APHA Parasitology discipline lead and laboratory coordinator
for E. multilocularis surveillance in Great Britain.
• Carcass collection coordinator for E. multilocularis
surveillance in GB.
• APHA discipline lead for wildlife epidemiology and modeling,
leading E. multilocularis sample selection, and risk modeling.
• Science Advice for Scottish Agriculture research coordinator,
rodenticide sampling in wildlife
• Fera Science research coordinator, rodenticide sampling
in wildlife
Additionally, we contacted the UK Health Security Agency
Emerging Infectious Zoonoses Team and DEFRA via email but were
unable to reach WOAH. When interviewing, we discussed the
following topics with each stakeholder:
• e role of the stakeholder within the system
• e perceived roles of other stakeholders in the system
• eir perceived understanding of how the surveillance system
practically functioned to deliver outputs
• eir perceived inuence on the system
• eir satisfaction with the system, particularly with regards to the
level of inuence they had on it.
For stakeholders that could not becontacted directly, attributes
were estimated from the expert knowledge of the other stakeholders;
from their past interactions with these stakeholders and their
experience working within the system. With the information compiled
in the interviews, it was possible to map each stakeholder on a
Mendelow matrix (Figure4).
In the future, DEFRA will receive the annual reports of the
surveillance, therefore, they have both high interest and high inuence
on the matrix. APHA, and WOAH are also in this quarter of the
matrix; APHA are responsible for carrying out the surveillance and
WOAH are responsible for producing the annualized reports to prove
disease freedom and publishing results shared by member states. With
the current GB situation for E. multilocularis, the UKHSA is in the low
interest, high inuence quarter of the Matrix. However, this would
likely change to high interest, high inuence, if there were changes to
the status of E. multilocularis in GB. When asked, satisfaction was very
high: no stakeholder felt they needed more or less inuence on
the system.
3.3.4. Minimum sample size evaluation
is evaluation calculates the minimum sample size required to
detect hazard at a set design prevalence and condence level. is
calculation is relevant for monitoring the hazard in the population. If
the sample size is too big it will result in excess nancial cost. If the
sample size is too small, it can lead to the system not achieving its
objectives. Scientic publications, international and governmental
statistical data, hunting associations or other professional
organizational data, expert opinions, and gray literature can all
provide relevant population size data and information about test
sensitivity. Furthermore, the sensitivity of the test can also
bedetermined via validation studies and in the case of a commercial
test, via the test manufacturer. is information can then beused to
calculate the minimum sample size needed for surveillance using the
online EpiTools calculator - “Sample size for demonstration of
freedom (detection of disease) in a nite population” (29).
is tool can calculate the sample size needed to achieve the
required probability of detecting disease or presence of a hazard
(herd-sensitivity) at the dened design prevalence for a nite
population, assuming a diagnostic assay with known sensitivity and
100% specicity. ese calculations use an approximation of the
hypergeometric distribution (29, 46). According to MacDiarmid (46)
the probability (β) that there are no test-positive animals in the sample
tested can becalculated as:
β
=−





1nSE
N
pN
where:
• p = true prevalence of infection
• SE = sensitivity of the test
• N = herd size

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 11 frontiersin.org
• n = sample size
e required parameters (inputs) for the calculator are:
• Population size
• Test sensitivity
• D esired herd-sensitivity
• Design (target) prevalence
The main output of this EpiTools analysis is the number of
samples required to provide the desired herd sensitivity for a
specified design prevalence. The results of this analysis are 383
sample required for both the SCT and IST, and 336 samples
required for the PCR. The calculations concerned
E. multilocularis in the red fox population in selected European
countries. In these calculations, the EpiTools calculator inputs
were set as follow:
• Red fox population size- dened according to the data from
publications and reports (Table2)
• Sensitivity of E. multilocularis detection test (sedimentation and
counting technique (SCT) 0.78, intestinal scraping technique
(IST) 0.78, or PCR method)- derived from publications and
reports as reported in Table3.
• Desired herd-sensitivity – was set at 0.95
• Design (target) prevalence – here was set in accordance with the
calculated true prevalence
Furthermore, this EpiTools calculator can generate graphs of the
sample sizes needed to achieve the desired herd sensitivity, for a
dened test sensitivity and range of population size and design
prevalence (Figure5).
3.3.5. True prevalence evaluation
is section estimates the true prevalence to conrm or correct
any previously calculated prevalence of disease (apparent prevalence).
Most diagnostic tests have imperfect sensitivity and specicity.
Calculation of true prevalence (the proportion of a population that is
actually infected) considers the sensitivity and specicity of the
applied test. Calculating the true prevalence can determine whether
the choice of design prevalence for the system is still appropriate. is
is more accurate than calculations of apparent prevalence (the
proportion of the population that tests positive for the disease) which
are reported in the majority of epidemiological studies/reports and do
not include these parameters. Scientic publications, international
and governmental reports, expert opinions, and gray literature can all
beused to nd these data.
A useful tool for calculating true prevalence is the EpiTools
calculator – “Estimated true prevalence and predictive values from
survey testing” (29). is tool calculates the true prevalence, as well as
positive and negative predictive values, and likelihood ratios based on
testing results using an assay of known sensitivity and specicity (29).
For example, true prevalence of E. multilocularis in Poland was
calculated by EpiTools calculator as 18.64% (95% CI, 16.64–20.82)
while apparent prevalence was 16.5%. Based on this example one can
FIGURE4
Stakeholders involved in GB E. multilocularis surveillance mapped to a Mendelow matrix, sorted by level of influence and interest in the surveillance
system.

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 12 frontiersin.org
see that number of tested samples, number of positive results, method
sensitivity as well as method specicity eect on calculation result. For
Poland and other selected countries of EU calculations of true and
apparent prevalence are presented in Table3. Furthermore, EpiTools
calculator enables graphical visualization of output results.
Using E. multilocularis prevalence in Poland as an example, the
inputs required to perform computations by the EpiTools calculator
are as follows:
• Number of examined samples obtained from red foxes (intestines
or faeces samples) and number positive samples- set according
to data from publications and reports as indicated in Table3.
• Sensitivity and specicity of the method (SCT, IST or
PCR method)
• Condence level – was set at 0.95
• Type of condence interval for apparent prevalence – Wilson CI
was used
• Type of condence interval for true prevalence – Blaker was used
To determine the true prevalence (TP) from these data, EpiTools
applies the Rogan-Gladen estimator, using the following formula:
TP AP SP
SP SE
=
+−
()



+−
()



1
1
where:
• AP = apparent prevalence
• SP = specicity
• SE = sensitivity
3.3.6. Cost-eectiveness analysis
It is important that the testing process and the overall cost of the
wider surveillance scheme is as cost eective as possible. is likely
also aects stakeholder satisfaction and may aect the long-term
sustainability of the system. To evaluate this, it is recommended to
carry out a cost-eectiveness or cost–benet analysis (or similar
applicable economic analysis method). is example specically looks
at cost eectiveness analyses (CEA).
Cost eectiveness analyses measure the input cost required for the
system to produce a given output. Unlike some other economic
analysis approaches, the ‘eectiveness’ component of a CEA can
bedened by the analyst. In output-based surveillance, the output is
already dened at the operational level (to detect a stated design
prevalence with a stated condence). Cost eectiveness analysis can
easily beapplied in these cases, to measure the cost input required to
meet these outputs. is can then becompared directly to alternative
approaches. Gathering data on the cost inputs of a system rst requires
an inventory of all materials and reagents used, sta time required,
and any transport and sample collection costs. Materials and reagents
can befound using laboratory standard operating procedures (SOPs).
e price of each cost component may beattainable through contact
with stakeholders working within the system. Alternatively, these may
befound on supplier websites. Sta time should ideally bederived
through contact with the sta themselves, preferably sta who have a
holistic view of the system from sample acquisition to result reporting.
TABLE2 Calculation of the number of samples required to detect E. multilocularis in the red fox population in selected European countries.
Country Red fox population Sample size for demonstration detection of disease
References 2009 2010 2011 2012 2013 2014 2022 2009 2010 2011 2012 2013 2014 2022
Poland [1] 193,402 210,332 198,679 19 19 19
Latvia [2] 35,000 34,800 9 9
Denmark [3] 31,100 405
Hungary [4] 78,000 60
Romania [5, 6] 53,292 63
Finland [7, 8] 150,000 384
Ireland [7, 9, 10] 150,000 339
Great Britain [7, 11] 240,000 353
Norway [7, 12] 70,000 70,000 70,000 151,000 475 476
References: [1] – e Forest Data Bank (47); [2] – Kirjušina etal. (48); [3] – Danish Centre For Environment And Energy (49); [4] – European Health and Digital Executive Agency European Food Safety Authority (50) (HaDEA); [5] – Şuteu etal. (51); [6] – Romanian
National Institute of Statistics (52); [7] – European Food Safety Authority (50); [8] – Kauhala (53); [9] – Hayden and Harrington (54); [10] – Marnell etal. (55); [11] – DEFRA Department for Environment and Agency (56); [12] – Sviland etal. (57).

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 13 frontiersin.org
TABLE3 Calculation of the true prevalence of E. multilocularis in red foxes in selected European countries.
Country Apparent prevalence calculation True prevalence calculation
Survey
references
No. of
tested
samples
Number of
positive
results
Method Apparent
prevalence (%)
Sensitivity and
specificity
references
Method
sensitivity
Method
specificity
True
prevalence (%)
95% CI
Poland [1] 1,546 255 SCT 16.5 [12] 0.885 1 18.64 16.64–20.82
Latvia [2] 45 16 SCT 35.6 [12] 0.885 1 40.18 26.24–56.68
France [3] 3,307 562 SCT 17 [12] 0.885 1 19.2 17.8–20.69
Germany
(northern) [4] 3,094 523 SCT 16.9 [12] 0.885 1 19.1 17.65–20.64
Denmark [5] 546 4 SCT 0.73 [12] 0.885 1 0.83 0.32–2.11
Hungary [6] 100 5 SCT 5 [12] 0.885 1 5.65 2.43–12.63
Romania [7] 561 27 IST/SCT 4.8 [13] 0.78 1 6.17 4.27–8.86
Belgium [8] 990 243 IST 24.55 [13] 0.78 1 31.47 28.16–35.03
Slovakia [9] 660 49 IST/SCT 7.4 [13] 0.78 1 9.52 7.26–12.41
Estonia [10] 17 5 SCT 29.4 [12] 0.885 1 33.23 15.01–60.04
Finland [11] 265 0 PCR 0 [11] 0.78 1 0 0–1.83
Ireland [11] 331 0 SCT 0 [12] 0.885 1 0 0–1.3
Great Britain [11] 434 0 PCR 0 [11] 0.85 1 0 0–1.03
Norway [11] 523 0 PCR 0 [11] 0.63 1 0 0–1.16
References: [1] – Karamon etal. (58); [2] – Bagrade etal. (59); [3] – Combes etal. (60); [4] – Berke etal. (61); [5] – Enemark etal. (62); [6] – Sréter etal. (63); [7] – Sikó etal. (64); [8] – Hanosset etal. (65); [9] – Bagrade etal. (59), 2001; [10] – Moks etal. (66); [11] –
European Food Safety Authority (50); [12]–(67); [13] – Hofer etal. (68).

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 14 frontiersin.org
When collecting data on alternative test types which are not yet in
use, it may beuseful to use proxies. Proxies can besimilar tests already
conducted for other pathogens, and hence already have internal costs
listed in the organization. Data on alternative tests may also befound
on supplier websites. Every test type will bedierent so it’s important to
avoid biases wherever possible. For example, if youare calculating costs
over a year and a piece of key equipment needs maintenance every
4 years, then this cost needs to be considered fairly: it should not
beignored but should also not beconsidered in full for a single year of
testing. A fair solution would beto divide this cost by the years between
maintenance activities to make it a normalized annual cost output.
Data for each testing type must be calculated per test and
multiplied by the required sample size based on the sensitivity of each
test. is can becalculated using the EpiTools online resource. Doing
so allows for direct comparison between the cost-eectiveness of each
test type.
Example: In the design section, in test costing, we used
hypothetical data as an example of the cost of the egg otation test for
E. multilocularis surveillance. An objective for this surveillance is to
ensure that the system uses a method that is practically and nancially
feasible. is can beconducted by comparing the costs of the current
testing method against the known surveillance budget. However, only
a comparison of multiple surveillance design options can optimize
value for money. For E. multilocularis weproduced a CEA comparing
the hypothetical costs of multiple testing methods; the egg otation
test, and two alternate methods identied in the sampling methods
section. When working with estimated costs, the CEA can beused
iteratively to generate a range of outputs or, if the upper and lower
bounds of cost data are known, then this can provide a minimum and
maximum cost for the surveillance.
Cataloging the other tests available was conducted through
discussions with the stakeholders and through literature research. e
annual EFSA report on E. multilocularis surveillance in Europe was
an essential resource, summarizing how each country in Europe was
conducting their tests, describing a range of alternative test-types (69).
We identied two alternative methods, the SCT and a real-time
PCR method. APHA conducts the SCT as part of the external quality
assurance and prociency testing schemes provided by the European
Union Reference Laboratory for Parasites (EURLP) for the detection
of Echinococcus spp. worms in intestinal mucosa. e instructions and
procedure provided by the EURLP for this testing was used to broadly
determine the consumables, reagents and equipment required for this
test (70). Prices per test were generated using hypothetical data. e
sta time spent processing samples, ‘lab time,’ was calculated using an
average sample throughput of 15 samples per day based on
information from literature (71). e additional time costs including
sample collection and post-mortems (‘non-lab time’) were assumed to
bethe same for all methods, and therefore are set at a blanket cost per
sample (hypothetical data).
e real time PCR method used in this evaluation is the QIAamp
Fast DNA Stool Mini Kit (QT) combined with a TaqMan PCR, the
method for which has been previously described in literature (72, 73).
A combination of this literature, and in-house SOPs were used to
populate a list of consumables, reagents, and equipment (74) which
were then assigned hypothetical costs.
e SOPs and information gathered for these tests were used to
create the consumables, reagents and equipment lists. Each component
was then assigned a hypothetical cost. Costs for two alternative
methods of testing previously identied were also produced based on
protocols found through literature searches, and the three methods
were compared in a cost eectiveness analysis (Table4). Hypothetical
values were also generated for sta time, sample transport and post-
mortems. All cost values were then added together to provide the
annual costs of maintaining a surveillance system using each test type,
including the costs for sample collection, post-mortem, testing, and
epidemiological services linked to the system.
FIGURE5
Plots generated by the EpiTools calculator showing predictions for dierent prevalence levels and population sizes for a specified test sensitivity.

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 15 frontiersin.org
e total annual cost for each testing methodology was
converted into a mean cost per test. e number of samples to
be taken was calculated using EpiTools, an online sample size
calculator developed by AUSVET (6) with the test sensitivity,
design prevalence, condence level and host population size as
inputs. Since positive results were assumed to befollowed up and
conrmed, the specicity of all tests was set to 1. e test sensitivity
of 0.78 for the zinc egg otation (EF) and SCT methods is the value
recommended for use by EFSA for this type of testing, whereas test
sensitivity for the qPCR method is the average of those sourced
from literature. From these data the qPCR is the most sensitive of
the testing methods with a sensitivity of 0.89.
e minimum number of tests required to detect a 1% prevalence
with 95% condence with the sensitivities specied by these tests was
then multiplied by the cost per test to provide the overall cost of each
testing methodology.
The costs of each methodology were compared. For annualized
costs, such as sample collection and post-mortem, the per test cost
was calculated based on the approximate number of samples
collected in GB for the sampling year 2021–2022: 800 (75). This
was multiplied by the number of tests required, determined using
the EpiTools calculator.
For this hypothetical scenario, the SCT is the most economical
when it comes to consumables and reagents, costing an estimated
€3.74 per test compared to the €12.48 and €56.88 required for the PCR
and EF, respectively. is is also true for the estimated annual cost of
equipment and maintenance, with the SCT requiring an estimated
€625.05 per year compared to €894.15 for the EF and €18.860.40 for
the PCR equipment. is dierence is mainly due to the comparatively
large maintenance cost for real time PCR equipment. Where these
outputs dier, however, is the cost of sta time associated with each
test. Weestimated the cost-per-test of both the EF and PCR at between
€9–11 whereas due to the time intensive nature of the SCT, the per
cost test was determined to be€17.57 based on sta processing an
average of 15 samples per day (71).
Overall, with this model the qPCR is shown to bethe most cost-
eective testing method due to its lower number of tests required
per year.
3.3.7. Propose improvements to the system (if
applicable)
Each evaluation from the previous section will have developed an
understanding of how well the surveillance system currently functions.
is may have highlighted areas where the surveillance system needs
improvement. Improvements do not necessarily mean increases in
testing output, but rather changes to the system that make it more
eective at achieving its objectives at the time of evaluation.
Examples of potential improvements include changes to test type
to increase cost-eectiveness or accuracy of surveillance, changes to
design prevalence to detect a higher or lower population prevalence
with greater condence or changes to sample number to better reect
the chosen design prevalence.
Any proposed improvements to the system constitute a change to
the design proposal of the surveillance system. Hence, it may
benecessary to go through the stages of design and implementation
to ensure improvements are properly considered from all angles by the
relevant stakeholders.
4. Discussion
Output-based standards can allow for variation in surveillance
activities to achieve a universal objective and may beuseful in the OH
context where surveillance for animal pathogens can act as risk
indicators for human health. In addition to the context of zoonotic
pathogens, OBS may also beuseful in other One Health Scenarios, for
example in detecting a bacterial hazard at a particular design
prevalence in a food product.
In the design section of this framework, werecommend a robust
method of objective setting and highlight this as a reference point for
all subsequent activities in the framework. Wealso emphasize the
importance of identifying all the stakeholders acting within the OBS
system and demonstrate how stakeholder engagement can guide the
design of successful surveillance systems with their expertise and
knowledge. Werecommend the EpiTools calculator for determining
sample size (29) in our worked examples. Later in the design section,
wedescribe a method for estimating the costs of the available test
TABLE4 Showing the cost-eectiveness of three dierent testing methodologies for E. multilocularis at detecting a 1% prevalence detection with 95%
confidence (hypothetical data).
Parameter Unit Test
Egg flotation SCT qPCR
Species sampled – Fox Fox Fox
roughput –Batch of 20 every 12 h 10–20 per day (Average 15) 12–30 min per sample
(Average 21)
Test sensitivity – 0.78 0.78 0.89
Test specicity – 1 1 1
Consumables and reagents Per test €56.88 €3.74 €12.48
Sta time (testing) Per test €9.26 €17.57 €10.32
Operational costs (excluding testing) Annual cost (800 tests) €291,593.12 €291,593.12 €291,593.12
Equipment Annual cost €894.15 €625.05 €18,860.40
Tests required at 1% prevalence No. of tests 383 383 336
Cost of testing at 1% prevalence € €165,823.53 €150,408.31 €148,989.54

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 16 frontiersin.org
options, helping predict the feasibility of implementing the chosen test
within the available surveillance budget.
In the implementation section, weshow how systems mapping
can be used to visualize the steps and stakeholders involved in
surveillance, facilitating clear communication of the intended system
design to all relevant stakeholders from an early stage. Later,
wehighlight the importance of le shi and operational risk analysis
to eective project implementation.
e evaluation section described in this framework rst
establishes whether the stated objectives of the system are still relevant
to the contemporary disease and legislative context. en, the
exibility of the system to adaptation and change is analyzed to
provide a holistic view of the relationships between system
components and the system’s capacity for change. By applying
technical evaluation tools such as EpiTools, wecan assess whether the
chosen prevalence estimations and sample sizes remain accurate to the
true disease situation. is provides an indication of whether
individual surveillance streams should beupscaled or downscaled to
meet the required output of the system. Along with a technical
performance assessment, this guidance provides advice on how to
evaluate the human factors within the system through stakeholder
evaluation. Financial viewpoints are considered in the cost-
eectiveness analysis section. is provides an example evaluation
method for multiple testing options. In completing the full evaluation,
the technical, human, economic, and practical elements of the system
can bevisualized in the wider context of the current disease situation.
However, there are limitations to some of the analyses described.
For example, because of the variation across laboratories, countries,
and sectors, the CEA did not consider the implementation costs of
changing the testing type used. ese are the additional costs required
to move from one testing type to another, including the cost of
retraining sta, and purchasing new equipment. Including
implementation costs would provide a better understanding of the real
costs of applying dierent test types. Any future expansions to this
work could integrate the payback times for dierent tests following
initial investment in them over a temporal dimension. is could say,
for example, that moving to a PCR and fecal sample-based testing
regime, while it would cost £3 M investment, would pay itself back in
savings from reduced year-on-year sample collection and material
costs in 10 years. Under this framework it was not possible to quantify
the implementation costs of new training and equipment without
knowing the existing laboratory capacity. us, to keep the analysis
generic to a range of end-users, this aspect was not included.
Additionally, because this guidance is designed for OBS systems
only, the recommendations it provides are more tailored than other
surveillance evaluation tools such as SERVAL and RISKSUR EVA,
which are generic to all forms of surveillance (4, 5). Its narrower scope
provided an opportunity to ground this framework to worked
examples that highlight immediate practical recommendations rather
than top-level areas for improvement. However, weacknowledge that
some elements of the framework may beprescriptive.
For instance, EpiTools is referenced throughout the guidance,
without consideration of other epidemiological calculators. e
calculator by Iowa state university, for instance, could equally beused
for sample size and probability of detection calculation (76).
We chose EpiTools for the examples because of its broad range of
available analysis applications, including sample size estimations
using both hypergeometric and binomial approaches and true
prevalence estimations using Bayesian and pooled computational
approaches. is range of analyses makes it applicable to OBS systems
with large or small population sizes, and with a broad design
prevalence range. In addition, the tool is free and has had usage
across several published articles, making it readily accessible to
analysts from a range of backgrounds (77–79).
Many of the ideas in the implementation section of this framework
are tied to systems engineering practices. ese have a good track
record of use across a range of science and technology-focused
projects (19, 80). However, several analyses in this framework could
beconducted dierently. For example, while causal loop diagrams
have been used in a wide range of disciplines to represent dynamic
systems (23, 44), analysts could equally use retrospective approaches
for exibility analysis as in the guidelines for evaluating public health
surveillance systems produced by the United States Centers for
Disease Control (81). Wealso acknowledge that not all sections of this
framework will berelevant to all users and that, depending on the
context of its users, there may begaps that require additional research.
is is expected given the broad scope of OBS in dierent situations,
and as such this guidance should be considered alongside other
training and literature from other sources. Nevertheless, webelieve
that the approaches described here encourage a holistic outlook on
OBS systems throughout. Above all, they encourage extensive
stakeholder engagement, not only with end users, but also with
delivery and governance teams. We hope this framework will
encourage cross-disciplinary implementations of OBS systems and
thereby improve their performance and sustainability.
In summary, this framework provides a range of relevant activities
and recommendations for the design, implementation, and evaluation
of output-based surveillance systems. It is a holistic toolkit with
applications from setting the objectives of a new system to analyzing
the cost-performance of an established system. Not all sections will
beapplicable to all end users. However, its promotion of systems
thinking, and stakeholder participation makes it a valuable tool in the
cross-disciplinary implementation of OBS.
Data availability statement
e original contributions presented in the study are included in
the article/Supplementary material, further inquiries can be directed
to the corresponding author.
Author contributions
SR wrote the rst dra of the manuscript. SR, MK, RD, AS, AZ-B,
and VH wrote sections of the manuscript. All authors contributed to
the article and approved the submitted version.
Funding
is work, carried out as part of the MATRIX project (Promoting
One Health in Europe through joint actions on foodborne zoonoses,
antimicrobial resistance and emerging microbiological hazards) has
received funding from the European Union’s Horizon 2020 research and
innovation programme under Grant Agreement No 773830 and from
the Department for Environment, Food and Rural Aairs (DEFRA)
UK. Research at the National Veterinary Research Institute (PIWet),

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 17 frontiersin.org
Poland, was also partially supported by the Polish Ministry of Education
and Science from the funds for science in the years 2018–2022 allocated
for the implementation of co-nanced international projects.
Acknowledgments
We acknowledge the important contributions made by Julia Coats
and Izzy Rochester from the APHA wildlife team helping us establish
key aspects of the sample collection pipeline for red foxes in Great
Britain, and the insights provided by Richard Budgey from the APHA
wildlife modeling team. Weacknowledge Steve Campbell (SASA) and
Libby Barnett (Fera Science) for providing insight into the external
uses for red fox samples by scientic collaborators. We also
acknowledge the valuable work of Mirek Rozycki (PIWET) in the
initial direction-setting of this project.
Conflict of interest
e authors declare that the research was conducted in the
absence of any commercial or nancial relationships that could
beconstrued as a potential conict of interest.
Publisher’s note
All claims expressed in this article are solely those of the
authors and do not necessarily represent those of their aliated
organizations, or those of the publisher, the editors and the
reviewers. Any product that may be evaluated in this article, or
claim that may be made by its manufacturer, is not guaranteed or
endorsed by the publisher.
References
1. Bordier M, Uea-Anuwong T, Binot A, Hendrikx P, Goutard F. Characteristics of
One Health surveillance systems: A systematic literature review. Prev Vet Med. (2018)
181. doi: 10.1016/j.prevetmed.2018.10.005
2. Cameron AR. (2012) e consequences of risk-based surveillance: Developing
output-based standards for surveillance to demonstrate freedom from disease. Prev Vet
Med. (2012) 105:280–6. doi: 10.1016/j.prevetmed.2012.01.009
3. Meletis , et al (2022) Output-based methodological approaches for
substantiating freedom from infection. Accessed 2022, Publisher/Project: SOUND
Control, https://sound-control.eu/wp-content/uploads/2021/12/Deliverable-3.1.-
Output-based-methodological-approaches-for-substantiating-freedom-from-
infection.pdf
4. Drewe J, Hoinville L, Cook A, Floyd T, Gunn G, Stärk K. SERVAL: a new framework
for the evaluation of animal health surveillance. Transbound Emerg Dis. (2015) 62:33–45.
doi: 10.1111/tbed.12063
5. Peyre M, Hoinville L, Njoroge J, Cameron A, Traon D, Goutard F, et al. e
RISKSUR EVA tool (Survtool): a tool for the integrated evaluation of animal health
surveillance systems. Prev Vet Med. (2019) 173:104777. doi: 10.1016/j.
prevetmed.2019.104777
6. Ausvet. Sample size to achieve specied population level (or herd, ock, cluster, etc)
sensitivity. AUSVET. (2022). Available at: https://epitools.ausvet.com.au/freedomss
(Accessed May 23, 2022)
7. Henaux V, Tegegne H, Bogaardt C, Lailler R, Collineau L, Prada J. Deliverable
D-JIP-MATRIX-WP4.2 OH-EpiCap tool and tutorial. Zenodo. (2022)
8. Costa L, Duarte EL, Knic T, Hodnik JJ, van Roon A, Fourichon C, et al.
Standardizing output-based surveillance to control non-regulated cattle diseases:
aspiring for a single general regulatory framework in the European Union. Prev Vet Med.
(2020) 183:105130. doi: 10.1016/j.prevetmed.2020.105130
9. Horigan V. Inventory of previous work and current practice using output based
standards. Deliverable D-WP3.2: Guidelines for the design, implementation, and
evaluation of ocial controls within the food sector using output-based standards (OBS)
Zenodo (2022).
10. Alban L, Rugbjerg H, Petersen JV, Nielsen LR. Comparison of risk-based versus
random sampling in the monitoring of antimicrobial residues in Danish nishing pigs.
Prev Vet Med. (2016) 128:87–94. doi: 10.1016/j.prevetmed.2016.04.007
11. Foddai A., Nielsen L. R., Willeberg P., Alban L. (2015). Comparison of output-based
approaches used to substantiate bovine tuberculosis free status in Danish cattle herds, 121,
21–29 doi: 10.1016/j.prevetmed.2015.05.005
12. Reber A, Reist M, Schwermer H. Cost-eectiveness of bulk-tank milk testing for
surveys to demonstrate freedom from infectious bovine rhinotracheitis and bovine
enzootic leucosis in Switzerland. Schweizer Archiv Fur Tierheilkunde. (2012) 154:189–97.
doi: 10.1024/0036-7281/a000329
13. Rutten N, Gonzales JL, Elbers AR, Velthuis AG. Cost analysis of various low
pathogenic avian inuenza surveillance systems in the Dutch egg layer sector. PLoS One.
(2012) 7:e33930. doi: 10.1371/journal.pone.0033930
14. Foddai A, Floyd T, Mcgiven J, Grace K, Evans S. Evaluation of the English bovine
brucellosis surveillance system considering probability of disease introduction and non-
random sampling. Prev Vet Med. (2020) 176:104927. doi: 10.1016/j.
prevetmed.2020.104927
15. Mastin AJ, Gottwald TR, Van Den Bosch F, Cunnie NJ, Parnell S. Optimising
risk-based surveillance for early detection of invasive plant pathogens. PLoS Biol. (2020)
18:e3000863. doi: 10.1371/journal.pbio.3000863
16. de Vos CJ, Saatkamp H, Huirne R. Cost-eectiveness of measures to prevent
classical swine fever introduction into the Netherlands. Prev Vet Med. (2005) 70:235–56.
doi: 10.1016/j.prevetmed.2005.04.001
17. Hadorn DC, Racloz V, Schwermer H, Stärk KD. Establishing a cost-eective
national surveillance system for bluetongue using scenario tree modelling. Vet R es .
(2009) 40:1–14.
18. Dewar R. (2021). System analysis of detection of outbreaks. COHESIVE: One
Health Structure In Europe, Deliverable-D-JIP2-D3.5.
19. Emes MR, Smith A, James AM, Whyndham MW, Leal R, Jackson SC. 8.1.2
principles of systems engineering management: reections from 45 years of spacecra
technology research and development at the Mullard space science laboratory. INCOSE
Int Symp. (2012) 22:1069–84.
20. Mendelow A. L. (1981). Environmental scanning--the impact of the stakeholder
concept. ICIS 1981 Proceedings. 20, https://aisel.aisnet.org/icis1981/20
21. COHESIVE consortium Cohesive One health: setting up a risk analysis system for
zoonoses. (2022). Available at: https://www.ohras.eu/page/home (Accessed May 17,
2022).
22. Dewar R. (2021). Review of economic analyses conducted on foodborne Zoonoses.
DeliverableD-JIP2-3.8.0.
23. Tip T. Guidelines for drawing causal loop diagrams. Systems inker. (2011)
22:5–7.
24. Stevenson M., Sergeant E. (2022). Design and analysis of disease surveillance
programs using epiR. Available at: https://cran.r-project.org/web/packages/epiR/
vignettes/epiR_surveillance.html (Accessed October 24, 2022).
25. Paoli BHL, Shah G. Condence intervals in public health Oce of Public Health
Assessment, Utah Department of Health (2002).
26. Rahmatian S. e hierarchy of objectives: toward an integrating construct in
systems science. Syst Res. (1985) 2:237–45. doi: 10.1002/sres.3850020307
27. EFSA (European Food Safety Authority)Beltran-Beck B, Zancanaro G. Scientic
report on the assessment of Echinococcus multilocularis surveillance reports submitted
in2017in the context of Commission Regulation (EU) No 1152/2011. EFSA Journal
2017. (2017) 15:5051–75. doi: 10.2903/j.efsa.2017.5051
28. Friedman A, Miles S. Stakeholders: eory and Practice. UK: Oxford University
Press (2006).
29. Sergeant E. Epitools epidemiological calculators. (2018). Canberra, Australia.
Available at: http://epitools.ausvet.com.au.
30. Mathews F, Kubasiewicz L, Gurnell J, Harrower C, Mcdonald RA, Shore R. A
review of the population and conservation status of British mammals: Technical summary.
Natural England (2018).
31. European Food Safety Authority. Annual assessment of Echinococcus multilocularis
surveillance reports submitted in 2020in the context of commission delegated regulation
(EU) 2018/772. EFSA J. (2021) 19:e06382. doi: 10.2903/j.efsa.2021.6382
32. European Food Safety Authority. Annual assessment of Echinococcus multilocularis
surveillance reports submitted in 2021in the context of commission delegated regulation
(EU) 2018/772. EFSA J. (2021) 19:e06945. doi: 10.2903/j.efsa.2021.6945
33. European Food Safety Authority. A framework to substantiate absence of disease:
the risk based estimate of system sensitivity tool (RiBESS) using data collated according
to the EFSA standard sample description – an example on Echinococcus multilocularis.
EFSA. (2012) 9:366E.

Rivers et al. 10.3389/fpubh.2023.1129776
Frontiers in Public Health 18 frontiersin.org
34. Emes M., Smith A., James A. (2014). ‘Le-shi’ vs the time value of money.
European Food Safety Authority, Italy.
35. Risksur Consortium. Risk-based animal health surveillance systems framework.
(2015). Available at: https://www.fp7-risksur.eu/ (Accessed March 08, 2023).
36. Weilkiens T. Chapter 1: e V model. Systems engineering with SysML/UML:
Modeling, analysis, design Elsevier (2011). USA: Morgan Kaufmann.
37. Davison N. (2016). e change control process. In: Resources, U. P. M. (ed.).
38. Devi TR, Reddy VS. Work breakdown structure of the project. Int J Eng Res Appl.
(2012) 2:683–6.
39. Lester A. Project management, planning and control: Managing engineering,
construction and manufacturing projects to PMI, APM and BSI standard s Elsevier (2006).
40. Wilson JM. Gantt charts: a centenary appreciation. Eur J Oper Res. (2003)
149:430–7. doi: 10.1016/S0377-2217(02)00769-5
41. Lavanya N, Malarvizhi T. Risk analysis and management: a vital key to eective
project management. PMI® global congress. Sydney: Project Management Institute
(2008).
42. Institution Of Civil Engineers. Risk analysis and management for projects ICE
Publishing (2002).
43. Kobusingye B, Mungatu J, Mulyungi P. Inuence of stakeholders involvement on
project outcomes. A case of water, sanitation, and hygiene (wash) project in Rwanda.
Eur J Bus Soc Sci. (2017) 6:195–206.
44. Haraldsson HV. Introduction to system thinking and causal loop diagrams. Lund:
Department of Chemical Engineering, Lund University (2004).
45. APHA carcass collection coordinator Interview with APHA carcass collection
coordinator. (2022).
46. Macdiarmid S. Future options for brucellosis surveillance in NewZealand beef
herds. N Z Vet J. (1988) 36:39–42. doi: 10.1080/00480169.1988.35472
47. e Forest Data Bank. (2022). Available at: https://www.bdl.lasy.gov.pl/portal/en
48. Kirjušina M, Deksne G, Marucci G, Bakasejevs E, Jahundoviča I, Daukšte A, et al.
A 38-year study on Trichinella spp. in wild boar (Sus scrofa) of Latvia shows a stable
incidence with an increased parasite biomass in the last decade. Parasit Vectors. (2015)
8:137–8. doi: 10.1186/s13071-015-0753-1
49. Danish Centre for Environment and Energy. (2022). Available at: https://dce.
au.dk/en/
50. European Food Safety Authority. Assessment of Echinococcus multilocularis
surveillance reports submitted in 2015in the context of commission regulation (EU) no
1152/2011. EFSA J. (2015) 13:4310. doi: 10.2903/j.efsa.2015.4310
51. Şuteu O, Mihalca AD, Paştiu AI, Györke A, Matei IA, Ionică A, et al. Red foxes
(Vulpes vulpes) in Romania are carriers of toxoplasma gondii but not Neospora caninum.
J Wildl Dis. (2014) 50:713–6. doi: 10.7589/2013-07-167
52. Romanian National Institute of Statistics. (2008). Available at: http://www.insse.
ro/cms/les/statistici/comunicate/com_anuale/fond%20cinegetic/Fondcinegetic_2008.
pdf
53. Kauhala K. (2007). Paljonko Suomessa on pienpetoja?
54. Hayden TJ, Harrington R. Exploring Irish mammals Town House (2000).
55. Marnell F, Kingston N, Looney D. Ireland red list no. 3: Terrestrial mammals.
Dublin: National Parks and Wildlife Ser vice, Department of the Environment, Heritage
and Local Government (2009).
56. Defra Department For Environment, F. R. A. & Agency, A. A. A. P. H. (2019).
Echinococcus multilocularis: How to spot and report the disease. Available at: https://
www.gov.uk/guidance/echinococcus-multilocularis-how-to-spot-and-report-the-
disease (Accessed September 30, 2022).
57. Sviland S, Johansen T, K levar S, Valheim M, Jonsson M. Surveillance programmes
for terrestrial and aquatic animals in Norway. Annu Rep. (2014)
58. Karamon J, Kochanowski M, Sroka J, Cencek T, Różycki M, Chmurzyńska E, et al.
e prevalence of Echinococcus multilocularis in red foxes in Poland—current results
(2009–2013). Parasitol Res. (2014) 113:317–22. doi: 10.1007/s00436-013-3657-z
59. Bagrade G, Šnábel V, Romig T, Ozoliņš J, Hüttner M, Miterpáková M, et al.
Echinococcus multilocularis is a frequent parasite of red foxes (Vulpes vulpes) in Latvia.
Helminthologia. (2008) 45:157–61. doi: 10.2478/s11687-008-0032-1
60. Combes B, Comte S, Raton V, Raoul F, Boué F, Umhang G, et al. Westward spread
of Echinococcus multilocularis in foxes, France, 2005-2010. Emerg Infect Dis. (2012)
18:2059–62. doi: 10.3201/eid1812.120219
61. Berke O, Romig T, Von Keyserlingk M. Emergence of Echinococcus multilocularis
among red foxes in northern Germany, 1991–2005. Vet Parasitol. (2008) 155:319–22.
doi: 10.1016/j.vetpar.2008.05.017
62. Enemark HL, Al-Sabi MN, Knapp J, Staahl M, Chríel M. Detection of a high-
endemic focus of Echinococcus multilocularis in red foxes in southern Denmark, January
2013. Euro Surveill. (2013) 18:20420.
63. Sréter T, Széll Z, Sréter-Lancz Z, Varga I. Echinococcus multilocularis in northern
Hungary. Emerg Infect Dis. (2004) 10:1344–6. doi: 10.3201/eid1007.031027
64. Sikó SB, Deplazes P, Ceica C, Tivadar CS, Bogolin I, Popescu S, et al. Echinococcus
multilocularis in South-Eastern Europe (Romania). Parasitol Res. (2011) 108:1093–7.
doi: 10.1007/s00436-010-2150-1
65. Hanosset R, Saegerman C, Adant S, Massart L, Losson B. Echinococcus
multilocularis in Belgium: prevalence in red foxes (Vulpes vulpes) and in dierent species
of potential intermediate hosts. Vet Parasitol. (2008) 151:212–7. doi: 10.1016/j.
vetpar.2007.09.024
66. Moks E, Saarma U, Valdmann H. Echinococcus multilocularis in Estonia. Emerg
Infect Dis. (2005) 11:1973–4. doi: 10.3201/eid1112.050339
67. Otero-Abad B, Armua-Fernandez MT, Deplazes P, Torgerson PR, Hartnack S.
Latent class models for Echinococcus multilocularis diagnosis in foxes in Switzerland in
the absence of a gold standard. Parasit Vectors. (2017) 10:1–14.
68. Hofer S, Gloor S, Müller U, Mathis A, Hegglin D, Deplazes P. High prevalence of
Echinococcus multilocularis in urban red foxes (Vulpes v ulpes) and voles (Arvicola
terrestris) in the city of Zürich. Switzerland Parasitol. (2000) 120:135–42. doi: 10.1017/
S0031182099005351
69. EFSA (European Food Safety Authority) and Zancanaro G. Annual assessment of
Echinococcus multilocularis surveillance reports submitted in 2021in the context of
Commission Delegated Regulation (EU) 2018/772. EFSA Journal 2021. (2021) 19:57.
doi: 10.2903/j.efsa.2021.6945
70. European Union Reference Laboratory for Parasites (2022) Prociency testing,
Istituto Superiore di Sanità. https://www.iss.it/en/web/iss-en/eurlp-prociency-testing
(Accessed October, 2022).
71. Eckert J. Predictive values and quality control of techniques for the diagnosis of
Echinococcus multilocularis in denitive hosts. Acta tropica. (2003) 85:157–63. doi:
10.1016/S0001-706X(02)00216-4
72. Knapp J, Millon L, Mouzon L, Umhang G, Raoul F, Zeinaba S, et al. Real time PCR
to detect the environmental faecal contamination by E chinococcus multilocularis from
red fox stools. (2014).
73. Maksimov P, Isaksson M, Schares G, Romig T, Conraths F. Validation of dierent
PCR-based protocols for the detection of Echinococcus multilocularis DNA in the nal
host using the Intestinal Scraping Technique as a reference. Food and Waterborne
Parasitology. (2019) 15:e00044. doi: 10.1016/j.fawpar.2019.e00044
74. Central Unit for Sequencing and PCR (CUSP) RE:qPCR reagents costs. (2022).
75. APHA parasitology lead Interview with APHA parasitology lead. (2022).
76. Iowa State University. Sample size calculators. (2022). Available at: https://eldepi.
research.cvm.iastate.edu/calc/ (Accessed October 26, 2022).
77. Charan J, Kantharia N. How to calculate sample size in animal studies? J Pharmacol
Pharmacother. (2013) 4:303–6. doi: 10.4103/0976-500X.119726
78. Laurin E, Bradshaw J, Hawley L, Gardner IA, Garver K, Johnson SC, et al.
Importance of sample size for estimating prevalence: a case example of infectious
hematopoietic necrosis viral RNA detection in mixed-stock Fraser River sockeye salmon
(Oncorhynchus nerka), British Columbia, Canada. Can J Fish Aquat Sci. (2021)
78:589–98. doi: 10.1139/cjfas-2020-0279
79. Villarta, Jr RL, Asaad AS. Sample size determination in an epidemiologic study
using the EpiTools web-based calculator. Act a Med Philipp. (2014) 48. doi: 10.47895/
amp.v48i1.1192
80. Leal R. (2020). e foundations of systems engineering. In: Engineering, U. C. F. S.
(Ed.). London.
81. German RR, Horan JM, Lee LM, Milstein B, Pertowski CA. Updated guidelines for
evaluating public health surveillance systems; recommendations from the guidelines
working group US CDC (2001).