Hazard Screening Methods for Nanomaterials: A Comparative Study

Abstract

Hazard identification is the key step in risk assessment and management of manufactured nanomaterials (NM). However, the rapid commercialisation of nano-enabled products continues to out-pace the development of a prudent risk management mechanism that is widely accepted by the scientific community and enforced by regulators. However, a growing body of academic literature is developing promising quantitative methods. Two approaches have gained significant currency. Bayesian networks (BN) are a probabilistic, machine learning approach while the weight of evidence (WoE) statistical framework is based on expert elicitation. This comparative study investigates the efficacy of quantitative WoE and Bayesian methodologies in ranking the potential hazard of metal and metal-oxide NMs—TiO2, Ag, and ZnO. This research finds that hazard ranking is consistent for both risk assessment approaches. The BN and WoE models both utilize physico-chemical, toxicological, and study type data to infer the hazard potential. The BN exhibits more stability when the models are perturbed with new data. The BN has the significant advantage of self-learning with new data; however, this assumes all input data is equally valid. This research finds that a combination of WoE that would rank input data along with the BN is the optimal hazard assessment framework.

Full text

International Journal of
Molecular Sciences
Article
Hazard Screening Methods for Nanomaterials:
A Comparative Study
Barry Sheehan 1, *ID , Finbarr Murphy 1, Martin Mullins 1, Irini Furxhi 1, Anna L. Costa 2,
Felice C. Simeone 2and Paride Mantecca 3ID
1Department of Accounting and Finance, University of Limerick, V94PH93 Limerick, Ireland;
finbarr.murphy@ul.ie (F.M.); martin.mullins@ul.ie (M.M.); irini.furxhi@ul.ie (I.F.)
2Institute of Science and Technology for Ceramics (CNR-ISTEC), National Research Council of Italy, Via
Granarolo 64, 48018 Faenza (RA), Italy; anna.costa@istec.cnr.it (A.L.C.); felice.simeone@istec.cnr.it (F.C.S.)
3
Department of Earth and Environmental Sciences, Particulate Matter and Health Risk (POLARIS) Research
Centre, University of Milano Bicocca, 20126 Milano, Italy; paride.mantecca@unimib.it
*Correspondence: barry.sheehan@ul.ie; Tel.: +353-61-213-188
Received: 30 January 2018; Accepted: 15 February 2018; Published: 25 February 2018
Abstract:
Hazard identification is the key step in risk assessment and management of manufactured
nanomaterials (NM). However, the rapid commercialisation of nano-enabled products continues to
out-pace the development of a prudent risk management mechanism that is widely accepted by the
scientific community and enforced by regulators. However, a growing body of academic literature
is developing promising quantitative methods. Two approaches have gained significant currency.
Bayesian networks (BN) are a probabilistic, machine learning approach while the weight of evidence
(WoE) statistical framework is based on expert elicitation. This comparative study investigates the
efficacy of quantitative WoE and Bayesian methodologies in ranking the potential hazard of metal and
metal-oxide NMs—TiO
2
, Ag, and ZnO. This research finds that hazard ranking is consistent for both
risk assessment approaches. The BN and WoE models both utilize physico-chemical, toxicological,
and study type data to infer the hazard potential. The BN exhibits more stability when the models
are perturbed with new data. The BN has the significant advantage of self-learning with new data;
however, this assumes all input data is equally valid. This research finds that a combination of WoE
that would rank input data along with the BN is the optimal hazard assessment framework.
Keywords:
nanomaterials; hazard assessment; Bayesian network; weight of evidence; multi-criteria
decision analysis; human health hazard screening
1. Introduction
Hazard identification is a primary step in the risk assessment of engineered nanomaterials
(NM) [
1
,
2
]. Four decades have passed since Norio Taniguchi first coined the term “nanotechnology” [
3
],
and hazard assessment remains a continuous research effort to support the development and
commercialization of nanomaterials [
4
]. A consensus acceptance of hazard and risk assessment
methodologies is essential in order to agree on accepted risk reduction measures for NM [
5
].
Effective risk communication between stakeholders is necessary for the sustainable growth of the
nanotechnology industry [
6
]. Notwithstanding this, the rapid commercialisation of nano-enabled
products continues to out-pace the development of a prudent risk management mechanism that is
accepted by the scientific community and enforced by regulators. The good news is that a growing
body of academic literature is contributing to the development of increasingly accurate quantitative
risk assessment methods, but a validated, replicable and transparent hazard identification tool remains
elusive. This paper represents a valuable addition to this literature set as it seeks to identify suitable
methodologies to contend with the complex nature of NM hazard identification.
Int. J. Mol. Sci. 2018,19, 649; doi:10.3390/ijms19030649 www.mdpi.com/journal/ijms

Int. J. Mol. Sci. 2018,19, 649 2 of 22
Recently, Bayesian methodologies have been gaining support in the context of NM risk
assessment [
7
–
10
], whilst more established weight of evidence (WoE) based frameworks have been
criticized for being overly reliant on expert judgement and qualitative data [
11
]. There remain, however,
significant deficiencies and inconsistencies in key experimental results required to facilitate conclusive
risk management decision-making [
4
,
12
,
13
]. An intermediate approach of appending scientific expert
opinion to real-world NM physico-chemical, biological, and toxicological data to determine NM hazard
potential has offered some degree of success. Both Bayesian networks (BN) and quantitative WoE
methods have been proposed as effective frameworks in achieving this task [
9
,
14
]. This paper offers
a timely comparison of both methodologies allowing for a meaningful comparison of both results
and performance.
Hazard screening (or ranking) is a method used to benchmark the intrinsic hazard potential
of several NMs against one another [
15
]. Expensive and time-consuming toxicological testing has
resulted in a concentration of focus towards specific NMs. Relative hazard screening can therefore be
used to read-across experimentally demonstrated adverse effects for a specific NM to one with similar
physico-chemical characteristics and little experimental evidence in terms of hazard potential. Through
this benchmarking approach, proactive risk management may be inferred by enforcing occupational
exposure limits (OEL) for NMs akin to their relative hazard score.
BNs are probabilistic hierarchical models that, given a dataset, express probabilistic causal
relationships (i.e., conditional probabilities) between the different parameters [
16
]. The chain of
influences between parameters can be rendered graphically by linking nodes (i.e., parameters) by
one-way directed links that determine the nature of the causal dependencies. Each individual
node has a finite set of mutually exclusive states, with each state described by a probabilistic
expression determined by empirical relationships, mechanistic descriptions, or expert judgement [
17
].
BN probabilistic models are suited to NM hazard identification through their ability to capture
heterogeneous datasets that may contain missing, or conflicting, information. The model is particularly
suited to problems with limited data through its ability to iteratively refine forecasts as new information
becomes available. The NM hazard ranking tool proposed by Marvin et al. [
9
] applied Bayesian
network (BN) construction, parameterisation, and uncertainty analysis to metal and metal-oxide NMs.
This BN tool showed high accuracy, with 72% hazard prediction precision in an out-of-sample test.
WoE represents a diverse collection of methods used to synthesise and evaluate individual lines
of evidence (LOE) to form a conclusion [
14
,
18
]. WoE approaches have been classified by the degree of
quantitative criteria incorporated to deduct decisions [
19
]. These methods range from basic qualitative
assessment in the form of listing evidence to fully quantitative procedures which include statistical
methods or multi-criteria decision analysis (MCDA) [
19
]. Hristozov et al. [
14
] developed the first
quantitative MCDA approach for human health hazard screening of NMs, and illustrated the approach
using a nano-TiO
2
case study. A logic WoE methodology was complemented with quantitative MCDA
to produce a “hazard score” for nano-TiO
2
which may be compared to those of other nanomaterials
for hazard ranking.
In this article, a quantitative WoE with MCDA framework was applied for metal and metal-oxide
NMs—TiO
2
, Ag, ZnO. The resulting hazard rankings were compared to those demonstrated via the
BN application in Marvin et al. [
9
]. The results of both methods were also tested for sensitivity to
input variables, and the validation of results was demonstrated for the BN and examined for the
WoE method. The sources of information are the same for both the quantitative WoE framework and
the BN allowing for a comparative analysis. This is the first study that compares the relative hazard
rankings of NMs using separate assessment tools with the same reference literature. This is also the
first application of the quantitative WoE tool used to rank the hazard potential of several NMs.
2. Materials and Methods
This article examines the BN hazard ranking tool made available and described by
Marvin et al.
[
9
].
Furthermore, the quantitative WoE with MCDA methodology is replicated from
Hristozov et al.
[
14
].

Int. J. Mol. Sci. 2018,19, 649 3 of 22
Hence, a concise account of both model formulations is presented in this section. Detailed
descriptions are provided where the methodologies are adapted or extended for the purposes of
this comparative analysis.
A quantitative WoE with multi-criteria decision analysis (MCDA) hazard ranking model is
demonstrated for TiO
2
, Ag and ZnO. The resultant hazard ranking of the WoE is contrasted to that
of the BN constructed in Marvin et al. [
9
] in both normal and stressed states by means of sensitivity
and uncertainty analysis. The sensitivity of the hazard potential for each NM to input variables is
investigated for both BN and WoE methods. The accuracy of the hazard prediction is tested with a
cross-validation analysis.
2.1. Data
Marvin et al. [
9
] gathered physico-chemical and toxicity data of metal and metal-oxide NMs from
studies reported in the scientific literature in the period of 2009–2015. In total, 32 scientific articles
were used resulting in 559 cases or “lines of evidence” (LOE) containing data which may influence the
hazard potential of NMs. For the purposes of this comparative analysis, the literature for TiO
2
, Ag,
and ZnO are investigated due to their prolificacy in the database. This represents 48% (or 225 cases) of
the total data.
For the quantitative WoE method, 26 of the 32 peer-reviewed articles were analysed with respect
to the information provided on physico-chemical properties, toxicity, and data quality as per the
REACH requirements [
20
]. The 6 remaining papers were omitted from the analysis because they did
not reference the NMs being examined. The next sections detail the methods used to construct and
evaluate both the BN and quantitative WoE hazard ranking tools. The full list of literature is provided
in Appendix A.
2.2. Bayesian Network Methodology
The process of building a BN consists of three steps: (i) node (or variable) identification,
(ii) establish directed links for a causal network, and (iii) determine the conditional probability tables
(CPTs) [
16
]. In the context of NM hazard assessment, the most relevant physico-chemical characteristics
and biological effects are selected as nodes via expert elicitation processes. Furthermore, the initial
causal structure and parameterisation of the CPTs is determined by two rounds of expert consultation.
Using the 559 cases derived from the literature data, the expectation-maximization machine learning
algorithm is used to further refine and optimize the causal structure and conditional probabilities of
the BN. The Hugin 8.5 software is used to construct and learn the BN.
For the purposes of this paper, the validation of the BN and sensitivity analysis is performed
specifically with respect to TiO
2
, Ag, and ZnO. To test the hazard prediction accuracy of the BN
tool, an out-of-sample test is carried out against 41 cases omitted from the network structure and
parameterisation learning procedure. This comprised of inputting the physico-chemical parameters of
each case as evidence into the BN and comparing the predicted NM hazard (the most likely state with
the highest % probability) to the true value of observed NM hazard determined from the literature.
Two methods of sensitivity analysis are performed on the BN. The first of which is a value of
information analysis, which uses the entropy function to measure the sensitivity of the hypothesis
variable, NM hazard, to the other nodes within the BN [
21
,
22
]. The entropy
H(X)
(measure of
randomness) of a discrete random variable Xis defined as:
H(X)=−∑XP(X)·log P(X), (1)
where P(X)is the probability distribution of X.
This analysis ranks the physico-chemical properties, administration route, and study type
variables in order of influence on the NM hazard node. Next, a scenario analysis is performed to assess
the sensitivity of the order of hazard ranking to changes in NM physico-chemical characteristics.

Int. J. Mol. Sci. 2018,19, 649 4 of 22
2.3. Quantitative Weight of Evidence Methodology
Following the methodology of Hristozov et al. [
14
], the hazard of each LOE is evaluated based on
three criterion: NM physico-chemical properties, toxicity, and data quality. Each study is considered
a single LOE unless multiple experimental results are observed. The model follows a Logic method,
where each LOE is evaluated according to a set decision steps comprehensively described in Hristozov,
Zabeo, Foran, Isigonis, Critto, Marcomini and Linkov [14], and briefly summarised below.
1.
LOE index values based on physico-chemical properties: Physico-chemical criterion (BET
surface area, primary particle size, aspect ratio, surface coating,
ζ
-potential, purity, composition,
bioaccumulation) are evaluated according a state-specific scoring system in the [0,100] range.
These discretised states, or classes, refer to the segregation of the criteria into their components of
increased/decreased hazard (i.e., aspect ratio
≥
1:3 = high hazard = 100; aspect ratio < 1:3 = low
hazard = 25). The LOE-specific index
Sp.chem
j
is subsequently determined by the arithmetic mean
of each score given to cp.chem
1,(j), . . . , cp.chem
n,(j).
2.
LOE index values based on toxicity: Five hazard classes (
Ctox
i
) of increasing evidence of toxicity to
humans according to US EPA guidelines are specified and mapped onto a scoring system within
the [0,100] range [
23
]. Specific rules apply for the study, or LOE, to be categorised into a specific
class. For example, for class
Ctox
5=
100, there must be convincing causal evidence between
the NM and biological effect. LOE may fall into one or more classes based on the conclusions
provided by the author. Hence, a percentage
Di,j
would be assigned according to the likelihood
the conclusions fit into a certain class. The LOE-specific index value
Stox
j
is then calculated by the
following equation:
Stox
j=
5
∑
i=1
Ctox
iDi,j(2)
3.
Total LOE index values: The LOE indices for physico-chemical data and toxicity are aggregated to
form a global LOE index (
Sj
) representing intrinsic hazard demonstrated by the study. Since both
do not have equal weight in the hazard assessment, a weighted sum (WS) operator is applied.
The weights
wp.chem
<
wtox
imply that toxicity evidence explains more about the intrinsic hazard
potential of a NM than physico-chemical evidence. The following equation illustrates the
aggregation of the indices:
Sj=Sp.chem
jwp.chem +Stox
jwtox . (3)
4.
LOE weight: The weight (
Wj
) of each LOE is established according to a Logic model that uses
regulatory data quality criteria (adequacy, reliability, statistical power, toxicological significance)
to infer the study’s relevance to measuring the hazard potential of a NM [
20
]. Each weight is
normalised by dividing them by their total sum:
w0
j=Wj
∑∀jWj
. (4)
5.
Weighted LOE index value: The impact of each LOE on the total hazard assessment is calculated
by obtaining the product of the global LOE index value (
Sj
) and normalised study quality
weight (w0
j):
WIj=Sjw0
j
The sum of each weighted LOE index value represents the hazard score (
V
) for the NM, which
can be compared to hazard scores computed for other NM for relative hazard ranking.

Int. J. Mol. Sci. 2018,19, 649 5 of 22
V=
n
∑
j=1
WIj.(5)
Hazard scores were calculated for TiO
2
, Ag, and ZnO using steps 1–5 and ranked accordingly.
Monte Carlo analysis was used to probabilistically assess the sensitivity of the hazard scores to the
weights applied to the physico-chemical, toxicity, and study quality criteria. This consists of random
sampling from the distribution of
Sp.chem
j
,
Stox
j
, and/or
Wj
in a finite number of simulations to derive a
distribution of results (
V0
). The variability of the distribution of results provides information on the
uncertainty inherent to the WoE methodology, and the sensitivity of the of the input parameters to the
hazard ranking of the three NMs.
The sensitivity and uncertainty analysis comprised of the following steps:
i.
The probability distributions for the input criterion were set at the full range of the normalisation
scale, that is, [0,100] for Sp.chem
jand Stox
jand [0,1] for Wj.
ii.
Four sampling scenarios were investigated. Three of which involved sampling input values of
one of the criterion (
Sp.chem
j
,
Stox
j
,
Wj
) from their probability distributions while holding the others
constant. The fourth sampled input values from the probability distributions of all three criterion.
The sampling was uniformly distributed within the interval.
iii.
Each sampling scenario was simulated 10,000 times and the total weighted LOE index value
V0
i
recorded at each iteration.
3. Results
3.1. Hazard Ranking of Nanoparticles Composed of TiO2, Ag and ZnO
Figure 1illustrates the graphical structure and parameterisation of the BN with TiO
2
as the sample
NM. This shows the marginal probability of each state within the nodes and their causal linkages
resulting from an expert elicitation process as well as structure and parameter machine learning.
The NM hazard node (Figure 1, red ellipse) represents the “hazard potential” of TiO
2
, implying the
probability of no, low, medium, and high hazard is 52.46, 7.38, 25.41, and 14.75% respectively. To obtain
a normalised variable for the purposes of hazard ranking, the weighted sum operation of the NM
hazard state probabilities and a uniform scale [0,
1
3
,
2
3
, 1] was applied to acquire a normalised hazard
score of 34%. The uniform scale represents the increasing hazard potential of the states “None”, “Low”,
“Medium”, and “High”. The same method was used to probabilistically characterise the hazard of Ag
and ZnO (see Appendix, Figures A1 and A2). The normalised hazard scores led to hazard potentials,
ranked from highest to lowest, of ZnO (91%), Ag (61%), TiO2(34%).
The quantitative WoE with MCDA methodology was applied to the same literature evidence used
to train the BN in order to produce a total weighted index value (
V
), representing the intrinsic hazard
potential of the NM. The application of the framework to the TiO
2
literature is provided in Table 1and
explained below. The results for TiO
2
is provided in Table 1, with representations for Ag and ZnO are
supplied in the Appendixes Cand D, Tables A1 and A2 respectively.
The first step of the method required the expert evaluation of the physico-chemical data according
to the index scoring system described in the methodology section. The aggregated LOE-specific score
based on physico-chemical properties ranged from 30.56 to 61.11 with an average value of 41.54.

Int. J. Mol. Sci. 2018,19, 649 6 of 22
Int. J. Mol. Sci. 2018, 19, x FOR PEER REVIEW 6 of 22
Figure 1. Graphical structure and parameterization of the Bayesian networks (BN) with TiO
2
as the sample nanomaterial (NM). Ellipses represent nodes
and directed links signify the conditional relationship between parent and child nodes. The accompanying bar charts denote the % state probabilities. The
nodes are colour categorised into green for physicochemical properties, yellow for experimental methodology, orange for biological effects, and red for
NM hazard potential. Adapted from Marvin et al. [9].
Figure 1.
Graphical structure and parameterization of the Bayesian networks (BN) with TiO
2
as the sample nanomaterial (NM). Ellipses represent nodes and
directed links signify the conditional relationship between parent and child nodes. The accompanying bar charts denote the % state probabilities. The nodes are
colour categorised into green for physicochemical properties, yellow for experimental methodology, orange for biological effects, and red for NM hazard potential.
Adapted from Marvin et al. [9].

Int. J. Mol. Sci. 2018,19, 649 7 of 22
Table 1.
Quantitative weight of evidence (WoE) results for nano-TiO
2
. Each line of evidence (LOE) represents experimental evidence from academic literature
evaluated based on physico-chemical properties, toxicity, and study quality. The overall hazard of nano-TiO
2
(
V
) is derived from sum of all LOE-specific hazard
scores (WIj).
ID (j) Reference
LOE Index Values Based
on Physico-Chemical
Properties
(Sp.chem
j)
LOE Index
Values Based
on Toxicity
(Stox
j)
Total LOE
Index Values
(Sj)
Study Quality
Weight
(Wj)
Normalised Study
Quality Weight
(wj)
Weighted LOE
Index Values
(WIj)
1 Baisch et al. [24] 41.67 87.50 73.75 0.61 0.04 3.24
2 Baisch et al. [24] 41.67 75.00 65.00 0.84 0.06 3.91
3 Baisch et al. [24] 50.00 75.00 67.50 0.84 0.06 4.06
4 Catalan et al. [25] 38.89 37.50 37.92 0.71 0.05 1.94
5 Catalan et al. [25] 38.89 62.50 55.42 0.79 0.06 3.13
6 Catalan et al. [25] 38.89 37.50 37.92 0.65 0.05 1.78
7 Chen et al. [26] 30.56 75.00 61.67 0.48 0.03 2.14
8 Duan et al. [27] 44.44 25.00 30.83 0.47 0.03 1.04
9 Duan et al. [27] 44.44 25.00 30.83 0.32 0.02 0.71
10 Farcal et al. [28] 61.11 25.00 35.83 0.77 0.06 1.99
11 Farcal et al. [28] 47.22 37.50 40.42 0.76 0.05 2.20
12 Fisichella et al. [29] 30.56 12.50 17.92 0.52 0.04 0.66
13 Fisichella et al. [29] 38.89 12.50 20.42 0.56 0.04 0.81
14 Gurr et al. [30] 33.33 62.50 53.75 0.50 0.04 1.94
15 Gurr et al. [30] 33.33 37.50 36.25 0.50 0.04 1.31
16 Hu et al. [31] 47.22 62.50 57.92 0.54 0.04 2.23
17 Leppanen et al. [32] 41.67 12.50 21.25 0.62 0.04 0.95
18 Lindberg et al. [33] 41.67 0.00 12.50 0.63 0.05 0.57
19 Lindberg et al. [33] 41.67 50.00 47.50 0.56 0.04 1.91
20 Shimizu et al. [34] 33.33 62.50 53.75 0.76 0.05 2.93
21 Tassinari et al. [35] 52.78 12.50 24.58 0.56 0.04 0.98
22 Wang et al. [36] 41.67 62.50 56.25 0.94 0.07 3.80
Hazard Score (V)=44.24

Int. J. Mol. Sci. 2018,19, 649 8 of 22
Each LOE was subsequently evaluated according to toxicological evidence, resulting in scores
ranging from 0 to 87.50 with an average of 43.18. These LOE indices were aggregated by a weighted
sum operator to form a global LOE index (
Sj
), representing the intrinsic hazard potential inferred
from each study. The contribution of each LOE to the concluding hazard score is regulated by means
of the study quality weighting procedure. This facilitates the inclusion of a heterogenous evidence
base, attributing higher weights to studies most relevant to hazard assessment. The product of the
LOE-specific index value (
Sj
) and the normalised study quality weight (
w0
j
) determines the weighted
LOE-specific hazard score (
WIj
) for study
j
. The scores are summed into the total weighted index
value of 44.24 (
V
), the hazard score of TiO
2
. This WoE methodology was applied to the Ag and ZnO
literature (see Appendixes Cand D), producing hazard scores of 45.26 and 52.34 respectively. Hence,
the relative hazard ranking of each NM from highest to lowest according to their hazard score
(V
):
ZnO (52.34), Ag (45.26), TiO
2
(44.24). The WoE model workings are provided in Excel format in the
Supplementary Material.
3.2. Evaluation of the Performances of Bayesian Networks (BN) and WoE
An out-of-sample, or cross-validation, test was used to evaluate the prediction accuracy of NM
hazard by the BN. This procedure involved applying the input parameters (physico-chemical data,
study type, administration route) for each case as evidence and observing the probability distribution
amongst the states of the hypothesis node, NM hazard. The NM hazard state (None, Low, Medium,
High) with the highest likelihood was chosen as the “predicted” state, which was compared to the
state observed in the literature.
A total of 43 TiO
2
, Ag, and ZnO cases that were not used in the structure and parameter learning
procedure of the BN were used in the cross-validation analysis by Marvin et al. [
9
] and examined
individually for the purposes of this paper. Table 2illustrates the results of the cross-validation test for
15 sample cases, showing that NM hazard is accurately predicted in 9 out of 15 cases. The prediction
accuracy for all 40 cases is 67%.
Out of the 43 cases analysed in the cross-validation test, 24 were TiO
2
, 10 Ag, and 9 ZnO.
The prediction accuracy by NM type shows 100% for ZnO, 70% for Ag and 54% for TiO
2
. The low
precision for TiO
2
was investigated further, and it was observed that the results may be skewed due
to a repetition of a study which produced varying levels of observed NM hazard with the same
input parameters. With these cases omitted, 67% of TiO
2
cases are predicted correctly. The full
cross-validation analysis is available in the Supplementary Material.
The evaluation of the performance of quantitative WOE, which is, strictly speaking, not a
prediction model, but an approach used to inform decision-making based on the strength of evidence,
relied on uncertainty and sensitivity analysis.
3.3. Sensitivity and Uncertainty Analysis of BN and WoE
An out-of-sample, or cross-validation, test was used to evaluate the prediction accuracy of NM
hazard by the BN. This procedure involved applying the input parameters (physico-chemical data,
study type, administration route) for each case as evidence and observing the probability distribution
amongst the states of the hypothesis node, NM hazard. The NM hazard state (None, Low, Medium,
High) with the highest likelihood was chosen as the “predicted” state, which was compared to the
state observed in the literature.
A value of information (VOI) analysis was applied to the BN to analyse the potential usefulness
of additional information (input nodes) to the hypothesis variable, NM hazard. The task of the VOI
analysis is to identify, using entropy reduction, the variables which are most informative with respect
to the hypothesis variable [
21
]. Entropy reduction calculated the degree to which the input variables
(physico-chemical properties, administration route, and study type) influenced the NM hazard node.
A higher value indicates a higher sensitivity of NM hazard to the corresponding input node.

Int. J. Mol. Sci. 2018,19, 649 9 of 22
Table 2. Sample (15 cases) results of out-of-sample validation test for BN.
Case
Test Data NM Hazard
Shape Nanop-
Article Dissolution Surface Area
(m2/g)
Surface Charge
(mV)
Surface
Coatings
Surface
Reactivity Aggregation
Particle
Size
(nm)
Administration
Route
Study
Type Actual Predicted
1 Irregular TiO20–25% 51–101.25
from
−
50 to
−
25
Silianes-
aluminium Low High 10–50 - In vitro None None
2 Amorph TiO2- - - - - - >100 Injection In vivo High Medium
3 Sphere TiO2- - - AHPP - Low 10–50 - In vitro None None
4 Irregular TiO2- 15–51 - - - High >100 Oral In vivo None Medium
5 Irregular TiO2- 51–101.25
from
−
50 to
−
25
Hydroxyl - Medium 50–100 Oral In vivo None Low
6 Sphere Ag - - - - - - 10–50 Inhalation In vivo High High
7 Sphere Ag - - - PVP - Low 50–100 Inhalation In vivo High Medium
8 Sphere Ag - - - - - - 10–50 Intravenous In vivo None None
9 Sphere Ag - - - Citrate - - 10–50 Oral In vivo Medium Medium
10 Sphere Ag - 0–15
from
−
50 to
−
25
PVP - High 10–50 - In vitro None Low
11 Sphere Ag 0–25% - - - - Low 10–50 Oral In vivo Medium Medium
12 Sphere Ag - - 0–25 - - Low 10–50 Oral In vivo High Medium
13
Elongated
ZnO - 0–15 0–25 None - Medium >100 - In vitro High High
14
Elongated
ZnO 0–25% 15–51 - Triethoxycapryl
silane - Medium >100 - In vitro High High
15 Irregular ZnO 0–25% - - - Low - 10–50 - In vitro High High

Int. J. Mol. Sci. 2018,19, 649 10 of 22
The results of the VOI analysis are presented in Table 3. Study type (0.34), particle size (0.28), and
surface coatings (0.26) are distinguished as properties that have significant influence over the NM
hazard node for TiO
2
. In contrast, administration route (0.64), surface coatings (0.53), and surface
charge (0.37) have the largest effect for the Ag hazard node. The entropy of the input parameters on
the NM hazard node for ZnO showed the least significance, with surface reactivity (0.16) being the
only meaningful result. Marvin et al. (2017) demonstrated that cytotoxicity evidence also has a highly
influential effect on the hazard potential of TiO2, Ag, and ZnO [9].
Table 3.
Sensitivity analysis of BN model. Entropy reduction indicates the degree to which NM hazard
was sensitive to each input nodes of the model. Higher values signify higher sensitivity of the NM
hazard mode to the input node.
Input Variable Nanomaterial
TiO2Ag ZnO
Surface coatings 0.26 0.53 0.01
Surface area 0.22 0.26 0.02
Particle size 0.28 0.13 0.05
Surface charge 0.08 0.37 0
Aggregation 0.09 0.22 0.01
Shape 0.26 0 0
Surface reactivity 0 0 0.16
Dissolution 0 0 0
Administration route 0.19 0.64 0
Study type 0.34 0.07 0.02
The sensitivity of individual input parameters on the NM hazard node may also be analysed for
the BN. The effect of evidence from each state for the physico-chemical input parameters particle size
(Table 4) and surface area (Table 5) on the predicted hazard potential is observed. The results include
the normalised NM hazard potential as discussed before. Table 4illustrates that the hazard ranking
with no evidence (from highest to lowest: ZnO, Ag, TiO
2
) remains consistent with one exception,
when particle size is within the range 0 nm to 10 nm. In this case TiO
2
becomes the highest hazard
(100%), followed by ZnO (86%), and Ag (50%). In contrast, Table 5shows that the original hazard
ranking order is true for only the surface area of between 189 and 2025 m
2
/g. The four other states of
surface area still rank ZnO with the highest hazard potential, then TiO2and finally Ag.
Table 4. Effect of particle size on the normalised NM hazard potential for TiO2, Ag, and ZnO.
Particle Size Nanomaterial Hazard Potential
TiO2Ag ZnO
from 0 to 10 100% 50% 86%
from 10 to 50 25% 58% 94%
from 50 to 100 42% 55% 100%
>100 73% 77% 89%
No Evidence 34% 61% 91%

Int. J. Mol. Sci. 2018,19, 649 11 of 22
Table 5. Effect of surface area on the normalised NM hazard potential for TiO2, Ag, and ZnO.
Surface Area Nanomaterial Hazard Potential
TiO2Ag ZnO
from 0 to 15 56% 54% 94%
from 15 to 51 71% 58% 89%
from 51 to 101.25 28% 27% 88%
from 101.25 to 189 73% 4% 67%
from 189 to 2025 15% 92% 100%
No Evidence 34% 61% 91%
A Monte Carlo analysis was applied to the quantitative WoE with MCDA methodology to evaluate
the model in terms of uncertainty of the final hazard ranking. In the literature, Monte Carlo analyses
have been used to analyse the sensitivity of decision criteria to input variables for quantitative WOE
models [
14
,
37
]. This approach allows for an examination of the influence of the input variables on the
total weighted index value (V) for each NM. Four sampling scenarios were performed:
1.
Vary LOE-specific index of physico-chemical properties (
Sp.chem
j
), while keeping all other input
parameters constant.
2. Vary LOE-specific index of toxicity (Stox
j), while keeping all other input parameters constant.
3. Vary the study quality weights (Wj), while keeping all other input parameters constant.
4. Vary all input parameters Sp.chem
j,Stox
j, and Wj
Descriptive statistics of the resulting probability distributions of
V0
i
each of the four sampling
scenarios are illustrated in Table 6. The metrics used to illustrate the influence of the variation of the
input parameters on the observed hazard score (V) are the mean, standard deviation, and average
absolute deviation of V0
i(for i=1 : 10, 000). The absolute deviation is calculated as [14]:
∆Vi=
V0
i−V
in [0, 100](6)
The average
∆Vi
is low for sampling scenarios (i) and (iii), increasing slightly for scenario (ii),
and increasing substantially for scenario (iv) where the average absolute deviation is 6.5% for TiO
2
,
6.3% for Ag, and 10.3% for ZnO when all the input parameters are considered uncertain. The analysis
indicates that hazard score produced by the WoE model is least sensitive to changes in the study
quality weight parameters, and influenced most by changes to the index of toxicity.
Table 6.
Results of Monte Carlo sensitivity analysis for quantitative WoE methodology displaying
the mean, standard deviation, and average absolute difference of the total weighted index value (V)
from 10,000 simulations for each uncertainty scenario proposed; (i) variation of physico-chemical input
parameters, (ii) variation of toxicity parameters, (iii) variation of study weight parameters, and (iv)
variation of all (i)–(iii) parameters.
Nanomaterial Parameter Variation of Input Parameters
Sp.chem
jStox
jWjSp.chem
j,Stox
j, and Wj
TiO2
V=44.2
Mean (Standard Deviation) 46.6 (1.9) 47.6 (4.4) 42.7 (2.2) 49.9 (5.4)
Average Absolute Deviation
2.6 4.5 2.2 6.5
Ag
V=45.3
Mean (Standard Deviation) 47.7 (2.1) 48.4 (4.9) 45.4 (2.3) 50.0 (6.2)
Average Absolute Deviation
3.3 4.7 1.9 6.3
ZnO
V=52.3
Mean (Standard Deviation) 53.6 (4.4) 48.6 (10.3) 52.7 (0.7) 49.8 (12.5)
Average Absolute Deviation
3.7 8.8 0.7 10.3

Int. J. Mol. Sci. 2018,19, 649 12 of 22
The uncertainties attributed to the WoE methodology originate from the expert elicitation methods
utilized to determine the indexes, metrics, and criterion in the initial model formation, and also in the
interpretations of the expert appraising each study. The stability of the final hazard ranking order was
assessed to ensure that the order is a function of intrinsic hazard associated with the NMs, or simply
the output of model noise. To evaluate the stability of the hazard ranking order, within each sampling
scenario the results
V0
i
(
i=
1
:
10, 000) for each NM were ranked. There are six possible permutations
for the hazard ranking order of the three NMs (see Table 7). For example, under sampling scenario
(ii) the index of toxicity (
Stox
j
) is varied resulting in simulated
V0
i
for TiO
2
, Ag, and ZnO. This results in
30,000 simulations in total (i.e., 10,000
V0
i
for each NM). Each simulation
i
was ranked according to the
hazard scores V0
icalculated for each NM.
Table 7.
Distribution of the hazard ranking order of nanoparticles resulting from Monte Carlo
uncertainty analysis varying input parameters: (i) physico-chemical properties, (ii) toxicity potential,
(iii) study weights, and (iv) all input parameters.
Alternative
Orders
Rank from Lowest (1) to
Highest (3) Hazard Ranking % by Variations of Input Parameters
Total
1 2 3 Sp.chem
jStox
jWjSp.chem
j,Stox
j,
and Wj
a TiO2Ag ZnO 55% 22% 80% 20% 44%
b TiO2ZnO Ag 6% 11% 0% 9% 7%
c Ag TiO2ZnO 31% 20% 20% 21% 23%
d Ag ZnO TiO22% 8% 0% 10% 5%
e ZnO TiO2Ag 3% 21% 0% 21% 11%
f ZnO Ag TiO22% 17% 0% 19% 10%
Table 7illustrates that the observed hazard ranking order (from lowest to highest: (a) TiO
2
, Ag,
ZnO) is stable across the four stressed scenarios in 44% of all samples. The order remains consistent to
the observed order (a) in 55% of simulations where the physico-chemical index was varied, in 22%
of simulations where the toxicity index was varied, in 80% of simulations where the study quality
weights were varied, and in 20% of simulations where all input parameters were varied. The second
highest hazard ranking order is permutation (c), Ag, TiO
2
, and ZnO. Significant sensitivity of the
ranking order to changes in the LOE-specific toxicity index is highlighted by the relative uniformity of
the ranking distributions across the permutations (a)–(e).
4. Discussion
This comparative study investigated the efficacy of quantitative WoE and Bayesian methodologies
in predicting the hazard potential of metal and metal-oxide. The BN and WoE models used the same
reference database to generate relative hazard rankings of TiO
2
, Ag, and ZnO. The results indicate
that, while the relative hazard ranking remain consistent across both models (ZnO, Ag, TiO
2
; from
highest hazard to lowest hazard), significant variability was observed when evaluated for stability and
predictive accuracy. The ranking order from the WoE model was stable for 44% of 40,000 sampling
scenarios with stressed input parameters. Cross-validation of the BN demonstrated 67% prediction
accuracy overall, with significant variation amongst the NMs: TiO2(54%), Ag (70%), ZnO (100%).
Both methodologies exhibit potential to support the comprehensive human health risk assessment
for NMs. The methods allow for the incorporation of expert judgement to bridge the gap where
experimental data is lacking, and to update hazard predictions as new information becomes available.
While expert elicitation methods form the basis of each model’s construction, the incorporation of data
to form a conclusion differs. The BN refines both its NM hazard probabilistic forecasts and causal
interdependencies between variables (model parameters) through the application of machine learning
techniques on the database. In contrast, the quantitative WoE model refines its “hazard score” every

Int. J. Mol. Sci. 2018,19, 649 13 of 22
time a new study, or line of evidence, is evaluated according to the pre-determined criteria and metrics.
Therein lies a significant advantage of the BN over the WoE model. For the BN, the model is created
and adapts to the input data, whereas the scoring criteria of the WoE model remains constant.
The BN and WoE models both utilize physico-chemical, toxicological, and study type data to
infer the hazard potential of TiO
2
, Ag, and ZnO. However, each experimental result contributes to the
resulting hazard prediction equally within the BN framework. This is a limitation of the model as it
neglects the relevance, quality, and reliability of the characterization experiments used within each
study. Given that the toxicity of NMs is a complex function of several properties that are experimentally
problematic to characterise, the inclusion of study quality criterion is important for a reliable hazard
assessment tool. The quantitative WoE methodology controls the influence of each LOE on the final
hazard score by weighting it according to study quality criteria.
A combination of both WoE and BN models would overcome the limitations described. Here,
the WoE would evaluate the experimental evidence available according to a set of rules for accepting
or rejecting evidence. This filtered evidence could then be used to train the BN, which, at this point,
should be much more reliable.
5. Conclusions
Responsible innovation requires safety protocols to be integrated prior to the commercialization
phase of any manufactured NM [
38
]. The proliferation of nano-enabled products has continued
and this suggests the implicit acceptance on the part of employers of the potential hazard, exposure,
and risk of nanomaterials [
39
]. The global market for manufactured nanomaterials was valued at
$7.3 billion in 2016 and is projected to expand to $16.8 billion by 2022 [
40
]. This rapid advancement
combined with the ambiguity of risk intelligence may result in many employers insufficiently reducing,
controlling, or transferring the risk, and hence, neglecting to adequately protect their workers.
Therefore, the establishment of appropriate human health risk assessment (RA) methodologies
and tools are considered crucial to the sustainable development and application of NMs. Hazard
identification, effects assessment, exposure assessment, and risk characterisation comprise the elements
of a comprehensive RA framework for chemicals [5].
Quantitative models for manufactured NM hazard screening enable proactive risk minimization
strategies in the design and development phase of NM production [
14
,
41
]. Researchers can ex-ante
predict the impact of varying physico-chemical properties on the resulting hazard potential, thus
promoting the safety-by-design principle of NM manufacturing. The BN model allows for this
probabilistic forecasting as illustrated in Tables 4and 5.
In a human health context, hazard identification involves inferring substance-specific biological
adverse effects from experimental (
in vitro
,
in vivo
data, in silico) observations [
14
,
42
]. Toxicological
studies provide the relevant criteria for hazard determination. However, studies have revealed that
size and physico-chemical properties of NMs induce unique or more aggressive biological activity at
the nanoscale [
43
]. Physico-chemical characteristics of nanomaterials known to influence toxicity are
surface area [
44
,
45
], surface coating [
46
], composition [
45
], purity, shape [
47
], primary particle size [
48
],
aggregation [45], and crystal structure [49].
Dose-response assessment quantitatively determines the relationship between adverse effects
(i.e., hazards) and a concentration of a substance in a controlled environment. Significant correlations
between dose and biologically relevant endpoints or biomarkers are therefore utilized to determine
no-observed-adverse-effect levels (NOAELs) and human health exposure thresholds, such as
recommended exposure limits (RELs) or occupational exposure limits (OELs). Exposure scenario
analysis subsequently forecasts the extent to which potentially vulnerable parties, such as factory
workers, are exposed to material concentrations during the life cycle of a NM. By comparing predictions
of scenario-based to threshold limits determined toxicologically, an explicit risk characterisation may
be ascertained and used to inform strategic risk management decisions [41].

Int. J. Mol. Sci. 2018,19, 649 14 of 22
Control banding (CB) or risk matrices have been proposed as an appropriate framework to
illustrate and measure the risk of NM to human health [
50
]. These tools determine the inherent
risk posed by an NM through the product of exposure and hazard metrics. Despite numerous
implementations of CB to assess the occupational risk NM [
51
,
52
], the preceding requirement of
validated and transparent quantitative hazard and exposure ranking methods has not yet been
conclusively fulfilled. Each methodology (e.g., hazard ranking, exposure prediction) must be
scientifically evaluated in isolation due to the complexities posed by NM. The BN and WoE hazard
screening methods implemented in this paper are fitting candidates for the hazard axis. However,
a combination of WoE that would weight the quality of evidence data along with the BN may prove to
be optimal hazard assessment framework.
Supplementary Materials: Supplementary materials can be found at www.mdpi.com/1422-0067/19/3/649/s1.
Acknowledgments:
This project has received funding from the European Union’s Horizon 2020 research and
innovation programme under grant agreement No. 720851. See www.protect-h2020.eu.
Author Contributions:
Barry Sheehan carried out the main part of data analysis and writing; Finbarr Murphy and
Martin Mullins contributed to the structure and context; Anna L. Costa and Felice C. Simeone contributed ideas
from nanotoxicology; Paride Mantecca contributed ideas from the biosciences; Irini Furxhi evaluated evidence.
All authors read and approved the final manuscript.
Conflicts of Interest: The authors declare no conflict of interest.
Abbreviations
NM nanomaterial
BN Bayesian network
CPT conditional probability table
CNS central nervous system
OEL occupational exposure limit
REL recommended exposure limit
WoE weight of evidence
MCDA multi-criteria decision analysis
CB control banding
WS weighted sum
VOI value of information
NM nanomaterial

Int. J. Mol. Sci. 2018,19, 649 15 of 22
Appendix A
Int. J. Mol. Sci. 2018, 19, x FOR PEER REVIEW 15 of 22
Appendix A
Figure A1. Graphical structure and parameterization of the BN with Ag input as evidence. Ellipses represent nodes and directed links signify the
conditional relationship between parent and child nodes. The accompanying bar charts denote the % state probabilities. The nodes are colour
categorised into green for physicochemical properties, yellow for study type, orange for biological effects, and red for NM hazard potential. Adapted
from Marvin et al. [9].
Figure A1.
Graphical structure and parameterization of the BN with Ag input as evidence. Ellipses represent nodes and directed links signify the conditional
relationship between parent and child nodes. The accompanying bar charts denote the % state probabilities. The nodes are colour categorised into green for
physicochemical properties, yellow for study type, orange for biological effects, and red for NM hazard potential. Adapted from Marvin et al. [9].

Int. J. Mol. Sci. 2018,19, 649 16 of 22
Appendix B
Int. J. Mol. Sci. 2018, 19, x FOR PEER REVIEW 16 of 22
Appendix B
Figure A2. Graphical structure and parameterization of the BN with ZnO input as evidence. Ellipses represent nodes and directed links signify the
conditional relationship between parent and child nodes. The accompanying bar charts denote the % state probabilities. The nodes are colour
categorised into green for physicochemical properties, yellow for study type, orange for biological effects, and red for NM hazard potential. Adapted
from Marvin et al. [9].
Figure A2.
Graphical structure and parameterization of the BN with ZnO input as evidence. Ellipses represent nodes and directed links signify the conditional
relationship between parent and child nodes. The accompanying bar charts denote the % state probabilities. The nodes are colour categorised into green for
physicochemical properties, yellow for study type, orange for biological effects, and red for NM hazard potential. Adapted from Marvin et al. [9].

Int. J. Mol. Sci. 2018,19, 649 17 of 22
Appendix C
Table A1.
Quantitative WoE results for nano-Ag. Each LOE represents experimental evidence from academic literature evaluated based on physico-chemical
properties, toxicity, and study quality. The overall hazard of nano-Ag (V) is derived from sum of all LOE-specific hazard scores (W Ij).
ID Reference
LOE Index Values
Based on
Physico-Chemical
Properties
(Sp.chem
j)
LOE Index Values
Based on Toxicity
(Stox
j)
Total LOE Index
Values
(Sj)
Study Quality
Weight
(Wj)
Normalised Study
Quality Weight
(w0
j)
Weighted LOE
Index Values
(WIj)
1 Braakhuis et al. [48] 47.22 62.50 57.92 0.84 0.07 4.25
2 Braakhuis et al. [48] 33.33 12.50 18.75 0.74 0.06 1.21
3 Braakhuis et al. [48] 47.22 12.50 22.92 0.60 0.05 1.21
4 Braakhuis et al. [53] 41.67 25.00 30.00 0.72 0.06 1.90
5 Braakhuis et al. [53] 36.11 62.50 54.58 0.71 0.06 3.41
6 Braakhuis et al. [53] 33.33 62.50 53.75 0.71 0.06 3.36
7 Braakhuis et al. [53] 33.33 12.50 18.75 0.66 0.01 0.20
8 Gaiser at al. [54] 47.22 75.00 66.67 0.79 0.07 4.60
9 Gaiser at al. [54] 47.22 75.00 66.67 0.63 0.05 3.65
10 Haberl et al. 2013 [55] 38.89 75.00 64.17 0.60 0.05 3.39
11 Lankveld et al. 2010 [56] 44.44 37.50 39.58 0.47 0.04 1.63
12 Lee et al. 2013 [57] 52.78 62.50 59.58 0.73 0.06 3.83
13 Loeschner et al. 2011 [58] 52.78 37.50 42.08 0.48 0.04 1.77
14 Nymark et al. 2013 [59] 58.33 50.00 52.50 0.58 0.05 2.67
15
Van der Zande et al. 2012 [
60
]
44.44 25.00 30.83 0.61 0.05 1.66
16
Van der Zande et al. 2012 [
60
]
61.11 25.00 35.83 0.61 0.05 1.93
17 Yun at al. 2015 [61] 44.44 62.50 57.08 0.92 0.08 4.59
Hazard Score (V)=45.26

Int. J. Mol. Sci. 2018,19, 649 18 of 22
Appendix D
Table A2.
Quantitative WoE results for nano-ZnO. Each LOE represents experimental evidence from academic literature evaluated based on physico-chemical
properties, toxicity, and study quality. The overall hazard of nano-ZnO (V) is derived from sum of all LOE-specific hazard scores (W Ij).
ID Reference
LOE Index Values
Based on
Physico-Chemical
Properties
(Sp.chem
j)
LOE Index Values
Based on Toxicity
(Stox
j)
Total LOE Index
Values
(Sj)
Study Quality
Weight
(Wj)
Normalised Study
Quality Weight
(w0
j)
Weighted LOE
Index Values
(WIj)
1 Farcal et al. [28] 50.00 50.00 50.00 0.76 0.29 14.63
2 Farcal et al. [28] 52.78 50.00 50.83 0.76 0.29 14.87
3 Lu et al. [62] 38.89 62.50 55.42 0.59 0.23 12.68
4 Zhang et al. [63] 36.11 62.50 54.58 0.48 0.19 10.15
Hazard Score (V)=52.34

Int. J. Mol. Sci. 2018,19, 649 19 of 22
References
1.
National Research Council. Science and Decisions: Advancing Risk Assessment; National Academies Press:
Washington, DC, USA, 2009.
2.
Oberdörster, G.; Maynard, A.; Donaldson, K.; Castranova, V.; Fitzpatrick, J.; Ausman, K.; Carter, J.; Karn, B.;
Kreyling, W.; Lai, D. Principles for characterizing the potential human health effects from exposure to
nanomaterials: Elements of a screening strategy. Part. Fibre Toxicol. 2005,2, 8. [CrossRef] [PubMed]
3.
Taniguchi, N. On the basic concept of Nano-technology. In Proceedings of the International Conference on
Production Engineering, Tokyo, Japan, 26–29 August 1974; Japan Society of Precision Engineering: Tokyo,
Japan, 1974. Part II. Volume 2, pp. 18–23.
4.
Stone, V.; Önlü, S.; Bergamaschi, E.; Carlander, D.; Costa, A.; Engelmann, W.; Ghanem, A.;
Hartl, S.; Hristozov, D.; Scott-Fordsmand, J.J. Research Priorities Relevant to Development or Updating of
Nano-Relevant Regulations and Guidelines; NanoSafetyCluster Research Regulatory Roadmap; January 2017;
Available online: http://www.safenano.org/news/news-articles/eu-nanosafety-cluster-publishes-research-
regulatory-roadmap/ (accessed on 25 February 2018). [CrossRef]
5.
Van Leeuwen, C.J.; Vermeire, T.G. Risk Assessment of Chemicals: An Introduction; Springer Science & Business
Media: Dordrecht, The Netherlands, 2007.
6.
Murphy, F.; Mullins, M.; Hester, K.; Gelwick, A.; Scott-Fordsmand, J.J.; Maynard, T. Insuring nanotech
requires effective risk communication. Nat. Nanotechnol. 2017,12, 717–719. [CrossRef] [PubMed]
7.
Money, E.S.; Reckhow, K.H.; Wiesner, M.R. The use of Bayesian networks for nanoparticle risk forecasting:
Model formulation and baseline evaluation. Sci. Total Environ. 2012,426, 436–445. [CrossRef] [PubMed]
8.
Murphy, F.; Sheehan, B.; Mullins, M.; Bouwmeester, H.; Marvin, H.J.; Bouzembrak, Y.; Costa, A.L.; Das, R.;
Stone, V.; Tofail, S.A. A Tractable Method for Measuring Nanomaterial Risk Using Bayesian Networks.
Nanoscale Res. Let. 2016,11, 503. [CrossRef] [PubMed]
9.
Marvin, H.J.; Bouzembrak, Y.; Janssen, E.M.; van der Zande, M.; Murphy, F.; Sheehan, B.; Mullins, M.;
Bouwmeester, H. Application of Bayesian networks for hazard ranking of nanomaterials to support human
health risk assessment. Nanotoxicology 2017,11, 123–133. [CrossRef] [PubMed]
10.
Tsang, M.P.; Hristozov, D.; Zabeo, A.; Koivisto, A.J.; Jensen, A.C.O.; Jensen, K.A.; Pang, C.; Marcomini, A.;
Sonnemann, G. Probabilistic risk assessment of emerging materials: Case study of titanium dioxide
nanoparticles. Nanotoxicology 2017,11, 558–568. [CrossRef] [PubMed]
11.
National Research Council. Review of EPA’s Integrated Risk Information System (IRIS) Process; National
Academies Press: Washington, DC, USA, 2014.
12.
Leso, V.; Fontana, L.; Chiara Mauriello, M.; Iavicoli, I. Occupational Risk Assessment of Engineered
Nanomaterials: Limits, Challenges and Opportunities. Curr. Nanosci. 2017,13, 55–78. [CrossRef]
13.
Hristozov, D.; Gottardo, S.; Semenzin, E.; Oomen, A.; Bos, P.; Peijnenburg, W.; van Tongeren, M.; Nowack, B.;
Hunt, N.; Brunelli, A.; et al. Frameworks and tools for risk assessment of manufactured nanomaterials.
Environ. Int. 2016,95, 36–53. [CrossRef] [PubMed]
14.
Hristozov, D.R.; Zabeo, A.; Foran, C.; Isigonis, P.; Critto, A.; Marcomini, A.; Linkov, I. A weight of evidence
approach for hazard screening of engineered nanomaterials. Nanotoxicology
2014
,8, 72–87. [CrossRef]
[PubMed]
15.
Zuin, S.; Micheletti, C.; Critto, A.; Pojana, G.; Johnston, H.; Stone, V.; Tran, L.; Marcomini, A. Weight
of evidence approach for the relative hazard ranking of nanomaterials. Nanotoxicology
2011
,5, 445–458.
[CrossRef] [PubMed]
16.
Nielsen, T.D.; Jensen, F.V. Bayesian Networks and Decision Graphs, 2nd ed.; Springer Science & Business Media:
New York, NY, USA, 2009; ISBN 0-387-68281-3.
17.
Wiesner, M.R.; Bottero, J.-Y. A risk forecasting process for nanostructured materials, and nanomanufacturing.
C. R. Phys. 2011,12, 659–668. [CrossRef]
18.
Linkov, I.; Welle, P.; Loney, D.; Tkachuk, A.; Canis, L.; Kim, J.; Bridges, T. Use of multicriteria decision
analysis to support weight of evidence evaluation. Risk Anal. 2011,31, 1211–1225. [CrossRef] [PubMed]
19.
Linkov, I.; Loney, D.; Cormier, S.; Satterstrom, F.K.; Bridges, T. Weight-of-evidence evaluation in
environmental assessment: Review of qualitative and quantitative approaches. Sci. Total Environ.
2009
,407,
5199–5205. [CrossRef] [PubMed]

Int. J. Mol. Sci. 2018,19, 649 20 of 22
20.
Klimisch, H.-J.; Andreae, M.; Tillmann, U. A systematic approach for evaluating the quality of experimental
toxicological and ecotoxicological data. Regul. Toxicol. Pharmacol. 1997,25, 1–5. [CrossRef] [PubMed]
21.
Kjærulff, U.B.; Madsen, A.L. Value of Information Analysis. In Bayesian Networks and Influence Diagrams: A
Guide to Construction and Analysis; Springer: New York, NY, USA, 2013; pp. 327–339, ISBN 978-1-4614-5103-7.
22.
Chen, S.H.; Pollino, C.A. Good practice in Bayesian network modelling. Environ. Model. Softw.
2012
,37,
134–145. [CrossRef]
23.
US EPA. Guidelines for Carcinogen Risk Assessment; US Environmental Protection Agency: Washington, DC,
USA, 1995.
24.
Baisch, B.L.; Corson, N.M.; Wade-Mercer, P.; Gelein, R.; Kennell, A.J.; Oberdorster, G.; Elder, A. Equivalent
titanium dioxide nanoparticle deposition by intratracheal instillation and whole body inhalation: The effect
of dose rate on acute respiratory tract inflammation. Part. Fibre Toxicol. 2014,11, 5. [CrossRef] [PubMed]
25.
Catalan, J.; Jarventaus, H.; Vippola, M.; Savolainen, K.; Norppa, H. Induction of chromosomal aberrations by
carbon nanotubes and titanium dioxide nanoparticles in human lymphocytes
in vitro
.Nanotoxicology
2012
,
6, 825–836. [CrossRef] [PubMed]
26.
Chen, J.; Dong, X.; Zhao, J.; Tang, G.
In vivo
acute toxicity of titanium dioxide nanoparticles to mice after
intraperitioneal injection. J. Appl. Toxicol. 2009,29, 330–337. [CrossRef] [PubMed]
27.
Duan, Y.; Li, N.; Liu, C.; Liu, H.; Cui, Y.; Wang, H.; Hong, F. Interaction between nanoparticulate anatase
TiO2and lactate dehydrogenase. Biol. Trace Elem. Res. 2010,136, 302–313. [CrossRef] [PubMed]
28.
Farcal, L.; Torres Andon, F.; Di Cristo, L.; Rotoli, B.M.; Bussolati, O.; Bergamaschi, E.; Mech, A.;
Hartmann, N.B.; Rasmussen, K.; Riego-Sintes, J.; et al. Comprehensive In Vitro Toxicity Testing of a
Panel of Representative Oxide Nanomaterials: First Steps towards an Intelligent Testing Strategy. PLoS ONE
2015,10, e0127174. [CrossRef] [PubMed]
29.
Fisichella, M.; Berenguer, F.; Steinmetz, G.; Auffan, M.; Rose, J.; Prat, O. Intestinal toxicity evaluation of TiO
2
degraded surface-treated nanoparticles: A combined physico-chemical and toxicogenomics approach in
caco-2 cells. Part. Fibre Toxicol. 2012,9, 18. [CrossRef] [PubMed]
30.
Gurr, J.R.; Wang, A.S.; Chen, C.H.; Jan, K.Y. Ultrafine titanium dioxide particles in the absence of
photoactivation can induce oxidative damage to human bronchial epithelial cells. Toxicology
2005
,213,
66–73. [CrossRef] [PubMed]
31.
Hu, R.; Gong, X.; Duan, Y.; Li, N.; Che, Y.; Cui, Y.; Zhou, M.; Liu, C.; Wang, H.; Hong, F. Neurotoxicological
effects and the impairment of spatial recognition memory in mice caused by exposure to TiO
2
nanoparticles.
Biomaterials 2010,31, 8043–8050. [CrossRef] [PubMed]
32.
Leppanen, M.; Korpi, A.; Miettinen, M.; Leskinen, J.; Torvela, T.; Rossi, E.M.; Vanhala, E.; Wolff, H.;
Alenius, H.; Kosma, V.M.; et al. Nanosized TiO(2) caused minor airflow limitation in the murine airways.
Arch. Toxicol. 2011,85, 827–839. [CrossRef] [PubMed]
33.
Lindberg, H.K.; Falck, G.C.; Catalan, J.; Koivisto, A.J.; Suhonen, S.; Jarventaus, H.; Rossi, E.M.; Nykasenoja, H.;
Peltonen, Y.; Moreno, C.; et al. Genotoxicity of inhaled nanosized TiO(2) in mice. Mutat. Res.
2012
,745,
58–64. [CrossRef] [PubMed]
34.
Shimizu, M.; Tainaka, H.; Oba, T.; Mizuo, K.; Umezawa, M.; Takeda, K. Maternal exposure to nanoparticulate
titanium dioxide during the prenatal period alters gene expression related to brain development in the
mouse. Part. Fibre Toxicol. 2009,6, 20. [CrossRef] [PubMed]
35.
Tassinari, R.; Cubadda, F.; Moracci, G.; Aureli, F.; D’Amato, M.; Valeri, M.; de Berardis, B.; Raggi, A.;
Mantovani, A.; Passeri, D.; et al. Oral, short-term exposure to titanium dioxide nanoparticles in
Sprague-Dawley rat: Focus on reproductive and endocrine systems and spleen. Nanotoxicology
2014
,8,
654–662. [CrossRef] [PubMed]
36.
Wang, Y.; Chen, Z.; Ba, T.; Pu, J.; Chen, T.; Song, Y.; Gu, Y.; Qian, Q.; Xu, Y.; Xiang, K.; et al. Susceptibility
of young and adult rats to the oral toxicity of titanium dioxide nanoparticles. Small
2013
,9, 1742–1752.
[CrossRef] [PubMed]
37.
Hristozov, D.R.; Gottardo, S.; Cinelli, M.; Isigonis, P.; Zabeo, A.; Critto, A.; van Tongeren, M.; Tran, L.;
Marcomini, A. Application of a quantitative weight of evidence approach for ranking and prioritising
occupational exposure scenarios for titanium dioxide and carbon nanomaterials. Nanotoxicology
2014
,8,
117–131. [CrossRef] [PubMed]

Int. J. Mol. Sci. 2018,19, 649 21 of 22
38.
Schulte, P.A.; Iavicoli, I.; Rantanen, J.H.; Dahmann, D.; Iavicoli, S.; Pipke, R.; Guseva Canu, I.; Boccuni, F.;
Ricci, M.; Polci, M.L.; et al. Assessing the protection of the nanomaterial workforce. Nanotoxicology
2016
,10,
1013–1019. [CrossRef] [PubMed]
39.
Del Castillo, A.M.P. Nanomaterials and Workplace Health & Safety: What Are the Issues for Workers?
European Trade Union Institute: Brussels, Belgium, 2013; ISBN 978-2-87452-288-8.
40.
Zion Market Research. Nanomaterials Market (Metal Oxide, Metals, Chemicals & Polymers and Others) for
Construction, Chemical Products, Packaging, Consumer Goods, Electrical and Electronics, Energy, Health
Care, Transportation and Other Applications: Global Market Perspective, Comprehensive Analysis and
Forecast, 2016–2022. Available online: https://www.zionmarketresearch.com/report/nanomaterials-market
(accessed on 22 February 2018).
41.
Schulte, P.A.; Geraci, C.L.; Murashov, V.; Kuempel, E.D.; Zumwalde, R.D.; Castranova, V.; Hoover, M.D.;
Hodson, L.; Martinez, K.F. Occupational safety and health criteria for responsible development of
nanotechnology. J. Nanopart. Res. 2014,16, 2153. [CrossRef] [PubMed]
42.
Foss Hansen, S.; Larsen, B.H.; Olsen, S.I.; Baun, A. Categorization framework to aid hazard identification of
nanomaterials. Nanotoxicology 2007,1, 243–250. [CrossRef]
43. Shin, S.; Song, I.; Um, S. Role of Physicochemical Properties in Nanoparticle Toxicity. Nanomaterials 2015,5,
1351–1365. [CrossRef] [PubMed]
44.
Kreyling, W.G.; Hirn, S.; Möller, W.; Schleh, C.; Wenk, A.; Celik, G.; Lipka, J.; Schäffler, M.; Haberl, N.;
Johnston, B.D.; et al. Air-Blood Barrier Translocation of Tracheally Instilled Gold Nanoparticles Inversely
Depends on Particle Size. ACS Nano 2014,8, 222–233. [CrossRef] [PubMed]
45.
Gatoo, M.A.; Naseem, S.; Arfat, M.Y.; Mahmood dar, A.; Qasim, K.; Zubair, S. Physicochemical Properties of
Nanomaterials: Implication in Associated Toxic Manifestations. BioMed Res. Int.
2014
,2014, 8. [CrossRef]
[PubMed]
46.
Worth, A.; Aschberger, K.; Asturiol Bofill, D.; Bessems, J.; Gerloff, K.; Graepel, R.; Joossens, E.; Lamon, L.;
Palosaari, T.; Richarz, A. Evaluation of the Availability and Applicability of Computational Approaches in the Safety
Assessment of Nanomaterials; EUR 28617 EN; Publications Office of the European Union: Luxembourg, 2017;
ISBN 978-92-79-68708-2.
47.
Eom, H.-J.; Jeong, J.-S.; Choi, J. Effect of aspect ratio on the uptake and toxicity of hydroxylated-multi
walled carbon nanotubes in the nematode, Caenorhabditis elegans.Environ. Health Toxicol.
2015
,30, e2015001.
[CrossRef] [PubMed]
48.
Braakhuis, H.M.; Gosens, I.; Krystek, P.; Boere, J.A.F.; Cassee, F.R.; Fokkens, P.H.B.; Post, J.A.; van Loveren, H.;
Park, M.V.D.Z. Particle size dependent deposition and pulmonary inflammation after short-term inhalation
of silver nanoparticles. Part. Fibre Toxicol. 2014,11, 49. [CrossRef] [PubMed]
49.
Perelshtein, I.; Lipovsky, A.; Perkas, N.; Gedanken, A.; Moschini, E.; Mantecca, P. The influence of the
crystalline nature of nano–metal oxides on their antibacterial and toxicity properties. Nano Res.
2015
,8,
695–707. [CrossRef]
50.
Mullins, M.; Murphy, F.; Baublyte, L.; McAlea, E.M.; Tofail, S.A. The insurability of nanomaterial production
risk. Nat. Nanotechnol. 2013,8, 222–224. [CrossRef] [PubMed]
51.
Bouillard, J.X.; Vignes, A. Nano-Evaluris: An inhalation and explosion risk evaluation method for
nanoparticle use. Part I: Description of the methodology. J. Nanopart. Res. 2014,16, 2149. [CrossRef]
52.
Jensen, K.A.; Saber, A.T.; Kristensen, H.V.; Koponen, I.K.; Liguori, B.; Wallin, H. NanoSafer vs.
1.1-nanomaterial risk assessment using first order modeling. In Proceedings of the 6th International
Symposium on Nanotechnology, Occupational and Environmental Health, Nagoya, Japan, 28–31 October
2013; p. 120.
53.
Braakhuis, H.M.; Cassee, F.R.; Fokkens, P.H.; de la Fonteyne, L.J.; Oomen, A.G.; Krystek, P.; de Jong, W.H.;
van Loveren, H.; Park, M.V. Identification of the appropriate dose metric for pulmonary inflammation of
silver nanoparticles in an inhalation toxicity study. Nanotoxicology 2016,10, 63–73. [CrossRef] [PubMed]
54.
Gaiser, B.K.; Hirn, S.; Kermanizadeh, A.; Kanase, N.; Fytianos, K.; Wenk, A.; Haberl, N.; Brunelli, A.;
Kreyling, W.G.; Stone, V. Effects of silver nanoparticles on the liver and hepatocytes
in vitro
.Toxicol. Sci.
2013,131, 537–547. [CrossRef] [PubMed]
55.
Haberl, N.; Hirn, S.; Wenk, A.; Diendorf, J.; Epple, M.; Johnston, B.D.; Krombach, F.; Kreyling, W.G.; Schleh, C.
Cytotoxic and proinflammatory effects of PVP-coated silver nanoparticles after intratracheal instillation in
rats. Beilstein J. Nanotechnol. 2013,4, 933–940. [CrossRef] [PubMed]

Int. J. Mol. Sci. 2018,19, 649 22 of 22
56.
Lankveld, D.P.; Oomen, A.G.; Krystek, P.; Neigh, A.; Troost-de Jong, A.; Noorlander, C.W.; van Eijkeren, J.C.;
Geertsma, R.E.; de Jong, WH. The kinetics of the tissue distribution of silver nanoparticles of different sizes.
Biomaterials 2010,31, 8350–8361. [CrossRef] [PubMed]
57.
Lee, J.H.; Kim, Y.S.; Song, K.S.; Ryu, H.R.; Sung, J.H.; Park, J.D.; Park, H.M.; Song, N.W.; Shin, B.S.;
Marshak, D.; et al. Biopersistence of silver nanoparticles in tissues from Sprague-Dawley rats.
Part. Fibre Toxicol. 2013,10, 36. [CrossRef] [PubMed]
58.
Loeschner, K.; Hadrup, N.; Qvortrup, K.; Larsen, A.; Gao, X.; Vogel, U.; Mortensen, A.; Lam, H.R.; Larsen, EH.
Distribution of silver in rats following 28 days of repeated oral exposure to silver nanoparticles or silver
acetate. Part. Fibre Toxicol. 2011,8, 18. [CrossRef] [PubMed]
59.
Nymark, P.; Catalan, J.; Suhonen, S.; Jarventaus, H.; Birkedal, R.; Clausen, P.A.; Jensen, K.A.; Vippola, M.;
Savolainen, K.; Norppa, H. Genotoxicity of polyvinylpyrrolidone-coated silver nanoparticles in BEAS 2B
cells. Toxicology 2013,313, 38–48. [CrossRef] [PubMed]
60.
Van der Zande, M.; Vandebriel, R.J.; van Doren, E.; Kramer, E.; Herrera Rivera, Z.; Serrano-Rojero, C.S.;
Gremmer, E.R.; Mast, J.; Peters, R.J.; Hollman, P.C.; et al. Distribution, elimination, and toxicity of silver
nanoparticles and silver ions in rats after 28-day oral exposure. ACS Nano
2012
,6, 7427–7442. [CrossRef]
[PubMed]
61.
Yun, J.W.; Kim, S.H.; You, J.R.; Kim, W.H.; Jang, J.J.; Min, S.K.; Kim, H.C.; Chung, D.H.; Jeong, J.;
Kang, B.C.; et al. Comparative toxicity of silicon dioxide, silver and iron oxide nanoparticles after repeated
oral administration to rats. J. Appl. Toxicol. 2015,35, 681–693. [CrossRef] [PubMed]
62.
Lu, S.; Zhang, W.; Zhang, R.; Liu, P.; Wang, Q.; Shang, Y.; Wu, M.; Donaldson, K.; Wang, Q. Comparison
of cellular toxicity caused by ambient ultrafine particles and engineered metal oxide nanoparticles.
Part. Fibre Toxicol. 2015,12, 5. [CrossRef] [PubMed]
63.
Zhang, H.; Ji, Z.; Xia, T.; Meng, H.; Low-Kam, C.; Liu, R.; Pokhrel, S.; Lin, S.; Wang, X.; Liao, Y.P.; et al.
Use of metal oxide nanoparticle band gap to develop a predictive paradigm for oxidative stress and acute
pulmonary inflammation. ACS Nano 2012,6, 4349–4368. [CrossRef] [PubMed]
©
2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).