Content uploaded by Liubo Ouyang
Author content
All content in this area was uploaded by Liubo Ouyang on Oct 22, 2021
Content may be subject to copyright.
978-1-61284-181-6/11/$26.00 ©2011 IEEE 2451
2011 Eighth International Conference on Fuzzy Systems and Knowledge Discovery (FSKD)
A Method of Ontology Evaluation Based on
Coverage, Cohesion and Coupling
Liubo Ouyang, Beiji Zou
School of Information Science & Engineering
Central South University
Changsha, China
oylb@hnu.edu.cn
Miaoxing Qu, Chengming Zhang
School of Software
Hunan University
Changsha, China
qumiaoxing118@163.com
Abstract-A novel method is proposed to evaluate the quality of
ontology. This paper presents a method including three criteria,
such as coverage, cohesion and coupling, to evaluate ontology by
comparing with corpus, designs some experiments for users
adjust the parameters of each criterion depending on their
preference. This paper combined the user-based method with
corpus-based method to propose a set of methods to evaluate
construction of ontology, and it can be proved more accurate and
effective in experiments.
Key words- Ontology Evaluation; Coverage; Cohesion; Coupling;
Corpus
I. INSTRUCTION
With the development of semantic web technology,
information system has become more and more intelligent.
Lots of research works have focused on how to provide more
efficient services to customers. In this regarding, the ontology
technologies which plays a critical role in semantic web and
knowledge system, is regarded as a hot and promising research
area. Particularly the scale and quality of ontology system has
so much impact on the serving capability of the knowledge
system. Meanwhile, with the rapid emergence of the ontology
system has evolved for quite a long time with a variety of
types formed, thereofore how to make a good choice among
those different ontology systems is most confused and
important as unsolved problem for customers. So, this leads to
the ontology evaluation can not be neglected anymore in the
ontology engineering. Ontology evaluation system aims to
calculate the score of ontology by a set of algorithms and
export the relevant results to uers, then enable the users to
select appropriate ontologies. Recently the research on
ontology evaluation has been focused on how to design
evaluation criteria, such as logical consistency and structures.
But the studies do not provide a feasible method to select and
reuse the existed ontologies. The goal of ontology evaluation is
to select suitable ontology for a particular application with
respect to the user requirements in a given context.
Thus, ontology evaluation is an important issue that must
to be addressed if ontologies are widely adopted in the
semantic web and other semantics-aware applications. Users
facing with a multitude of ontologies need to find a way of
assessing these ontologies and deciding which one best meets
their requirements. Likewise, constructing an ontology need to
evaluate the result ontology and guide the constructing process
or any possible refinement steps. Automated or semi-
automated ontology learning schemes also require effective
evaluation measures, which can be used to select the “best”
ontology among different candidates, to set the tunable
parameters values of the learning algorithm, or to direct its
own learning process (if the latter is formulated as a path
through a search space).
The main methods of ontology evaluation in recent
years[1] are listed as below:
• The methods based on user. The main idea of this kind
of method is to evaluate ontologies through users'
experience. The problem is that these criterions are
subjective and lack of objective standards;
• The methods based on the application of ontology. The
risk using this method is that all of the ontologies have
already been in an application before they are evaluated,
hence the users can not know the ontology evaluation
results before using it;
• The methods based on comparing the ontology with a
“golden standard” which may itself be an ontology. The
shortage of these methods is the difficulty to measure the
quality of “golden standard”;
• The methods based on comparing with the source of data,
such as a collection of documents, which covered the
domain of the ontology. Much research has been focused
on these methods, but the drawback is its low efficiency
in comparing the ontology with a source of data.
In summary, it’s extremely a hard work to find a perfect
method for ontology evaluation. In this paper, we design a
mothod based on users and propose a set of selection criteria
and metrics to evaluate ontologies. The most difference
between our method and the existed methods is that we
combine the user-based method with corpus-based method.
Firstly, calculate the value of each criterion by comparing
2452
with corpus, and then users can set their own weights on each
evaluation criterion by themselves. The coverage, cohesion
and coupling criteria of ontology are defined and formulated.
In addition, these criteria are proposed because previous
researches seldom support the evaluation of multiple
ontologies. The proposed metrics can be applied to evaluate
multiple ontologies together.
II. RELATED WORKS
Currently, many research works have focused on the
ontology evaluation. Burton-Jones [2,3] proposed a theoretical
framework, including 10 properties and 4 criterion of
ontology. The score ontology is weighted as the sum of each
criterion. They considered that the scale of ontology is the
determinant of ontology quality. It’s obviously too simple to
measure an ontology by this way. Ruan Jiabin et al.[4]
proposed a novel method to evaluate the comprehensiveness
of ontology. They compared with source of data, improved the
algorithms, and proved efficiency of their methods. However,
as that of Burton-Jones，they did not consider the information
redundancy of big ontology. Oh et al.[5] proposed cohesion
and coupling metrics for ontology modules. They focused on
adapting module metrics of software engineering to the
domain ontology. Their cohesion metric for a module, number
of relations (NR), refers to the number of all the relations
between classes in the module. Coupling metrics number of
separated hierarchical links (NSHL) and number of separated
nonhierarchical links (NSRL) represent the number of
disconnected relations during modularization. They assumed
that a module is more consistent with original ontology than
other modules if fewer relations are disconnected in the
module. Yao et al.[6, 7] considered a set of ontology cohesion
metrics as part of a measure for modular relatedness of OWL
ontologies. They defined three cohesion metrics: Number of
Root (NoR), Number of Leaf (NoL), and Average Depth of
Inheritance Tree of Leaf Nodes(ADIT-LN). Similar to
software cohesion metrics, these cohesion metrics were used
to measure the interrelation of elements in ontologies. NoR is
defined as the number of root classes explicitly defined in
Ontology O. NoL is defined as the number of leaf classes
explicitly defined in Ontology O. ADII-LN is the sum of
depths of all paths divided by total number of paths. Cohesion
metrics are computed based on predefined OWL primitives,
which explicitly define tree-based semantic hierarchies in
OWL ontologies. Orme et al. [7] defined ontology coupling
metrics to measure coupling between ontologies: Number of
External classes (NEC), References of External Classes
(REC), and Referenced Includes (RJ). NEC is the number of
distinct external classes defined outside O but used to define
new classes and properties in the ontology. REC is the number
of references to external classes in Ontology O. RI is the
number of includes at the top of the ontology definition file O,
which the ontology uses to define ontology language
primitives and ontology information defined by other
ontology developers. Yang et al.[8] proposed metrics to
measure the complexity of ontologies, with an emphasis on
the problem of increasing the complexity of maintenance and
management as ontologies evolve. The concepts, the hierarchy
in the conceptual model, and the common features between
most ontologies reflect their fundamental complexity, and
these variables were analyzed. In [9], H. Alani et al. proposed
a method to rank ontology by evaluating concepts
construction, they analysis the concepts and relations in
otology.
Increasing needs for ontology applying and reusing make
it necessary to design the relevant ontology evaluation metrics.
Users have their special ontology applications respectively.
Thus, new criteria and metrics to evaluate ontologies based on
users are very important. Users could set the weight value to
each criterion to meet their requirements.
III. THE PROPOSED METHODS
In order to help users to choose the ontology to meet their
demand best, an evaluation method is designed in this paper.
Firstly we proposed three metrics based on semiotic
framework. The metrics are coverage, cohesion, coupling.
Then we improved the method of measuring coverage by
comparing the number of concepts and relations with a set of
ontologies, improved the method of measuring cohesion by
calculating the tightness of relationship between concepts; and
improved the method of measuring coupling by computing the
number of external classes. Finally we designed two
experiment models to test and verify the influence of different
users and different corpus. Users can set weight value to each
metrics according their requirements.
We propose the definitions of coverage, cohesion and
coupling, and give the calculating formulas of these three
metrics respectively.
A. Coverage
Coverage includes the coverage of concepts and coverage
of relations. Coverage of concepts describes the range of
concepts in ontology. It’s meaningless to measure the quality
by calculating the number of concepts only. Our approach is
to compare the number of concepts with regarding to an
ontology set. Relations include inheritance, synffonyms, and
antonyms and so on; these relations can measure the tightness
among concepts. Coverage of relations can describe the range
of relationships. We propose a definition to estimate the
coverage of ontology.
Definition 1: Let c1, c2,…,cn be the set of n concepts
explicitly defined in an ontology, let S i be a corpus, let Oi be
an ontology in the same domain of S, let m be the number of
concepts in S, let t be a term in search term set, let Con_cov(O)
be the coverage of concepts, Rel_Cov(O) be the coverage of
relations, Cov(O) be the coverage of ontology. The coverage
of concepts calculating is presented as
()
(,)
_(,)
ccO tT
I
ct
Concept Cov O T n
∈∈
=
∑∑
（1）
If there is a t in one ontology, set I be 1; if t is not in
ontology, set I be 0:
1, ( )
(,) 0, ( )
label c t
Ict label c t
=
⎧
=⎨≠
⎩ （2）
2453
1
_(,)
_(,)
k
k
Concept Cov O T
Concept Cov S T m
=
=
∑
（3）
According to formula (1) and formula (3), the coverage of
concepts is obtained:
_(,)
_cov( ) _(,)
Concept Cov O T
Concept O Concept Cov S T
=
（4）
Formula (4) can calculate the rate of concepts related with the
term.
[]
[] []
(,)
Re _ ( , ) (, )
i
ij
i
ccotT
ij
ccocco
J
ct
lCovOT
J
cc
∈∈
∈∈
=
∑∑
∑∑
（5）
If any relation between c and t exists, let J(c,t) be 1, while if
no relation between c and t, let J(c,t) be 0:
J(c,t) = 1, if any relation between c and t
J(c,t) = 0, if no relation between c and t
J(ci,cj ) = 1, if any relation between ci and cj
J(ci,cj ) = 0, if no relation between ci and cj
Formula (5) can get the coverage of relations in ontology set.
First of all, calculate the number of relations between t and
concepts in O. Then let it division with the number of
relations in S:
1
2Re_ (,)
Re _ ( , ) (1)
k
k
lCovOT
lCovST mm
=
=
−
∑
（6）
According to formula (4) and formula (5),we can get the
coverage of relations:
Re _ ( , )
Re _ ( ) Re _ ( , )
lCovOT
lCovO lCovST
=
（7）
Finally, we obtain the coverage of ontology as below:
() _ () Re_ ()Cov O Concept Cov O l Cov O=+
（8）
B. Cohesion
Cohesion traditionally refers to the degree to which the
elements in a module belong together. More particularly, in
object-oriented software, cohesion refers to the degree of the
relatedness or consistency in functionality of the members in a
class; strong cohesion is recognized as a desirable property of
object-oriented classes because it measures separation of
responsibilities, independence of components and control of
complexity [7, 8].
In this paper, cohesion metrics are obtained by compute
the number of relation between different concepts. The idea is
that these ontology cohesion metrics, similar to software
cohesion metrics, can be used to measure separation of
responsibilities and independence of components of
ontologies.
Definition 2: Let c1, c2, …, cn be the concepts in ontology O,
let I(ci,cj) be the relation function, let Coh(O) be the cohesion
of ontology. The cohesion of ontology calculating is presented
as
1[][]
0, 0
1
() ( (,) 1), 1
1, 1
ij
n
ij
iccOccO
n
Coh O I c c n n
n
n
=∈ ∈
⎧=
⎪
⎪
=−>
⎨
⎪
⎪=
⎩
∑∑∑ （9）
In formula（9）, if any relation between ci and cj, let I(ci,cj)
be 1, if no relation between ci and cj, let I(ci,cj) be 0, If there is
no concept in ontology, the cohesion is 0, if there is only one
concept in ontology, the cohesion is 1, because this concept
itself must be the closest construction.
1),(
][][
−
∑
∑
∈∈Occi Occj
ncjciI means compute the number of
concepts related with c
i; make the results divided by the
number of concepts except ci.
Cohesion metrics are used to measure modularity, the
metrics similar to the software cohesion metrics can be
defined to measure relatedness of elements in ontologies.
C. Coupling
Coupling of ontology is the number of class in external
ontologies which referenced by the discussed ontology.
Similar to measuring software modules coupling metrics,
coupling of ontology is measured with classes interfaces
between different ontologies modules.
Definition 3: Let C1, C2, …,Cm be the set of m classes
defined in an ontology, let E={E1,E2,...,Ej} be a set of class
defined in external Oi, let Rj be the total number of references
to Ej, 1<j<m
The number of external classes is formulated as (10).
1
()
m
ij
j
NO E
=
=∑
（10）
The number of external classes referenced is formulated as
(11).
1
Re ( ) ( )
m
ii
j
f
ORO
=
=∑
（11）
The coupling of ontology calculating as (12).
()1Re() ()iiiCou O f O N O=− （12）
The stronger coupling in ontologies, the more difficult to
understand, change and apply ontologies.
D. Ontology Evaluation
1) User’s evaluation
According to the definition and formula above, we can
obtain the evaluating value of each ontology. Although each
2454
metric is considered important, and could be equally weighted,
the weights for each one could be allowed to set according to
the users' requirements.
So two evaluating value of ontology is obtained, one is to
set variable weight by users, the other is to set weights equally.
We formulate the two values respectively as (13) and (14).
Var ( O ) =w 1*Cov(O)+w2*Coh(O)+w3*Cop(O), 0<wi<1 (13)
Equ(O)=(Cov(O)+Coh(O)+Cop(O))/3 (14)
And the evaluating value of users is formulated as (15):
User(O)=(Var(O)+Equ(O))/2 (15)
2) Experts’ evaluation
The evaluated results of domain experts are divided into 5
grades which from grade 1 to 5. So we set coefficient on the
scale of 0.1-0.5 to get the value of experts evaluation. For
example, Exp_V(O)=0.1*Va r ( O ) , Exp_E(O)= 0.1*Equ(O).
The evaluated value of experts as formula (16):
Exp(O)=(Exp_V(O)+Exp_E(O))/2 (16)
3) value of evaluation
Finally, the evaluated value of ontology is presented as
(17):
Score(O)=User(O)+Exp(O) (17)
IV. EXPERIMENTS AND VALIDATION
We choose two groups in different domains, the
information of ontology is shown in table I.
• Group 1: project1, project2 and project3, respectively
come from “http://www.cs.umd.edu/projects/plus /DAM
L/onts/base1.0.daml”,“http://www.cs.umd.edu/projects/pl
us/DAML/onts/base2.0.daml”,“http://www.cs.umd.edu/pr
ojects/plus/DAML/onts/base3.0.daml”, these ontologies
are in the domain of project.
• Group 2: environment1, environment2, respectively from
“http://sweet.jpl.nasa.gov/1.0/environment.owl”, “http://
sweet.jpl.nasa.gov/2.0/environment.owl”, these two
ontologies concern about environment and health.
TABLEI. ONTOLOGY INFORMATION
ontology Number
of
concepts
Number
of
relations
Number
of class
Number of
external
classes
referenced
project1 189 124 132 38
project2 163 129 121 35
project3 168 130 124 32
environment1 436 247 269 46
environment2 493 243 258 47
Then, we aim to calculate the value of coverage, cohesion
and coupling respectively. Based on the coverage calculating
method, we select two corpus in OANC(the open part of the
American National Corpus ) to be the references. The corpus
1 is about technical; the corpus 2 is about ecology. To be
convenient, we only abstract the noun in corpus. Whereas the
concept which has low rate of appearance has little influence
to ontology coverage, we descend order of concepts by their
frequency. The results is clearly shown in figure 1 and figure
2 , the Horizontal axis represents the set of concepts in
descending order, and the Vertical axis represents the score of
ontology after tested. We compared group 1 with corpus 1;
compared group 2 with corpus 2.
As shown in the following figures, the experiment results
imply that small ontology may obtain a high score of coverage
because of selected a valid corpus, big ontology may obtain a
low score because of selected an unsuitable corpus. We can
infer that the score of ontology is variable depending on their
corpus selected. In the beginning of the test, we see the line
increasing rapidly, that’s because the most frequent concepts
are matching at first, and the low frequency concepts influence
the result rarely. And so, we made the conclusion that
pretreated the concepts can improve the efficiency of
computing coverage.
Figure1. The coverage value of group 1 compared with corpus 1
Figure2. The coverage value of group 2 compared with corpus 2
2455
We then calculate the value of cohesion and coupling.
Finally, we obtained the value of each criterion showed in
table II.
TABLEII. VALUE OF EACH CRITERION
ontology coverage cohesion coupling
project1 0.543 0.431 0.723
project2 0.482 0.445 0.711
project3 0.586 0.469 0.742
environment1 0.231 0.428 0.829
environment2 0.247 0.419 0.818
In order to make the results more clearly, we let w1=0.5,
w2=0.3, w3=0.2, and we get the value of users’ evaluation in
table III:
TABLEIII. NORMAL USERS’ EVALUATION
ontology Var(O) Equ(O)
project1 0.545 0.56
project2 0.517 0.546
project3 0.582 0.599
environment1 0.410 0.496
environment2 0.411 0.495
Ten domain experts and ontology experts are selected in
the following experiment, and then obtained the final score of
ontology as shown in table IV.
TABLEIV. EXPERTS EVALUATION AND FINAL SCORE
ontology Exp(O) Score(O)
project1 0.14 0.693
project2 0.15 0.682
project3 0.23 0.821
environment1 0.21 0.668
environment2 0.23 0.683
According to the tables above, we get that the number of
concepts can not measure the quality of ontology. That is
because the coverage must be estimated by comparing with a
corpus, and if the number of concepts in corpus is exactly far
more than it is in ontology, even though the number of
concepts in this ontology is large, the value of coverage is low.
In other words, the value depends on the scale of corpus. The
second point is that the weights can influence the score of
ontology also.
V. CONCLUSION
In this paper, we proposed three metrics for measuring
ontology coverage, cohesion and coupling which can enable
ontology users to better understand ontology and select the
desired ontology meeting their requirements.
We have performed theoretical and empirical analysis of
the ontology using three metrics, designed two experiments to
prove our method is effective. In first experiment, we
calculate the score of five ontologies, analyze the results, and
select the corpus which will affect the results. In the second
experiment, we focus on the influence of corpus, and
following points should be noticed.
1) The ontology evaluation should be discussed in the same
domain as the ontology evaluation will be meaningless in
facing of different domains;
2) Choose suitable corpus can make score more reliable.
According to the experiments, it implies that the value of
coverage will be different by comparing with different
corpus. In order to improve the efficiency of computing,
we need to abstract noun in a descending order to a new
corpus.
3) The user’s preference influences how to set weights to
each criterion, and impacts on the final scores.
In the future, we will focus on the ontology evaluation
system especially. We may add other criteria, such as
consistence, and design an evaluation system. For improving
the efficiency of calculating, we will do some research in
computing the frequency of concepts. Also, further research
may include how to use additional metrics and how the
metrics effect ontology system development.
ACKNOWLEDGMENT
This paper is supported by the National Natural Science
Funds of China (No.60970098 and No.60803024), and Major
Program of National Natural Science Foundation of China
(No. 90715043).
REFERENCES
[1] Brewster, et al. Data driven ontology evaluation, In Proceedings of
International Conference on Language Resources and Evaluation,
Lisbon, 2004
[2] Burton-Jones, S. Purao and V.C. Storey. A semiotic metrics suite for
assessing the quality of ontologies. Data and Knowledge Engineering,
2004
[3] Burton-Jones, S. Purao and V.C. Storey. Context-aware query
processing on the semantic web. In Proceedings of the 23rd
International Conference on Information Systems, Barcelona, Spain,
2002 , pp. 16-19
[4] J.B.Ruan,Y.B.Yang, J.J.Lin, H.G.Nan. Evaluate the comprehensiveness
of ontology(CH). Journal of Frontiers and Computer Science and
Technology, 3(6), 2009, pp. 633-640
[5] Oh,S.J., J.Ahm. Ontology Module Metrics. In Proceedings of the
International Conference on E-Business Engineering, 2009
[6] H.Yao, A. M. Onne and L. Etzkorn. Cohesion Metrics for Ontology
Design and Application(CH), Journal of Computer science, 1(1), 2005,
pp.107-113
[7] M. Orme, H. Yao and L. H. Etzkorn, Coupling Metrics for Ontology
-Based Systems. IEEE Software, 2006, pp.102-108
[8] Z.Yang, D. Zhang and C. Ye. Evaluation Metrics for Ontology
Complexity and Evolution Analysis. In Proceedings of IEEE
International Conference on E-Business Engineering , 2006, pp.34-39
[9] H. Alani, C. Brewster. Ontology Ranking based on the Analysis of
Concept Structures. In Proceedings of the Third International
Conference on Knowledge Capture ( K-CAP 05), Banff, Canada, 2005
