FIG 5 - uploaded by Carlo Meghini
Content may be subject to copyright.
Source publication
A multimedia document may be seen as a complex information object, with components of different kinds, such as text, images, video and sound, all in digital form. An ever growing amount of people need to retrieve the multimedia documents that would prove useful to their information needs. Research on multimedia information retrieval (MIR) has witne...
Contexts in source publication
Context 1
... QUERIES. The grammar given in Figure 5 defines the document query language. The categories capturing text and image queries will be defined later and are underlined for the reader's convenience. ...
Context 2
... the most gen- eral case, a query begins by addressing document structure, so the definition of begins, in Figure 9, from document queries, whose syntax is given in Figure 5. In illustrating , we follow the order established by Figure 9 and the forthcoming figures that make up 's definition. ...
Citations
... Single modal information search for these studies mainly focused on the influence factors in the field of information search behavior (Liu et al., 2019) and behavioral characteristics research (Su, 2011), technical aspects of a single modal feature extraction, the search engine (Pan, 2004), search technology, and research on MMISB is less. In addition, some experts and scholars have also studied MMIS, but their research focus is multi-modal information fusion research (Carlo et al., 2001), which focuses on the research of combinatorial ordering of diversified information features related to different or the same description target. Finally, research on MMIS still focuses more on technical fields, such as models and algorithms used cross-modal, with the purpose of improving the efficiency of MMIS through technical means. ...
Purpose
With the increasing abundance of network resources and big data, multi-modal information search (MMIS) has been paid more and more attention, but the research results of MMIS are relatively few. This paper attempts to put forward constructive suggestions for the design of multi-modal information system, so that the system can have a better user experience, help users improve the efficiency of obtaining information and optimize the information service mode.
Design/methodology/approach
A research model of influencing factors is established by using the TAM (technology acceptance model) theory. The influencing factors of users' multi-modal information search behavior (MMISB) are analyzed by using questionnaire, experiment and the structural equation model. On the basis of this, some suggestions are put forward to build the multi-modal search (MMS) system and improve the efficiency of MMIS.
Findings
The research shows that users' MMISB is directly related to their search intention, and the search intention can influence users' cognition of the usefulness and ease of MMIS through their own information search ability and system characteristics. The user's MMIS ability is affected by the demand expression ability and retrieval ability cognition; the user's cognition of system characteristics is affected by the system function and information quality. This shows that the user's MMISB is closely related to the user's cognitive situation, but due to the author's limited time and research ability, only the questionnaire survey method cannot be used to in-depth research and explore the influencing factors of MMIS. Therefore, in the future research, we should combine the interview method to further track the user's emotional factors and scene factors.
Originality/value
For the first time, TAM theory is combined with cross-modal retrieval behavior and the paper explores the influencing factors and evaluation indexes of users' MMISB. The second, the questionnaire was compiled to investigate the influencing factors of the MMISB of the university group, and the reliability analysis, validity analysis, correlation analysis and structural equation model analysis of the survey data are carried out . The survey data and analysis results are original, which can provide a theoretical basis for improving the service level of MMIS.
... CBIR systems with high-level semantics are essentially CBIR systems that are able to learn high-level semantic concepts from low-level visual features using computer vision and machine learning techniques, focusing on reducing the semantic gap. Such systems include semantic-based image retrieval systems [5,21,23] and signal/semantic-based systems [3,29]. Bradshaw et al. [5] use a probabilistic model in their semantic-based system to recognize four concepts (i.e. ...
The contextual information of Web images is investigated to address the issue of enriching their index characterizations with semantic descriptors and therefore bridge the semantic gap (i.e. the gap between the low-level content-based description of images and their semantic interpretation). Although we are highly motivated by the availability of rich knowledge on the Web and the relative success achieved by commercial search engines in indexing images using surrounding text-based information in webpages, we are aware that the unpredictable quality of the surrounding text is a major limiting factor. In order to improve its quality, we highlight contextual information which is relevant for the semantic characterization of Web images and study its statistical properties in terms of its location and nature considering a classification into five semantic concept classes: signal, object, scene, abstract and relational. A user study is conducted to validate the results. The results suggest that there are several locations that consistently contain relevant textual information with respect to the image. The importance of each location is influenced by the type of webpage as the results show the different distribution of relevant contextual information across the locations for different webpage types. The frequently found semantic concept classes are object and abstract. Another important outcome of the user study shows that a webpage is not an atomic unit and can be further partitioned into smaller segments. Segments containing images are of interest and termed as image segments. We observe that users typically single out textual information which they consider relevant to the image from the textual information bounded within the image segment.
... video sequences or video still images, rather than to others first. The general model we are inspired on is based on [20]. Conceptually, according to [20], a media object o (e.g., an image region, a video sequence, a piece of text, etc.) is annotated with one (or more) entities t of the ontology (see e.g. Figure 7). ...
... The general model we are inspired on is based on [20]. Conceptually, according to [20], a media object o (e.g., an image region, a video sequence, a piece of text, etc.) is annotated with one (or more) entities t of the ontology (see e.g. Figure 7). As specified in [20], such an annotation may come manually from a user or, if, available, from an image classifier. ...
... Conceptually, according to [20], a media object o (e.g., an image region, a video sequence, a piece of text, etc.) is annotated with one (or more) entities t of the ontology (see e.g. Figure 7). As specified in [20], such an annotation may come manually from a user or, if, available, from an image classifier. In the latter case, it may annotate the image automatically, or, semi-automatically by suggesting to a human annotator, which are the most relevant entities of the ontology that may be used for a specific media object o. ...
The detection and representation of events is a critical element in automated surveillance systems. We present here an ontology for representing complex semantic events to assist video surveillance-based vandalism detection. The ontology contains the definition of a rich and articulated event vocabulary that is aimed at aiding forensic analysis to objectively identify and represent complex events. Our ontology has then been applied in the context of London Riots, which took place in 2011. We report also on the experiments conducted to support the classification of complex criminal events from video data.
... Fuzzy set theory and fuzzy logic [318] have proved to be suitable formalisms to handle fuzzy knowledge. Not surprisingly, fuzzy ontologies already emerge as useful in several applications, such as information retrieval [3,67,175,298,299,311,319], recommendation systems [71,164,224,314], image interpretation [95,96,97,215,254,258,259], the Semantic Web and the Internet [80,226,241], ambient intelligence [103,104,174,235], ontology merging [75,301], matchmaking [2,79,227,228,229,230,231,296,297], decision making [281], summarization [163], robotics [111,112], machine learning [166,167,168,169,170,171,172,173,294] and many others [7,94,113,140,158,165,176,208,225,234,248,282]. ...
We present the state of the art in representing and reasoning with fuzzy knowledge in Semantic Web Languages such as triple languages RDF/RDFS, conceptual languages of the OWL 2 family and rule languages. We further show how one may generalise them to so-called annotation domains, that cover also e.g. temporal and provenance extensions.
... In order to cope with the uncertainty issue in multimedia indexing, Fuzzy-DL [35], [36] is considered as an interesting formalism for representing a multimedia ontology. In fact, the fuzzy Description Logics is considered as a very interesting logical formalism as it can be used in several domains like multimedia information retrieval [37] to provide ranking degrees and to manage with indistinguishable concepts like tall, short and many more. ...
Through the success of deep convolutional neural networks (CNN) for image classification and semantic concepts detection, the multimedia retrieval community provides interesting image analysing approaches and tools in order to deliver accurate semantic interpretation. Never the less, such approaches still focusing only on explicit information and objects that exist in content. Considering that implicit information could enhance the semantic interpretation, we are interested in a knowledge based framework to detect the semantic context. In this paper, we discuss a fuzzy ontology based approach for understanding image content through the detection of the contained context. We conducted preliminary experiments on the IMAGENET2017 dataset. While the obtained results still not impressive, many open research direction could be tackled. Indeed, we think that a deep based knowledge management (in particular knowledge extraction and reasoning) could be considered as interesting and promising.
... The second class of approaches is mainly based on description language formalization. In multimedia information retrieval, many works proposed logic based models [17,33,50]. These models formalize the semantic dimension of multimedia content using logical formalism such as the description logic which is considered as the most important knowledge representation formalism. ...
... Several authors [17,33] propose to enhance the description logic by a rules layer which provides a powerful query language and improves roles expression. The hybrid language (DL and rules) combined with the visual dimension are used as a complementary answer to users' requests [17]. ...
In the literature, several image retrieval approaches that allow mapping between low-level features and high-level semantics have been proposed. Among these one can cite object recognition, ontologies, and relevance feedback. However, their main limitations concern their high dependence on reliable external resources (existing ontologies, learning sets, etc.) and lack of capacity to combine semantic and visual information and provide relevant results. This paper proposes a system aiming to improve image retrieval results. The proposed system is based on a pattern graph combining semantic and visual features. The idea is (1) to automatically build a modular ontology based on a learning step from textual corpus and terminological resource, (2) to organize visual features in a graph-based model where the combined module and graph represent a unique component called “pattern,” and (3) to build a pattern graph. To this end our system has been implemented. The obtained experimental results show that the pattern graph that we propose enables an improvement of retrieval task.
... The importance of this study is highlighted by several applications that have been considered. Some of these applications include medicine [81], information retrieval [79,101], recommendation [50,53], and detection [52,54]. Another important aspect is the problem of constructing such ontologies in the first place. ...
Mathematical Fuzzy Logics [51, 60] have a long tradition with roots going back to the many-valued logics of Łukasiewicz, Gödel, and Kleene [57, 68, 73] and the Fuzzy Set Theory of Zadeh [111]. Their purpose is to model vagueness or imprecision in the real world, by introducing new degrees of truth as additional shades of gray between the Boolean true and false. For example, one can express the distinction between a person x having a high fever or a low fever as the degree of truth of the logical statement \(\mathsf {Fever} (x)\). One of the central properties of fuzzy logics is truth functionality—the truth degree of a complex logical formula is uniquely determined by the truth degrees of its subformulas. This is a fundamental difference to other quantitative logics like probabilistic or possibilistic logics [56, 83].
... Therefore, multimedia technology, mainly images, are increasingly used. Works in this research area are multiple [28,60] and the problems are various [68,52,62]. Image databases now represent very large volume of information but they are unfortunately poorly exploited. ...
... Semantic image retrieval approaches use generally a formalism of knowledge representation such as description logics or semantic networks [62]. The goal is to find a model of image representation. ...
... Their ontology is a multi levels model: graphic descriptors, images subjects and other medical terms. A 1 recall [60] is the number of relevant images returned divided by the number of all relevant images 2 precision [60] is the number of relevant images returned divided by the number of images returned last work can be cited is the work of Meghini et al. [62] which propose a framework based on fuzzy description logics to integrate the multidimensional aspects of multimedia information retrieval. ...
Querying efficiently images using an image retrieval system is a long standing and challenging research problem.In the medical domain, images are increasingly produced in large quantities due their increasing interests for many medical practices such as diagnosis, report writing and teaching. This thesis proposes a semantic-based gastroenterological images annotation and retrieval system based on a new polyp ontology that can be used to support physicians to decide how to deal with a polyp. The proposed solution uses a polyp ontology and rests on an adaptation of standard reasonings in description logic to enable semi automatic construction of queries and image annotation.A second contribution of this work lies in the proposition of a new approach for computing relaxed answers of ontological queries based on a notion of an edit distance of a given individual w.r.t. a given query. Such a distance is computed by counting the number of elementary operations needed to be applied to an ABox in order to make a given individual a correct answer to a given query. The considered elementary operations are adding to or removing from an ABox, assertions on atomic concept, a negation of an atomic concept or an atomic role. The thesis proposes several formal semantics for such query approximation and investigates the underlying decision and optimisation problems.
... This need addresses all dimensions of a document: it addresses structure because it states conditions on several parts of the desired documents; it addresses profile because it places a restriction on the production date; it addresses form-(in particular color-) and semantic-based image retrieval on a specific region of the involved image (the region must be blue and represent the singer Kiri) as well as on the whole image (must be a scene of a Mozart's opera); it addresses from-based text retrieval by requiring that the document contains a piece of text of a certain type and content. This is an example of mixed MIR, allowing the combination of different types of MIR in the context of the same query [10]. ...
... In order to cope with the uncertainty issue in multimedia indexing, Fuzzy-DL [Straccia 1998;, Stoilos et al. 2007, Ma et al. 2013] is considered as an interesting formalism for representing a multimedia ontology. In fact, the fuzzy Description Logics is considered as a very interesting logical formalism as it can be used in numerous domains like multimedia and information retrieval [Meghini et al. 2001] to provide ranking degrees and to cope with vague concepts like "near", "far" and many more. ...
Our thesis work deals with the video indexing based on semantic interpretation (an abstrac-
tion of objects or events that figure in a content), more particularly, the semantic indexing
enhancement. Various approaches for semantic multimedia content analysis have been pro-
posed addressing the discovery of features ranging from low-level features (color, histograms,
sound frequency, motions, . . . ) to high-level ones (semantic objects and concepts). However,
these earlier approaches failed to reduce the semantic gap and were not able to deliver an
accurate semantic interpretation. Under such a context, exploring further semantics within
a multimedia content to improve semantic interpretation capabilities, is a major and a pre-
requisite challenge.
Towards exploring further semantic information within a multimedia content (other than
low-level and semantic concepts one), valuable information (mainly concepts interrelation-
ships and contexts) could be gathered from a multimedia content in order to enhance semantic
interpretation capabilities. Motivated by a kindred vision of human perception, yet targeting
automated analysis of a multimedia content, the multimedia retrieval community addressed
more attention to multimedia ontologies.
Aiming to contribute towards this direction, we focus on modeling an automated fuzzy
context-based ontology framework for enhancing a video indexing accuracy and efficiency.
Key dimensions of this inquiry constitute the main issues addressed by the use of ontologies for
multimedia indexing, namely: (1) the knowledge management and evolution, (2) the ability
to handle uncertain knowledge and to deal with fuzzy semantics, and (3) the scalability and
the ability to process a growing multimedia content volume with a continuous request for a
better machine semantic interpretation capacities.
What was accomplished in our study is a novel ontology management which is intended
to a machine-driven knowledge database construction. Such a method could enable semantic
improvements in large-scale multimedia content analysis and indexing.
In order to illustrate the semantic enhancement of concept detection introduced by our
proposed scalable and generic ontology-based framework, we have conducted different ex-
periments within three multimedia evaluation campaigns: TrecVid 2010 (within Semantic
Indexing Task), ImageClef 2012 (within Photo Annotation and Retrieval Task), and Image-
Clef 2015 (within Scalable Concept Image Annotation Task).
