A Novel Approach based on Commonsense Knowledge Representation and Reasoning in Open World for Intelligent Ambient Assisted Living Services

Abstract

The next generation of ambient assisted living services will be based on eco-systems or organizations of intelligent artificial agents embodied in companion robots and smart objects. To provide, anywhere and anytime, smart assistance services to people, these agents need to be endowed with advanced knowledge representation, reasoning and communication capabilities. In this paper, we propose a distributed cognitive architecture allowing to integrate seamlessly the actors of the ambient system and an expressive model for commonsense knowledge representation and reasoning on events. This model allows a common description of the actors in an open world and a management of interactions with humans using natural language. A scenario dedicated to the cognitive assistance of frail people is implemented and analyzed for validation purposes of the proposed approach.

Full text

A Novel Approach based on Commonsense Knowledge Representation
and Reasoning in Open World for Intelligent Ambient Assisted Living
Services
N. Ayari1, A. Chibani1, Y. Amirat 1and E. T. Matson 2
Abstract— The next generation of ambient assisted living
services will be based on eco-systems or organizations of intelli-
gent artificial agents embodied in companion robots and smart
objects. To provide, anywhere and anytime, smart assistance
services to people, these agents need to be endowed with
advanced knowledge representation, reasoning and communi-
cation capabilities. In this paper, we propose a distributed cog-
nitive architecture allowing to integrate seamlessly the actors of
the ambient system and an expressive model for commonsense
knowledge representation and reasoning on events. This model
allows a common description of the actors in an open world
and a management of interactions with humans using natural
language. A scenario dedicated to the cognitive assistance
of frail people is implemented and analyzed for validation
purposes of the proposed approach.
I. INTRODUCTION
With the emergence of Ambient Intelligence (AmI) in
our everyday life, the interaction between human users and
their environment is becoming ubiquitous. According to this
vision, ambient-assisted living spaces can be seen as smart
spaces composed of pervasively distributed entities (devices,
sensors, actuators, smartphones, appliances, etc.) offering
heterogeneous capabilities abstracted as software services.
According to the paradigm of context awareness, such sys-
tems will be able to monitor humans, and provide them
services according to their context. To provide, anywhere and
anytime, smart assistance services to people, these agents
need to be endowed with advanced knowledge representa-
tion, reasoning and communication capabilities [3], [9], [15].
One of the most important challenges in ambient intelli-
gence is to have a common description of the world, shared
by actors of the ambient environment, such as human users
and artificial agents. This description is often based on the
closed world assumption where agents take into account only
what is modeled before execution. However, these agents
operate in an open world where they can discover new
entities that are unknown before execution such as new
objects and new words. To address these challenges, an
automatic update of the commonsense knowledge is required.
Building efficient cognitive models for better handling
human-environment-robots interactions requires a suitable
architecture allowing to coordinate between different cog-
nitive entities. Another important challenge is to allow these
1N. Ayari, A. Chibani and Y. Amirat are with LISSI
Laboratory, University of Paris-Est Cr´
eteil (UPEC), France
{naouel.ayari,chibani,amirat}at u-pec.fr
2E. T. Matson, M2M Lab/Rice Center, Purdue University, USA
ematson at purdue.edu
cognitive entities to collaborate seamlessly. This poses some
specific requirements. The first requirement regards the com-
munication capabilities and distributed intelligence for ratio-
nal decision making. The second requirement is related to
natural human-environment interactions which are addressed
in this paper through the natural language modality.
In terms of application, this study focuses on cognitive
assistance of dependent people at home. According to previ-
ous social studies, two kinds of cognitive assistance services
appear of major importance in ambient assisted living en-
vironments. The first one concerns the notification services
to notify a person about important events such as: a product
in the refrigerator is missing, remind an elderly person to
take medicine, etc. The second kind of service involves a
natural language interaction between the assisted person and
the system. Such a service aims to help a person to know
what is in, what is missing or what is happening in their
living environment and, what is the possible cause of a given
event or situation. To address these challenges, expressive
and efficient knowledge representation and reasoning models
for the open world are required to render assistive agents
more autonomous and able of taking context aware decisions
based on the current context/situation of the assisted person.
In this paper, a novel approach for commonsense knowl-
edge representation and reasoning in the open world is
proposed. The resulting model exploits both the narrative
knowledge representation language (NKRL) [7] and Col-
lective Intelligence [20]. The contributions of the paper
can be summarized as follows: (i) a distributed cognitive
architecture allowing to integrate seamlessly the actors of the
ambient system, (ii) an expressive model for commonsense
knowledge representation and reasoning on events. This
model allows a common description of the actors in the open
world and a management of interactions with humans using
natural language.
The paper is structured as follows: Section II presents a
review of related works concerning, on the one hand, the
integration of heterogeneous and distributed components in
network robot systems, and on the other hand, ontology-
based knowledge representation and reasoning in the robotics
field. Section III describes the different layers of the proposed
distributed architecture for human-robot-agents interaction.
Section IV describes the novel approach for commonsense
knowledge representation and reasoning in open world. Sec-
tion V details the implementation and the evaluation of the
proposed approach through a scenario of cognitive assistance

of frail people. This paper is concluded with a short review
of the proposed approach and a summary of the ongoing
works.
II. REL ATE D WORK
Integrating, into one uniform system, robots, devices,
everyday objects, and humans is an important topic that
has been addressed these latest years in ambient intelligence
and robotics communities. Multi-agents and Service Oriented
Computing paradigms have inspired numerous approaches.
The ecology of physically embedded intelligent system
(PEIS Ecology) [2] and GiraffPlus system [18] are the
most popular middleware approaches and a good example
of the application of these paradigms. In [10], an agent-
based approach along with belief models based on first-
order logical predicates and automated planning components
are used for coordination in Human-Robot Teams. Authors
propose in [17] an approach to endow robots with capability
to discover the evolving world and update their common-
sense knowledge by proactively cooperating with humans.
In [8], by using the design pattern of proxies, the author
proposes a representation of the properties and capabilities
of humans inhabitants in a network robot system in a similar
way as of other network components. Most of the proposed
approaches addresses the problem of interoperability in terms
of service accessibility or uses multi-agent systems for a
more flexible management of systems with respect to their
intrinsic complexity.
Semantic knowledge management using ontologies has
been intensively studied in the fields of ambient intelligence,
pervasive computing, and recently in robotics as a mean
to endow Ambient Assited Living (AAL) and robots with
advanced cognitive capabilities for reasoning on contexts
and situations. Most of these approaches concern the use
of the Web Ontology Language (OWL). With respect to the
field of assistive & service robotic, most of the proposed
approaches suggest the use of OWL and RDF Schema (RDF-
S) for defining and representing standard semantic models
with conceptual query tools for managing commonsense
knowledge. Managing commonsense knowledge is relevant
for planning robots’ tasks and handling their dialogues with
humans. In [11], the authors propose the Cognitive Robot
Abstract Machine (CRAM) software for the implementation
and control of complex mobile manipulation robotic tasks.
The KNOWROB (KNOWledge processing for ROBots)
component has been used in CRAM and recently in the
RoboEarth infrastructure [12], as a platform enabling the
robots to share and manage an abstract and consistent view
of the observable world and plans by using computable pred-
icates expressed in SPARQL according to an OWL Ontology.
The latter is a domain ontology that extends a subset of the
concepts introduced in the OpenCyc upper ontology. In [17],
authors propose an ontological platform, called OpenRobots
Common Sense Ontology (ORO), dedicated for grounding
robots natural interaction with humans and resolving in
some cases the ambiguities in dialog situations. Most of
the aforementioned approaches are valuable for describing
easily the entities composing the real world and their mutual
relations. However, the unary and binary structures of the
predicates in OWL and RDF-S result in complex models
when they are used to describe the semantics of dynamic
entities omnipresent in the interactions between humans,
robots and other objects of an ambient intelligent space,
namely, events, actions, situations, circumstances [7]. In-
deed, the highly dynamic nature of Ambient Intelligence
spaces, poses an important challenge to encode spatio-
temporal relations between static and dynamic entities. The
application of the concept of reification to express these
relations renders knowledge representation models and their
associated reasoning models more complex and less efficient
[1]. The literature shows that merging n-ary ontologies
and the reasoning with multi-agent system has not been
investigated in the field of cognitive robots. Besides, natural
language based interactions between human users and robots
or smart systems have also not been sufficiently investigated
due to the lack of expressiveness of the existing knowledge
representation models.
III. DISTRIBUTED COGNITIVE ARCHITECTURE FOR
HU MAN -ROBOT-AGENTS INTERACTION
In this paper, a distributed cognitive architecture for
human-robot-agents interaction is proposed (Fig. 1). This
architecture is based on the HARMS model [19] and NKRL,
a knowledge representation model for narrative reasoning
and natural language processing. It includes four layers that
are embeded in each artificial agent.
Fig. 1. Distributed cognitive architecture
The Network layer represents the basic communication
between the actors of the system. Each system actor must
have basic capabilities to connect, via wired or wireless
network, to any other system actor and to send/receive
messages, using the standardized middleware technologies.
A group of actors is defined by a set Actor =
{H,A,R,M,S}where His a group of humans, Ais a group
of agents, Ris a group of robots, Mis a group of machines
and Sis a group of sensors. Indeed, an actor a1∈Actor
exchanges messages with any actor ai∈Actor;i=2, ..., n.

The Communication layer enables the basic exchange
capability between any system actors by more focusing on
the encoding of messages’ content, which is defined by
elements such as lexicon, grammar, speech acts, semantics
and ontology. In this paper, we focus on two speech acts
types: declaration (e.g. information message) and directive
(e.g. requests and commands).
Formally, the messaging function N ot i f , represented in
Eq. 1, describes the fact that the actor exp is sending an
information MsgContent to the actor dest.
In f M sgexp ←Not i f (dest,MsgContent );
exp,dest ∈Actor;(1)
The query function Query, Eq.2, describes the actor exp
asking the actor dest for an information QContent.
Questexp ←Query(dest,QContent );
exp,dest ∈Actor;(2)
The Interaction layer provides a set of common techniques,
algorithms and technologies for group rational decision
making.
The collective intelligence and organization layers are
independent from the other three layers. They concern team
work behaviors that involve a collection of agents, robots and
humans. Such a behavior can lean in a number of different
directions and quickly become a very complex aggregated
behavior. Moreover, the collective intelligence will not
only allow emergent behaviors, but also the connection of
multiple organizations into higher-level collectives such as
societies or organizations. An organization is defined by a
group of actors Awho have a capability Cto play a role R
in order to satisfy a set of goals G. Formally, it is defined
by O=hA,C,R,Gi
The concepts of indistinguishability and context awareness
are very important in the design of cognitive capabilities
of HARMS actors. Indistinguishability enables an actor to
choose between ndifferent options of minimally capable
actors relative to some task or goal [19]. Context awareness
refers to the ability of actors to take into account any
sensed information or recognized patterns that are essential
to characterize the situations of the actors themselves and/or
the situations concerning the actors with which they may
interact or communicate.
IV. COMMONSENSE KNO WL EDG E REPR ESE NTATI ON AN D
REASONING IN OPE N WOR L D
A. Advanced Knowledge Representation
To take into account both the ’static’ and ’dynamic’ char-
acteristics of any entity populating the world, an ontological
model based on the Narrative Knowledge Representation
Language (NKRL) is proposed. While static characteris-
tics concern in general identity and structural description,
dynamic characteristics are used to describe what can be
observed or inferred in a specific context such as events,
actions and situations. Without ambitioning to provide any
sort of universal coverage, the model proposed in this work
describe in ’simplest and conceptual’ way any ’static’ or
’dynamic’ entity. Static entities and dynamic entities are
modeled, respectively, by using the hierarchy of classes
”NKRL HClass ontology” and the hierarchy of templates
”NKRL n-ary HTemp ontology”.
1) HClass: The first ontology, HClass, is a hierarchy of
binary classes close to any DL or RDF representation used
in the semantic web upper ontologies to describe plain/static
commonsense knowledge. This ontology is characterized
by the following common properties of semantic web on-
tologies: the generalization/specialization relationship (IsA
link) and individual (instantiate of concept). The relatively
semantic interpretation of IsA states that this relationship
among concepts, denoted (IsA C2C1), means that the concept
C2is a specialization of the more general concept C1. This
assertion can be expressed by formula 3, which means , for
instance, that any humanoid (C2) entity IsArobot (C1) entity
and if KOMPAI is a humanoid , then it is also a robot . This
relationship is implemented according to binary model.
∀x(C2(x)→C1(x)) (3)
Furthermore, an individual is an instance I1of a concept C1
that is represented with the property InstanceOf and denoted
(InstanceOf I1C1). Instead of the relation IsA where the
subclass C2of C1denoted by C2⊂C1, the instance I1of
the concept C1is denoted by I1∈C1. The HClass ontology
allows for the non-differentiation between instance or leaf
concept in the taxonomical hierarchy. This property is very
helpful during the pattern matching process of query an-
swering. The HClass ontology has an autonomous existence,
with respect to the description of the dynamic entities in
the HTemp ontology. This allows to take into account well-
defined classes of cognitive phenomena without considering
the specificities of their contexts.
2) HTemp: The second ontology, HTem p, is an n−ary
ontology that allows to define templates for representing
dynamic entities based on the notions of ”conceptual pred-
icate” and ”functional roles”. Recursive lists of symbolic
labels are used for representing correctly complex dynamic
entities and their relationships. Conceptual predicates are
used to represent the spatio-temporal context as temporally
ordered and logically/semantically coherent blocks of dy-
namic entities, called NKRL events. These predicates consist
of the description of a particular action, state, situation, etc.
They correspond to the syntactic/grammatical ”verbs”, also
they correspond to adjectives and nouns when they have a
predicative function. NKRL allows to represent in the best
way each context of the elementary events using a structured
n-ary scheme, whose ’core’ is denoted by Eq. 4:
(Li(Pj(R1a1)(R2a2)...(Rnan))) (4)
Where:

•Liis a ”symbolic label” identifying the elementary event
to be represented (e.g., the event : ”The Robot moves
to Person’s Location”),
•Pjis a ”conceptual predicate”, i.e., a deep level gener-
alization of a particular surface predicate, independent
from a specific natural language (e.g. the predicate
”MOVE ” represents the action of moving from one
place to another or from one state to another).
•Rkis a generic ”functional role”, like SUBJ(ect),
OBJ(ect), etc., i.e., the formalization of the relationship
between the predicate and one of its arguments akthat
explains the specific function of akin the context of the
global meaning of the elementary event.
•aiis then a generic ”argument” of the predicate intro-
duced by a specific functional role. For instance, the
individuals ROBOT1and BILL, etc.
Let us consider another example where Julia’s smartphone
sends an information about Julia’s location. The semantic
description of this event is given in table I by the symbolic
label of the predicative occurrence (L=”event2”). In this
case, the conceptual predicate (P=”EX I ST ”)indicates
the presence of entities specifically a human. This human
entity is represented with the role (R=”SUBJ”)and its
argument (a=”JULI A”), an instance of the HClass concept
”human being”.
TABLE I
REPRESENTATION OF THE NL STATEMENT: JULIA IS BESIDE
CASINO SUPERMARKET
event2: EXIST
SUBJ JULIA : CASINO BUILDING
date-1: 14/02/2015 12:25
date-2:
Exist:HumanPresentAutonomously
3) Knowledge automated extraction: The automated ex-
traction of knowledge requires a reasoning model for map-
ping lexical/syntactic information, supplied by sensors and
humans, into an ontological knowledge. This model allows
the matching of patterns, created from the perception of
the real world, and the antecedent (condition part) of the
corresponding conversion rule, in the HClass ontology.
a) Unification Process based on Parallel Base Ex-
ploration:The unification process consists of matching
the syntactic tree of the NL statement with the antecedent
(condition part) of a conversion rule. In this paper, a parallel
implementation of the unification process is proposed in
order to minimize the response time. This process is based
on the semantic tree matching algorithm presented in [13].
When the syntactic tree of the NL statement is unified
with the antecedent of the conversion rule, patterns are
created from lexical/syntactic information to be mapped into
conceptual knowledge.
b) Pattern matching and disambiguation:The pattern
matching consists of checking the existence of the patterns
in the HClass ontology. Terms queries usually lead to
significant ambiguity. The main challenge for this pattern
matching model is to automate the resolution of ambiguities.
In this paper, the disambiguation is based on the extraction
of semantic relations between the candidate terms. Roget
thesaurus, WordNet and Verbnet dictionaries and DBPedia
[4] are used to resolve these ambiguities.
c) Automatic commonsense knowledge extension:
To enable the matching of patterns, a large commonsense
knowledge base is required. However, a negative matching
may occur when knowledge is incomplete. In this case, the
HClass ontology requires to be updated with the discovered
knowledge.
To update the HCl ass ontology, the hypernym relation
is used to discover new commonsense knowledge. Discov-
ering commonsense knowledge consists of extracting the
generalization/specification relation of the term xin the
HClass ontology. The commonsense of this term, denoted
by HClass(x), is a sub class of the concept Cyif its hy-
pernym y, denoted HY P(x,y), is matched in HClass where
HClass(y) = Cy. Formally, Eq. 5 describes the fact that the
HClass(x)extracted from the term xis a specialization of
the concept Cyin HCl ass ontology.
HY P(x,y)∧(HClass(y) = Cy)⇒HClass(x)⊂Cy(5)
For example, the term ”mirror” is not matched in the
HClass ontology. The term ”ob ject”, the hypernym of
”mirror”, is matched with the concept ”ob ject ”. Thus, we
can automatically infer that the concept ”mirror ” is a new
specialization of the concept ”ob ject ”.
The extension of the commonsense knowledge through the
HClass ontology allows an advanced representation of the
knowledge in open world.
d) Example:Let us consider the following statement
that can be captured from an intelligent refrigerator: ”beer
is missing”. The creation of the NKRL annotation is done
through a positive semantic unification of the syntactic tree
of the NL statement with the antecedent (condition part) of
the conversion rule described in table II. The NKRL HTem p
template instantiated in this rule is Produce:Entity, which
means the creation of all sort of HClass entities including
documents, messages, news, etc. For this unification, we used
a parallel exploration based on a semantic tree matching
algorithm.
The event ”beer is missing” is described in table III by
the predicative occurrence ”event1”.
B. Reasoning Techniques for Cognitive Actors
Each actor has reasoning capabilities to communicate and
interact with other agents regarding the capabilities of each,
and reason on. The main objectives of the introduced rea-
soning techniques are to create intelligent assistive services.
1) Context-based decision making: To model a context of
an AAL application, first of all, the different elements that
affect the application and allow to infer the context have to be
identified. In this paper, the states of all actors of the system
are used for modeling the context. The user profile is one
of those contextual attributes. As the actors act in dynamic

TABLE II
NL/NKRL CONVERSION RULE
Condit ionPart(Antecedent):
(S (NN var3) (VP (S (NP (EX There)) (VP (RB not) (NN var1)))))
ConsequentPart :
PRODUCE SUBJ var2
OBJ notification message
SOURCE var4
TOPIC SPECIF(entity SPECIF(cardinality none ))
Produce:Entity (6.24)
Constraint Group :
var1=? ; var2=Gvar1 and HClass(var2) = food
HClass(var3)=human being; var4=Gvar3
HClass(var3)= electric/electronics equipment sector;
var4=SPECIF(Hvar3 detection )
TABLE III
REPRESENTATION OF THE NL STATEMENT: THE RE FR IG ER ATOR
DE TE CT S TH AT THE RE I S NO B EE R
event1: PRODUCE
SUBJ BEER: KITCHEN 1
OBJ NOTIFICATION MESSAGE 1
SOURCE SPECIF(REFREGIRATOR LG detection )
TOPIC SPECIF(entity (SPECIF cardinality none ))
date-1: 02/08/2014 12:22
date-2:
Produce:Entity
environments, a description of the world continuously up-
dated is required. The reasoning core of each actor supports
spatio-temporal reasoning about the changing locations of
humans and objects which are described using the HClass
ontology. For instance, a smart notification service can be
conceived using the context awareness paradigm and the
spatio-temporel reasoning. It allows a relevant notification
at the best moment and place.
TABLE IV
TRANSFORMATION RULE N◦1
Antecedant :
COORD (C1 C2)
C1: EXIST SUBJ : var1 : var2
var1: human being
var2: supermarket building
C2: PRODUCE SUBJ: var3 :
OBJ : notification message
TOPIC: SPECIF (entity (SPECIF cardinality none ))
var3: basic food
Consequent :
C3: MOVE SUBJ ROBOT:
OBJ notification message
BENF var1
TOPIC SPECIF( var2 (SPECIF cardinality none ))
var1: human being
var2: basic food
Move:GenericInformation
a) Example: Let us consider the transformation rule
depicted in table IV. The antecedent of this rule aims to
retrieve the information, in the knowledge base, concern-
ing the absence of a basic food and the presence of a
person near a supermarket. The event2, see table I, rep-
resents the following statement: ”Julia is beside a super-
market”. This event is matched with the first condition of
the antecedent C1 while JU LI A and CASI NO BUI LDING
are individuals respectively of the concepts ”human being”
and ”supermarket building” in the HClass ontology. In
addition, the event1 is matched with the second con-
dition of the antecedent C2 while the explicit variable
”noti f icat ion messsage” and the variable ”basic f ood ” are
matched with a predicative occurrence. The basic foods
are defined with respect to user profile. With respect to
the transformation rule, the variables of consequent are
substitutable by variables defined in the antecedent. The
consequent means that the robot notifies the missing foods to
the adequate person, who is in this case, Julia. The generated
predicative occurrence is an instance of the template Move :
GenericIn f or mat ion. This template allows to easily manage
communication and messages between actors. Thus, from
this predicative occurrence, a message content is generated
by the NL Generator module.
Figure 2 shows another case of smart notification where
”EDEN” is located near to ”LIDL supermarket” and ”milk
is missing”.
Fig. 2. The refrigerator notifies Eden that milk is missing
2) Reasoning based on collective intelligence: When a
knowledge is missing, an actor isn’t able to make any
decision. In this paper, we investigate the reasoning with
incomplete knowledge. A reasoning-based collective intelli-
gence model is proposed. Collective intelligence is defined
by Malone as a group of people and computers, connected

by the internet, collectively doing intelligent things [20].
Adopting this definition in the application context of this
study, actors’ system, humans, robots, agents, machines
and sensors are collectively reacting to give the missing
knowledge that is indispensable for decision making.
•Task selection based on capabilities:
In order to reason about which capabilities are needed
for performing an action, agents have a semantic de-
scription of their capabilities such as navigation, com-
munication or object recognition. In this study, a task
selection model is able to check which capabilities can
be acquired to achieve a task. Formally, a task selection
consists of a tuple Π=hA,K,C,Tiwhere Ais a group
of actors, Kis a set of tasks, and Cis a set of capabilities
evolving over the time T.
Selecting a particular task kfor an actor aiconsists of
determining the probability P
kaithat reflects how large
a part its capability is out of the sum of the capabilities
of all the tasks known to that actor.
Pai
k=Cai
k
∑kCai
k
The capability Cai
kof each actor encodes the capability
of the task stimulus as perceived by specific actors. This
depends on various parameters: the urgency φkof the
task k; the distance dbetween the actor and the task; and
the skill Sai
kof the actor, indicating the correspondance
of the actor aiwith respect to task k.
Cai
k=Sai
k
dφk
•Knowledge Exchange:
As each actor has some capabilities enabling only to
sense specific information, the actors need to exchange
the information captured by each one. Exchanging
knowledge allows to complete missing knowledge for
decision making and answering correctly to queries. Ex-
changing knowledge could also, significantly speed-up
commonsense knowledge extraction using a distributed
approach.
The formulation below defines a set of actors ai;(i=1,
..., n)having a required capability Caiat time Tto answer.
These actors cooperate together by exchanging information
Query.
Questa←Query(a2,Qa2)∧Query(a3,Qa3)
∧... ∧Query(an,Qan);
Ai∈Actor;i=1..n,Can∈CatT
(6)
NKRL rules define the capability of each actor and allow to
manage the interactions between actors to solve a specific
issue.
3) Causal reasoning: Different formalisms have been de-
veloped to model action preconditions and effects, and solve
both deduction and abduction problems about a multitude
of commonsense reasoning phenomena. In this paper, the
NKRL model is used to establish a linear time structure
enabling temporal reasoning to infer the intervals in which
certain events hold. In addition, the NKRL model includes a
second structure called binding occurrences, such as GOAL
and CAU SE , allowing to easily model a cause, and its
effects. In order to answer ”what is the possible cause”
question, the high-level inference based on hypothesis rules
is used. This inference mechanism is able to infer causality
relationships from a set of observed elementary events, and
provide plausible answers.
V. IM P LE M EN TATIO N A ND E VALUATION O F THE
PRO POS ED AP PROAC H
A. Cognitive assistance scenario
To validate the proposed approach and its concepts such
as context-based decision making and reasoning in open
world, a cognitive assistance scenario is proposed. In this
scenario, Bill, a frail person, is living alone in a smart home
equipped with a companion robot, called Kompai, and smart
objects that cooperate together to assist him in different
daily tasks. Julia, Bill’s sister, is responsible for monitoring
the smart house from which she receives notifications. In
this scenario, a smart refrigerator is used as actor able to
determine the status of basic foods specified in the user
profile. The refrigerator sends the status of these foods
to the adequate actor at the appropriate time and place
before asking for this information. Another important issue
addressed in this scenario concerns the capability of actors
to reason with incomplete knowledge by exchanging the
missing knowledge. For example, when the human asks
for an item such as cheese, the refrigerator has explicit
knowledge about that. The robot, which is endowed with
more reasoning capabilities than the refrigerator, can look
for and interact with the closest actor that has the ability
to provide this knowledge. The aforementioned examples
show different complexities in how the reasoning and the
interaction among actors can be ascertained.
B. General description of the implementation
The proposed approach has been implemented as a Java
service component that can be easily instantiated in a cloud
platform. Sensing, actuation and interaction with smart ob-
jects and the robot are implemented in the same software
layer by using the Ubistruct middleware. The description of
the living lab and the middleware are reported in the follow-
ing video http://youtu.be/XicBDjGSxYc. Accord-
ing to the scenario, the refrigerator is able to detect that a
basic food, such as beer, is missing through the correlation
of events concerning the absence of the RFID tag placed on
the beer bottle, within the coverage area of the RFID reader
attached on the refrigerator. A communication module is
implemented as web service to poll the RFID area coverage
and send an event of tag absence after a given timeout.
This work is reported in a multimedia video file that can be
downloaded from http://youtu.be/I361szW0LM4.
C. Evaluation and discussion
To evaluate the performance of the proposed approach,
several experiments have been conducted. We have tested

255 examples of queries including 471 messages. The pro-
cessing of each query implies firing at most 5 inference rules,
that allow to handle the basic operations concerning the man-
agement of actors’ local knowledge bases. In these tests, the
queries are in natural language under the form of a written
message or vocal message through the companion robot. The
number of accesses to the commonsense knowledge, HClass,
is proportional to the search patterns. During these tests, an
increase of the knowledge base of each actor is observed.
The execution time of each module is independent of the
size of the knowledge base due to the parallel computing
of the NKRL annotation production and reasoning. These
tests show that the proposed distributed cognitive architecture
achieves a success rate about 90 %, improving significantly
the performance obtained in our previous work where the
success rate was about 75% [14]. Some errors that occurred
during these experimentations are due to the limited sen-
sitivity of the microphone embedded into the companion
robot. In addition, implementing the communication module
as service instead of XMPP message allows a message
exchange between actors without transmission errors.
Additional experiments driven by different users have been
conducted in the living-Lab of LISSI- UPEC. 25 people
(19 single people and 6 couples) agreed to test assistive
services involving the Kompai robot for one day each. These
people have almost a similar daily agenda where they are
present in their office all day with a lunch break where
they go to buy foods from supermarket and put them in
the refrigerator. Each person has filled a form to indicate
her/his preferred food items. During the experiments, ob-
servations and questionnaires have been used as evaluation.
The evaluation consists of collecting a qualitative feedback
that provides valuable data about the user perception and
satisfaction. The obtained satisfaction rate is 70%. All users
describe the experiments as an enjoyable experience. Some
appreciations from the end users reflect the added value
of the provided service: ”Interesting services”, ”Oh, great
services, I always forget to buy coffee capsules”.
VI. CONCLUSION
In this paper, we presented a semantic approach for robots
proactively interacting with humans, agents and systems in
ubiquitous environments using natural language. Its principle
consists of advanced commonsense knowledge represention
and reasoning in open world. It exploits both the narrative
knowledge representation language (NKRL) and collective
intelligence to reason with incomplete knowledge. The sce-
nario dedicated to the cognitive assistance of frail people
showed promising results in terms of pertinence of the
provided service. The ongoing works address the extention
of the proposed approach for spatial modeling and reasoning
with the aim to endow the system actors with additionnal
capabilities to improve their perception of the context and
allow a better decision making.
REFERENCES
[1] A. Chibani, A. Bikakis, T. Patkos, Y. Amirat, S. Bouznad, N. Ayari, L.
Sabri, ”Using Cognitive Ubiquitous Robots for Assisting Dependent
People in Smart Spaces”, in Intelligent Assistive Robots- Recent
advances in assistive robotics for everyday activities, S. Mohammed,
J.C. Moreno, K. Kong and Y. Amirat (Eds), Springer Tracts on
Advanced Robotics (STAR) series, 2015, pp. 297-316.
[2] A. Saffiotti, M. Broxvall, M. Gritti, K. LeBlanc, R. Lundh, J. Rashid,
B.S. Seo and Y.J. Cho, ”The PEIS-Ecology Project: vision and results”,
In Proc. of the IEEE/RSJ Intern. Conf. on Intelligent Robots and
Systems (IROS 2008), France, 2008, pp. 2329-2335.
[3] A. Tapus, M. J. Mataric, B. Scassellati, ”The grand challenges in
socially assistive robotics”, IEEE Robotics and Automation Magazine,
2007, vol. 14, no. 1, pp. 35-42.
[4] C. Bizer, L. Jens, G. Kobilarov, S. Auer, C. Becker, R. Cyganiak,
and S. Hellmann, ”Dbpedia - a crystallization point for the web of
data”, Web Semantics: Science, Services and Agents on the World
Wide Web, 2009, vol. 7, no. 3, pp. 154-165.
[5] C. Galindo and A. Saffiotti, ”Inferring robot goals from violations
of semantic knowledge”, Robotics and Autonomous Systems Journal,
2013, vol. 61, no. 10, pp. 1131-1143.
[6] G. H. Lim, I. H. Suh, H. Suh, ”Ontology-Based Unified Robot Knowl-
edge for Service Robots in Indoor Environments”, IEEE Transactions
on Systems, Man and Cybernetics, Part A: Systems and Humans, 2011,
vol. 41, no.3, pp.492-509.
[7] G.P Zarri, ”Representation and Management of Narrative Informa-
tion”, Theoretical Principles and Implementation, 2009, Springer.
[8] J. Rashid, ”Extending a Networked Robot System to Include Humans,
Tiny Devices, and Everyday Objects”, PhD Thesis, Orebro University,
Sweden, 2011.
[9] K. Dautenhahn, ”Socially intelligent robots: dimensions of human-
robot interaction”, Philosophical Trans. of the Royal Society of
London- Series B: Biological Sciences, 2007, vol. 362, no. 1480, pp.
679-704.
[10] K. Talamadupula, G briggs, T. Chakraborti, M. Scheutz, S. Kambham-
pati, ”Coordination in human-robot teams using mental modeling and
plan recognition”, In Proc. of the IEEE/RSJ Intern. Conf. on Intelligent
Robots and Systems (IROS 2014), 2014, pp. 2957-2962.
[11] M. Beetz, L. Mosenlechner, M. Tenorth, ”CRAM A Cognitive Robot
Abstract Machine for Everyday Manipulation in Human Environ-
ments”, In Proc. of the IEEE/RSJ Intern. Conf. on Intelligent Robots
and Systems (IROS 2010), 2010, pp. 1012-1017.
[12] M. Tenorth, A. C. Perzylo, R. Lafrenz, & M. Beetz, ”The roboearth
language: Representing and exchanging knowledge about actions,
objects, and environments”, In IEEE Intern. Conf. on Robotics and
Automation (ICRA 2012), 2012, pp. 1284-1289.
[13] N. Ayari, A. Chibani, Y. Amirat, ”Semantic Management of Human-
Robot Interaction In Ambient Intelligence using N-ary ontologies”, in
Proc. Of the IEEE Intern. Conf. on Robotic and Automation, ICRA
2013, Karlsruhe, Germany, May 6-10, 2013, pp. 1164-1171.
[14] N. Ayari, A. Chibani, Y. Amirat, and E. Matson, ”A Se-
mantic Approach for Enhancing Assistive Services in ubiq-
uitous robotics”, Robotics and Autonomous Systems, Elsevier,
DOI:10.1016/j.robot.2014.10.022, 2015.
[15] S. Hodges, N. Villar, J. Scott, A. Schmidt, ”A new era for ubicomp
development”, IEEE Pervasive Computing Magazine, vol. 11, no. 1,
pp. 5-9, 2012.
[16] S. Joshi, P. Schermerhorn, R. Khardon and M. Scheutz, ”Abstract
planning for reactive robots”, In Proc. of the IEEE Intern. Conf. on
Robotics and Automation (ICRA 2012), 2012, pp. 4379-4384.
[17] S. Lemaignan, R. Ros, L. Mosenlechner, R. Alami, & M. Beetz,
ORO: a knowledge management platform for cognitive architectures
in robotics, In Proc. of IEEE/RSJ Intern. Conf. on Intelligent Robots
and Systems (IROS 2011), pp. 3548-3553, 2011.
[18] S. Coradeschi, A. Cesta, G. Cortellessa, L. Coraci, C. Galindo, J.
Gonzalez, L. Karlsson, A. Forsberg, S. Frennert, F. Furfari, A. Loutfi,
A. Orlandini, F. Palumbo, F. Pecora, S. von Rump, A. Stimec, J. Ull-
berg, and B. Otslund, ”GiraffPlus: A System for Monitoring Activities
and Physiological Parameters and Promoting Social Interaction for
Elderly”, In Human-Computer Systems Interaction: Backgrounds and
Applications 3, Advances in Intelligent Systems and Computing, vol.
300, pp 261-271, Springer International Publishing, 2014.
[19] V. Raskin, J. Taylor, E. T. Matson, ”Towards an Ontological Modeling
of Something Very Much Like Consciousness: The HARMS Way”,
In Proc. of the Society for Design and Process Science (SDPS)
Conference, Berlin, Germany, 2012.
[20] T. W. Malone, ”MIT conference focuses on ’collective intelligence”,
Boston Herald, MA: MIT Press, 2012.