Fig 9 - uploaded by Michael Spranger
Content may be subject to copyright.
3 Measuring estimation errors. The block was put at 500, 1000 and 1500 mm distance from the robot. Each time the vision system estimates the features (width, height, x and y) of the block. The graph to the right shows the root-mean-square-error (RMS) for each measurement. The x feature ( x -axes runs towards the front) is most heavily affected by increasing distance.
Source publication
Grounding language in sensorimotor spaces is an important and difficult task. In order, for robots to be able to interpret and produce utterances about the real world, they have to link symbolic information to continuous perceptual spaces. This requires dealing with inherent vagueness, noise and differences in perspective in the perception of the r...
Similar publications
Mathematics is often conducted with a writing implement. But is there a relationship between numerical processing and sensorimotor ‘pen’ control? We asked participants to move a stylus so it crossed an unmarked line at a location specified by a symbolic number (1–9), where number colour indicated whether the line ran left–right (‘normal’) or vice v...
Citations
... The underlying conceptual model and usage of Prototype Theory is not a new proposal for spatial language and follows (Eyre & Lawry, 2014;Gärdenfors, 2004;Mast et al., 2016;Spranger & Pauw, 2012). Of particular interest is the work of Mast et al. (2016) where a pragmatic model is developed to tackle problems involving referring expressions. ...
In this paper we analyse the issue of reference using spatial language and examine how the polysemy exhibited by spatial prepositions can be incorporated into semantic models for situated dialogue. After providing a brief overview of polysemy in spatial language and a review of related work, we describe an experimental study we used to collect data on a set of relevant spatial prepositions. We then establish a semantic model in which to integrate polysemy (the Baseline Prototype Model), which we test against a Simple Relation Model and a Perceptron Model. To incorporate polysemy into the baseline model we introduce two methods of identifying polysemes in grounded settings. The first is based on ‘ideal meanings’ and a modification of the ‘principled polysemy’ framework and the second is based on ‘object-specific features’. In order to compare polysemes and aid typicality judgements we then introduce a notion of ‘polyseme hierarchy’. Finally, we test the performance of the polysemy models against the Baseline Prototype Model and Perceptron Model and discuss the improvements shown by the polysemy models.
... Similarly, Degen et al. [66] show that various phenomena can be explained in the RSA model when a continuous rather than binary semantics is used. Also, in the context of modelling spatial language for referring expressions, Spranger and Pauw [67] argue for a 'lenient' semantics which considers similarity of entities to concepts rather than strict concept membership. ...
... In general, it appears that these types of disagreements are not strong enough to severely limit communication and semantic models can perform well even when making inevitable generalisations about peoples' standpoints, for example see the performance of the Refined Model in Section 5.6.2. Moreover, there are pragmatic strategies which may help to overcome these disagreements [65,67]. ...
Spatial prepositions in the English language can be used to denote a vast array of configurations which may greatly diverge from any canonical meaning and this semantic variability poses challenges for many systems where commands or queries are given in natural language. There have been many accounts from Linguistics and Cognitive Science highlighting the various phenomena which contribute to this semantic variability --- primarily, spatial prepositions appear to encode functional as well as spatial information and to also exhibit polysemy. Both these issues represent significant challenges for grounded natural language systems and have not yet been accounted for in semantic models of spatial language. To begin exploring the semantic variability of spatial prepositions, I will compare various cognitive accounts which incorporate the functional notions of support and location control and I will provide methods for constructing a semantic model based on Prototype Theory. In order to incorporate polysemy into this model, I will contribute methods for identifying polysemes based on Herskovits' notion of `ideal meanings' as well as a modification of the `principled polysemy' framework of Tyler and Evans. I will also introduce a notion of `polyseme hierarchy' which will allow these polysemes to be incorporated in the semantic model. By including functional relationships as well as polysemy into a semantic model we are able to provide a measure of typicality which is useful in interpreting referring expressions. However, this model does not yet account for `object-specific features' which seem to influence categorisation decisions involving spatial prepositions. In the final chapter I will provide insight into the nature of categorisation and typicality for spatial prepositions and highlight the importance of object-specific features. Though a concrete solution to including these features will not be provided, I will provide suggestions of how these features may be included in our semantic model.
... Given a preposition and a configuration (figure-ground pair) in a scene, the model assigns a value of typicality for how well the configuration fits the preposition. Following (Eyre and Lawry 2014, Gärdenfors 2004, Mast, Falomir, and Wolter 2016, Spranger and Pauw 2012, typicality in our model is calculated by considering the semantic distance to a prototype. ...
In previous work exploring how to automatically generate typicality measures for spatial prepositions in grounded settings, we considered a semantic model based on Prototype Theory and introduced a method for learning its parameters from data. However, though there is much to suggest that spatial prepositions exhibit polysemy, each term was treated as exhibiting a single sense. The ability for terms to represent distinct but related meanings is unexplored in the work on grounded semantics and referring expressions, where even homonymy is rarely considered. In this paper we address this problem by analysing the issue of reference using spatial language and examining how the polysemy exhibited by spatial prepositions can be incorporated into semantic models for situated dialogue. We support our approach on theoretical developments of Prototype Theory, which suggest that polysemy may be analysed in terms of radial categories, characterised by having several prototypicality centres. After providing a brief overview of polysemy in spatial language and a review of the related work, we define the Baseline Model and discuss how polysemy may be incorporated to improve it. We introduce a method of identifying polysemes based on `ideal meanings' and a modification of the `principled polysemy' framework. In order to compare polysemes and aid typicality judgements we then introduce a notion of `polyseme hierarchy'. Subsequently, we test the performance of the extended Polysemy Model by comparing it to the Baseline Model as well as a data-driven model of polysemy which we derive with a clustering algorithm. We conclude that our method for incorporating polysemy into the Baseline Model provides significant improvement. Finally, we analyse the properties and behaviour of the generated Polysemy Model, providing some insight into the improvement in performance, as well as justification for the given methods.
... Of particular interest is the work of Mast et al. [22] where a pragmatic model is developed to tackle problems involving referring expressions. In order to model the semantics of the terms involved, following [7,14,31], they use the notion of semantic distance in a feature space, where graded category membership can be determined by calculating the semantic distance to a prototype in the space. Mast et al. also focus on projective prepositions (in particular, 'left of', 'right of', 'in front of' and 'behind') and as a result, the challenge of assigning parameters to the model is simpler and appears to be achieved via the researchers' intuition. ...
... The models that are trained on the data rely on a notion of semantic distance and, following [7,14,22,31], typicality in our proposed model is calculated by considering the semantic distance to a prototype. ...
In cognitive accounts of concept learning and representation three modelling approaches provide methods for assessing typicality: rule-based, prototype and exemplar models. The prototype and exemplar models both rely on calculating a weighted semantic distance to some central instance or instances. However, it is not often discussed how the central instance(s) or weights should be determined in practice. In this paper we explore how to automatically generate prototypes and typicality measures of concepts from data, introducing a prototype model and discussing and testing against various cognitive models. Following a previous pilot study, we build on the data collection methodology and have conducted a new experiment which provides a case study of spatial language for the current proposal. After providing a brief overview of cognitive accounts and computational models of spatial language, we introduce our data collection environment and study. Following this, we then introduce various models of typicality as well as our prototype model, before comparing them using the collected data and discussing the results. We conclude that our model provides significant improvement over the other given models and also discuss the improvements given by a novel inclusion of functional features in our model.
... There is also empirical evidence 7 which confirms that object colour perception changes with change in illumination and background for people with normal colour vision. a Spranger and Pauw 8 show that overlapping, graded colour categories can improve the chances of communicative success under circumstances of perceptual and conceptual deviation. Furthermore, it is well-known that the concept of fuzzy set provides a methodology a Note that the contextual sensitivity to brightness perception that we refer to affects people with normal colour vision. ...
In human-machine communication situations, perceptual and conceptual deviations can appear. The challenge of categorising colours is tackled in this paper. Colour perception is very subjective. Colours may be perceived differently depending on a person’s eye anatomy and a person’s sense of sight which adapts to the surroundings and perceives different brightness of hues depending on the context. Distinguishing more/less quantity of hues depends also on the level of expertise but also on the cultural and social environment. Colours naming involves conceptual alignment with human cognition, meaning and human understanding for referring to an object and even for discriminating among objects. Studies in cross-cultural linguistics say that humans determined prototypical colours as the centre of colour categories. Hence, a cognitive colour model should distinguish/indicate when a colour coordinate is close/far to the centre of its category. And these centres of categories should be adaptable and customisable depending on the society.
A fuzzy colour model based on HSL colour space and radial basis functions is presented in this paper. Logics have been defined to combine this fuzzy-colour model with a Probabilistic Reference And GRounding mechanism (PRAGR) in order to obtain the most discriminative colour descriptor for an object depending on the context. Two case studies related with human cognition are presented. Then further tests are carried out on a dataset where the first and second most discriminative colour is computed for each object in each scene. Finally, a survey is conducted to find out the cognitive adequacy of the obtained discriminative colour names.
... For example, the problem of producing referring expressions when it is not certain that the other participant shares the same perception and understanding of the scene has been addressed by [15] and [35]. Another example of research investigating language misunderstanding based on perceptual errors is [43] which examines the effect of perceptual deviation on spatial language. However, [43] deals with robot-robot dialogues and the evolution of spatial term semantics in robot populations. ...
... Another example of research investigating language misunderstanding based on perceptual errors is [43] which examines the effect of perceptual deviation on spatial language. However, [43] deals with robot-robot dialogues and the evolution of spatial term semantics in robot populations. ...
... Given this array of factors it is not surprising that there is a spectrum of approaches to creating computational models of spatial language semantics, each motived by different considerations. For example, integrating world-knowledge [32] and/or linguistic ontological knowledge [3]; integrating spatial semantics into a compositional/attentional accounts of reference [23,24,31]; learning spatial semantics directly from sensor data using machine learning techniques [12,34]; modelling the functional aspects of spatial semantics in terms of predicting the dynamics of objects in the scene [10,42]; capturing the vagueness and gradation of spatial semantics [17,22,43]; and leveraging analogical reasoning mechanisms to enable agents to apply spatial semantics to new environments [13]. ...
Errors in visual perception may cause problems in situated dialogues. We investigated this problem through an experiment in which human participants interacted through a natural language dialogue interface with a simulated robot.We introduced errors into the robot’s perception, and observed the resulting problems in the dialogues and their resolutions.We then introduced different methods for the user to request information about the robot’s understanding of the environment. We quantify the impact of perception errors on the dialogues, and investigate resolution attempts by users at a structural level and at the level of referring expressions.
... The learner learns from the observation of the same object but from perspective of the other robot (matrix 2). GRO2 is used to evaluate what happens if there is perceptual deviation [21]. That is tutor and learner see the scene from different viewpoints and therefore have different feature estimations for objects. ...
This paper discusses lexicon word learning in high-dimensional meaning spaces from the viewpoint of referential uncertainty. We investigate various state-of-the-art Machine Learning algorithms and discuss the impact of scaling, representation and meaning space structure. We demonstrate that current Machine Learning techniques successfully deal with high-dimensional meaning spaces. In particular, we show that exponentially increasing dimensions linearly impact learner performance and that referential uncertainty from word sensitivity has no impact.
... This process can fail, for instance, when a particular spatial scene does not fit the program. More precisely, evaluation can succeed, or fail, but all successful evaluations are also scored [21], as to how much the program fits the scene. Let us assume the hearer wants to interpret the example phrase and has decoded the IRL-program in Figure 5. Evaluation proceeds as follows. ...
This paper presents a computational model of the processing of dynamic spatial relations occurring in an embodied robotic interaction setup. A complete system is introduced that allows autonomous robots to produce and interpret dynamic spatial phrases (in English) given an environment of moving objects. The model unites two separate research strands: computational cognitive semantics and on commonsense spatial representation and reasoning. The model for the first time demonstrates an integration of these different strands.
... Most importantly, each robot sees the world from his perspective. This means that robots always deal with issues of perceptual deviation [Spranger and Pauw, 2012]. Robots have different viewpoints on the scene, which impacts on issues of egocentric spatial language. ...
We present a system for generating and understanding of dynamic and static spatial relations in robotic interaction setups. Robots describe an environment of moving blocks using English phrases that include spatial relations such as "across" and "in front of". We evaluate the system in robot-robot interactions and show that the system can robustly deal with visual perception errors, language omissions and ungrammatical utterances.
... Unlike the Global Decision Approach, which attempts to learn the parameters for the entire decision process in an integrated fashion, the Modular Decision Approach models acceptability of features independently of the referential situation, using a combinatory scoring mechanism for taking into account context and referential goals. This approach is taken by Spranger and Pauw (2012) and Meo, McMahan, and Stone (2014). Kelleher (2011) uses this approach for RR, although he uses the Independent Decision Approach for REG. ...
... The core concept used for this end is some variant of discriminatory power (DP), a measure of how likely an expression as a whole is to distinguish the target object from the distractor object. Spranger and Pauw (2012) calculate acceptability for individual properties based on the distance to prototypes, an approach shared with PRAGR. Multiplicative scores are calculated for complex expressions, aiming for the expression that maximises the distance of the target to the closest distractor. ...
... For these reasons, with PRAGR we follow the Modular Decision Approach, providing individual feature models and a probabilistic integrated mechanism to calculate appropriateness of descriptions based on the acceptability of individual properties. Spranger and Pauw (2012) provide successful evaluation of robot-robot communication with spatial relations using this approach, while Meo et al. (2014) demonstrate the ability of their approach to mimic human decisions for discriminatory colour naming. In an exemplary evaluation, Kelleher (2011) demonstrates the ability of his system to successfully handle one-anaphora. ...
In this paper, we present a system for effective referential human-robot communication in the face of perceptual deviation using the Probabilistic Reference And GRounding mechanism PRAGR and vague feature models based on prototypes. PRAGR can handle descriptions of arbitrary complexity including spatial relations and uses flexible concept assignment in generation and resolution of referring expressions for bridging conceptual gaps in referential robot-robot or human-robot interaction. We evaluate the benefit of using vague as compared to crisp properties regarding referential success and robustness towards perspective alignment error in referential robot-robot and human-robot communication.
