Content uploaded by Laura Malinverni
Author content
All content in this area was uploaded by Laura Malinverni on Mar 21, 2016
Content may be subject to copyright.
1
Towards Methods for Evaluating and Communicating Participatory Design: a Multimodal
Approach
Laura Malinverni, Joan Mora-Guiard, Narcis Pares
Universitat Pompeu Fabra,
C/Roc Boronat 138, 08018 Barcelona, Spain
/ laura.malinverni@upf.edu / joan.mora@upf.edu / narcis.pares@upf.edu
Abstract
Participatory Design (PD) has been proposed as a useful strategy to address pitfalls in the design of serious games
for children with special needs. Nonetheless, methodological weaknesses in the analysis of the results of PD
workshops may hinder its effectiveness in providing useful and robust design contributions and facilitating
communication in multidisciplinary teams. To address this issue we propose the use of multimodal analysis to
evaluate participants’ contributions during PD workshops. We present an analysis that was applied in an informant
workshop with autistic children, aimed at refining the design of a serious game based on Full-Body Interaction.
Results show that multimodal analysis constitutes an effective and coherent method to capture and analyze users’
contributions across a wide range of semiotic resources, thus extending the richness of insights that can be derived
from a PD workshop and communicated to the rest of the team. Furthermore, the analysis allowed the identification
of fundamental design questions, thus offering a robust empirical ground for supporting dialogue and reflection
between multiple stakeholders.
Keywords: Participatory Design, Multimodal Analysis, Full-Body Interaction, Children, Autism, Interaction Design,
Research Methods
2
1. Introduction
Autism is a neurodevelopmental disorder characterized by impairments in social and communication
abilities and by restrictive interests and sensory abnormalities (DSM –V). Several therapeutic treatments have been
proposed to improve the quality of life of individuals with Autistic Spectrum Disorder (ASD) by facilitating learning
of everyday skills (Fuentes-biggi et al., 2006). However, the effectiveness of these treatments is mainly associated
with their intensity in terms of weekly hours (Boyd et al., 2014). The time-consuming nature of these treatments led
mental health professionals to start exploring the use of serious games to complement traditional treatment methods
(Goh et al., 2008). However, gaming interventions showed pitfalls related to difficulties in reaching the therapeutic
goals and shortcomings in designing experiences that can be truly engaging and motivating for children with ASD
(Goh et al., 2008).
A possible reason behind these pitfalls can be found in the shortage of inclusive design approaches which
are capable of integrating the knowledge of mental health experts with the interests of children and the experience of
designers. To address this issue several researchers proposed the use of Participatory Design (PD) methods to
properly grasp children’s interests and improve user experience (Benton et al., 2014; Frauenberger et al., , 2011;
Millen et al., 2010; Piper et al., 2006).
However, PD can represent a demanding experience for children with ASD (Frauenberger et al., 2013).
Difficulties in communication may hinder the opportunities for properly capturing and identifying users’
contributions, thus reducing the quality of insights that researchers may obtain from a PD workshop. Nonetheless,
children with special needs are often the population that benefits the most from design processes that include them
(Frauenberger et al., 2011). Therefore, proper methods to include them in the PD of serious games are needed.
To address this issue we propose combining PD techniques with the use of an analytical framework aimed
at offering instruments to deepen into the different dimensions of users’ contributions that can be derived from a PD
workshop. Specifically, we propose the use of a framework derived from multimodal analysis (Jewitt, 2013; Kress,
2010) to collect and interpret users’ contributions during PD and hence be able to better transfer them to the rest of
the team composed of experts and designers.
Multimodal analysis, by providing concepts and methods which collect and analyze a wide range of
resources employed by people to construct meaning (i.e. gestures, gaze, movements, etc.), extends the analysis
beyond the intentional communicational prompts, thus overcoming the barriers of communication and expression.
3
We suggest that multimodal analysis does not only respond to the specific challenges of PD with ASD
children, but also represents a useful contribution to research related to defining robust and coherent evaluation
methods for PD (Frauenberger et al., 2014). Until now, most research in PD has focused on proposing techniques for
eliciting users contributions (Druin, 2002; Walsh et al., 2010), while robust approaches for the analysis of PD are
scarce. This methodological weakness may generate biases and unbalanced power structures that may be particularly
delicate in the complex network of human relations present in the multidisciplinary teams required to design serious
games.
Starting from this perspective, a case study will be presented which shows the application of multimodal
analysis in a PD workshop with ASD children. The workshop was aimed at refining the design of a serious game
based on Full-Body Interaction. Through the description of the proposed approach we will then argue the potential
of multimodal analysis as an evaluation tool for PD and as an instrument to guide a design process.
In particular, our study suggests the effectiveness of this approach not only for analyzing participants’
contributions and emergent behaviors from a broad and rigorous perspective, but also as an instrument to guide
design refinements and to facilitate communication within the design team and with experts. Within that, we will
place special emphasis on discussing its suitability for eliciting crucial design questions and for spotting unsolved
design aspects, which are not transparent in other types of analyses.
Thus, we suggest that a multimodal analytical approach, by raising design questions based on empirical
data, can offer a fertile ground to support dialogue and reflection between multiple stakeholders (e.g. mental health
professionals, designers). In particular, this method provides insights into guiding the discussion on issues related to
interaction design, content definition, aesthetics and alignment between educational requirements and design
choices.
In the following sections we will introduce the theoretical framework of the research and report the
procedure and results related to the application of multimodal analysis to PD. Finally, the benefits of the proposed
approach for improving iterative design processes will be considered and guidelines will be drawn for future
research.
2. Theoretical framework
2.1 Participatory Design for Serious Games for ASD children
4
In the last decades a growing attention has been posed on the potential of using serious games to
complement traditional treatments for children with ASD (Ben-Sasson et al., 2013; Whalen et al., 2010; Yan, 2011).
This approach is based on the interest that children show in digital games and the capacity of games to foster
motivation, engagement (Prensky, 2003; Resnick, 2002), and produce behavioral changes (Deterding et al., 2011).
Moreover, game-based interventions with children within the Autism Spectrum Disorder (ASD) have been proven to
accelerate learning processes (Charlton et al., 2005) by increasing the willingness to complete the required tasks
(Hoque et al., 2009). These findings have lead to a widespread proliferation of serious games for children with ASD.
Nonetheless, gaming interventions showed pitfalls related to difficulties in reaching the therapeutic goals and to
shortcomings in designing an experience that can be truly engaging for children (Goh et al., 2008).
This shortcoming is due in part to the challenges related to properly combining therapeutic methods,
effective gameplay and children’s interests. Often, serious games oversimplify game design principles by using
games elements as rewards according to a behaviorist gamification approach. As a consequence, these games may
have a reduced appeal for children, who are used to much more sophisticated commercial games. This risk is
exacerbated by the lack of efforts to properly include ASD children in the design process. Several serious games
have been developed using design methods that are mainly designer-driven, in which little or no space is provided
for childrens contributions and interests (Bertacchini et al., 2013; Hansen et al., 2013; Yan, 2011).
This approach is a product of what Rogers and Marsden (2013) define as “the rhetoric of compassion”
which may occur when designing technology for people with special needs. As the authors point out, often research
in HCI attempts to help people with special needs through solutions that are based on our understanding of what
they need, instead of giving them the voice to express their views. This tendency reflects a power dynamic where
clinicians and designers are considered to have “enough knowledge,” so it is considered unnecessary to listen to
children express their own motivations.
To address this issue several researchers proposed the use of Participatory Design (PD) to properly grasp
children’s interests and improve user experience (Frauenberger et al., 2011; Malinverni et al., 2014; Millen et al.,
2010; Piper et al., 2006). In these projects, a number of PD techniques have been used, such as getting feedback
from children on design choices (Frauenberger et al., 2012; Millen et al., 2010), analyzing their preferences
(Frauenberger et al., 2011), and creating scenarios (Millen et al., 2010).
5
Nonetheless, carrying out PD with ASD children is often a challenging and demanding task. PD workshops
are often based on open-ended and loosely structured formats that require flexibility and social interactions, which
may turn into barriers for children with ASD (Frauenberger et al., 2013). Moreover, difficulties in communication
may reduce the ability of children to properly express their needs and preferences. Due to these difficulties, in
several cases children’s contributions are often limited to accessory aspects (e.g. asking if they prefer the visual
aspect of the option A or B) or only influence aesthetic features to embellish an existing design. This tendency
reflects the practical challenge of involving children with ASD in PD activities, but also implies an ethical reflection
on the amount of space that is provided for their contributions.
This latter aspect is particularly sensitive in a multidisciplinary context such as technology for autism,
where multiple perspectives need to be merged. As Scaife and Rogers (1999) point out, it is often difficult to
combine the perspectives of children and experts since some proposals can be unworkable in computational terms or
the preferences of children could actually come into conflict with the therapeutic goals. It becomes therefore
necessary to identify methods capable of orchestrating different contributions (experts, designers, and children)
within a coherent holistic design.
However, to our knowledge, none of the projects aimed at including ASD children in a design process,
proposes a clear approach to properly collect, select and integrate children’s voices in the design of the final product.
It is therefore necessary to define methodological instruments that can adequately and robustly facilitate the
gathering, analysis and interpretation of users’ contributions. This research also implies an epistemological reflection
on the implications and assumptions underlying different methodological approaches. As Bachelard (2002) states,
knowledge always acts as a light that casts shadows. As a consequence, it is necessary to pay a close attention on the
employed research methods, as they shape the way in which information is constructed, collected and interpreted
(Harding, 1989).
2.2 Issues in the analysis of Participatory Design
The importance of Participatory Design is increasingly acknowledged in the HCI community. According to
ACM digital library database, there has been a 580% growth, from 2003 to 2013, in the number of papers published
on PD. Given this impressive growth, it becomes evident the need for defining more rigorous methods for the
analysis of PD (Frauenberger et al., 2014).
6
Participatory Design originates in Scandinavian countries, where from the 1970s a new design approach
was developed to meet requirements related to the pragmatic necessity of understanding users’ needs and to the
political claim for horizontality and participation (Muller, 2003). Thus, at a political and philosophical level, the
roots of PD can be traced in postmodern tradition, phenomenology, Marxism and feminist epistemologies
(Frauenberger et al., 2014; Harrison et al., 2011).
This theoretical background indicates that, when we define methods for the evaluation of PD, we cannot
employ approaches based on positivistic stances but instead we need to consider methods that allow investigating
meaning and interaction from a situated perspective (Hoepfl, 1997). This claim is supported both by the need for
coherence within the epistemological roots of PD and by the practical evidence that PD does not typically allow for
quantification and generalization (Frauenberger et al., 2014).
To tackle this issue, different evaluation approaches have been proposed, addressing different units of
analysis such as the evaluation of the overall process, the assessment of the proposed elicitation activities, the
analysis of users contributions, etc. Frauenberger et al. (2014) propose a holistic approach to evaluate the overall
PD. They define a “tool-to-think-with”, composed by four lenses (epistemology, values, stakeholders, outcomes)
aimed at critically analyzing the global coherence of the different aspects of PD. On the other hand, Mazzone et al.
(2012) provide two dimensions (suitability and capability) to evaluate whether the elicitation activities are engaging
and effective. Finally, both Moser et al. (2014) and Schaper et al. (2014), focus on the analysis of users
contributions by performing content analysis of the games created by children. These frameworks provide valid and
useful contributions to understand and analyze PD. However, additional research is needed to further appropriate
research methods. Specifically, we identified major shortcomings in the availability of robust and coherent
methodological tools for collecting and analyzing stakeholders’ contributions. This lack can affect research in three
main aspects: 1) the reliability of PD outcomes, 2) the dynamics of multidisciplinary teams involved in PD and 3)
the richness of data available to inform design.
As Scaife & Rogers (1999) suggested, the lack of robust methodological tools may produce confirmation
biases when researchers collect and select the contributions that will inform the design, such as the tendency to
prioritize contributions that confirm their initial ideas. As a consequence, these biases may hinder communication
and decision-making in a multidisciplinary team, due to the lack of robust evidences, upon which design decisions
can be grounded and justified.
7
On the other hand, even in the few cases where users’ contributions are robustly analyzed, the data
collection tends to focus only on intentional communicational prompts (i.e. written and verbal reports, drawings)
thus excluding other resources employed for meaning construction (e.g. gaze, movement, actions, etc.). This neglect
could be particularly problematic in PD activities, as they involve a wide variety of employed media and resources.
Participants may draw, build low-tech prototypes, discuss or enact specific actions (Druin, 2002; Giaccardi et al.,
2012; Hummels et al., 2006; Walsh et al., 2010) thus distributing meaning across a wide range of possible resources.
The need for attending to a broader range of resources is even more relevant when designing technologies
based on novel interaction paradigms or when PD addresses children or populations with special needs. Novel
interaction paradigms (e.g. tangible interfaces, full-body interaction, wearable, etc.), by involving an explicit use of
the body and the space, need to meaningfully incorporate physicality and spatiality in the very design and evaluation
process (Malinverni and Parés, 2014). Furthermore, when working with children or users with special needs, the
eventual difficulties in verbal articulation can drastically reduce the amount of contributions taken into account. This
claim is supported by studies that show that paying attention to children’s embodied forms of expression (e.g.
gestures) allows capturing ideas that are not verbally reported (Church, 1999; Goldin-Meadow, 2011).
To sum up, three main needs have been identified for the analysis of PD. Firstly, the necessity of
employing methods that coherently fit with its philosophical roots. Secondly, the need to define robust evaluation
methods to adequately analyze and interpret users’ contributions. Thirdly, the importance of analytical frameworks
capable of going beyond the limitation of language-based approaches and suitable for interpreting contributions
produced across a wide range of different modes (e.g. drawing, verbal reports, prototypes, actions, etc.). To address
these needs we propose the use of multimodal analysis to analyze and interpret users’ contributions in PD.
2.3 Multimodal analysis
Multimodality is an interdisciplinary approach, derived from socio-semiotics and aimed at analyzing
communication and situated interaction from a perspective that encompasses the different resources that people use
to construct meaning (Jewitt, 2013). At a theoretical level multimodality is grounded on the key concept of mode,
which constitutes a set of socially and culturally shaped resources for making meaning, for instance the ensemble of
writing and images on a page (Jewitt, 2013). According to this framework, each mode has a set of modal
affordances, which refers to “what is possible to express, represent or communicate easily with the resources of a
8
mode and what is less straightforward or even impossible” (Kress, 2010). In this context, thus, meaning is
considered to be realized in the iterative connection between the affordances of a material artifact, the affordances of
the socio-cultural environment and the resources, intentions and knowledge that people bring to that encounter
(Jewitt, 2013).
Therefore, if we consider the relational nature of meaning construction, this approach coherently fits with
the need for situated and post-positivistic approaches in the analysis of PD. Furthermore, if we also consider the
interplay between embodied and material resources, this approach allows us to compensate for the weaknesses
found in other analytical approaches. Specifically, at a methodological level, multimodal analysis provides
“concepts, methods and a framework for the collection and analysis of visual, aural, embodied and spatial aspects of
interaction and environments” (Jewitt, 2013). The wide range of included units of analysis allows a fine-grained
understanding of situated interactions and permits to overcome the limitations of frameworks that focus only on
intentional communication prompts such as written or spoken language (Crescenzi et al., 2014; Jewitt, 2009, 2013).
This approach has been applied in a wide range of contexts, including the analysis of ASD children’s gaze
in naturalistic settings (Korkiakangas & Rae, 2014), workplace interactions (Hindmarsh and Pilnick, 2002), learning
contexts (Mondada, 2011), playgrounds (Goodwin, 2000), tourist spaces (Jaworski and Thurlow, 2009) and
classroom interactions (Bezemer, 2008). Through utilizing contextual, embodied and visual resources, these
analyses have shown the effectiveness of this methodology to properly grasp meaning in its situated and multimodal
nature.
Hence, an increasing number of studies are employing this method for the analysis of user interaction with
digital technologies (Crescenzi et al., 2014; Price and Jewitt, 2013). This approach is shown to be particularly
suitable for the analysis of highly multimodal experiences, such as digital environments based on embodied
interaction, where different semiotic resources are employed to construct meaning. For instance, Crescenzi et al.
(2014) employ multimodal analysis to analyze losses and gains in children’s drawing by comparing finger painting
with painting on an iPad. Price and Jewitt (2013) also use multimodal analysis to examine communication in
children interacting with tangible interfaces to understand real time cognition and action in meaning making.
Relevant opportunities can be found in extending this approach to the analysis of PD. A preliminary study
in this area has been proposed by Derboven et al (2015), which employed multimodal analysis to identify values in
PD with children. Starting from this perspective, we propose to broaden this research field, hypothesizing that the
9
use of multimodal analysis in PD can provide a fine-grained understanding of participants’ contributions across
different modes and can meaningfully be embedded in an iterative design process by offering insights to guide
design refinements.
3. Applying multimodal analysis to Participatory Design
In the present study we describe a PD workshop based on the “children as informants” model (i.e. testing
raw prototypes with children in order to get inputs during the design process (Nesset & Large, 2004)). The
workshop was aimed at evaluating and improving a serious game based on Full-Body Interaction for ASD children.
Specifically, the goals were, on the one hand, to analyze how children interact with and interpret the initial design
proposal and, on the other, to include children’s interests and contributions in the design of the experience. In the
following sections we will describe the analyzed environment, the structure of the workshop and the framework
used for its analysis.
3.1 The Full-Body Interaction Environment
The design of the system was derived from the requirements established by the psychologists of the team to
achieve an experience capable of fostering exploration and social interaction in ASD children. Since restrictive
interests and impairments in social skills represent one of the target goals of therapeutic treatments in Autism (DSM-
V) they hypothesized that exploration and discovery could lead to surprise and a need to communicate such surprise.
On the other hand, exploration could also entail the need to search for help and/or collaboration in order to discover
new situations. These hypotheses were also supported by informal observation of children with ASD interacting in
other full-body interactive experiences in the past. The behaviors mentioned above were seen to emerge in these
previous experiences and hence, the current design was meant to formally assess this potential. This way the
proposal was based on having children playing together in pairs to explore a space (physical and virtual) in which
interaction with the virtual elements could offer a trigger to promote social interaction. The aforementioned
requirements led to a physical configuration of the prototype that was based on a circular floor projection
approximately 6 meters in diameter (Figure 1). This should provide a sufficiently large space for children to explore
it in a natural way.
10
For the workshop, a preliminary version of the environment was designed for pilot testing. The aim of this
prototype was to offer a flavor of the possibilities of the interactive experience, so children can have an initial
starting point to ground their contributions. This first prototype only presented the main scenario of the experience,
which was a land covered with fog. Its design was based on merging the requirements of the psychologists with the
designers knowledge. During the experience, children could move freely in the space to explore it through the use
of a physical object (a butterfly net). In this very basic and preliminary prototype, we decided not to include any
virtual elements, specific content, or game mechanics in order to provide space for childrens contributions.
During the workshop, the basic prototype was implemented and deployed as a Wizard of Oz system.
Hence, although the behavior of the fog was already computationally implemented, a researcher actually remotely
controlled the fogs reactions to childrens behavior. Specifically, a researcher manually tracked the position of the
butterfly net of the child and made the fog opening as a peephole that allowed the children to see a small portion of
the underlying world (Figure 2).
Figure 1. Physical configuration of the system
11
Figure 2. Children interacting with the system during the PD workshop
3.2 The workshop
The workshop took place in a large multi-purpose room in our university. The overall workshop lasted for
three sessions of one and a half hours each. In the first session we focused on childrens embodied interaction and on
their interpretations of the environment, whereas in subsequent sessions we addressed the design of the visual
appearance and behavior of the games characters. Since the purpose of this study was to address the construction of
meaning through embodied interaction, this paper will focus on the first session only. The participants, selected by a
psychologist, were four high-functioning ASD children (A, B, C & D), all male, between 10 and 12 years old, who
had not previously visited the university. Some of the children had previously met from sharing group therapy
sessions. All four children presented normal cognitive capabilities and functional language. During the session, three
researchers and a psychologist were present as observers and facilitators.
3.2.1 Workshop structure
After introducing ourselves, we explained to children the goals of the workshop and the importance of their
participation in the design. During the session two main activities were carried out: the “exploratory activity” and the
“drawing activity”. The aim of this structure was to provide children with activities that offered different resources to
12
approach the environment (e.g. physical interaction, verbal explanations, drawings, etc.), hence offering different
modal affordances to communicate their understanding.
1) The exploring activity
After the presentation, children were divided into pairs. Each pair received a specific role, either “explorer”
or “detective”. One of the explorers was given a camera while the other was given a butterfly net, and both were
invited to enter the floor projection area showing the digital environment (Figure 2). Specifically, they were
instructed to walk over the floor projection and explore the space together. The butterfly net served as a magic wand
to open the fog wherever it was placed. The child holding the camera was instructed to take pictures of anything that
captured his interest. They were also told that they could exchange their tools (net and camera) whenever they
wanted. The interaction with the digital environment was reduced to a minimum. When a child moved the net, the
fog opened, generating a circular area through which they could see part of the underlying world. In this prototype,
the world was merely a number of colored areas to provide a sense of moving through different types of land or
water.
The children assigned to the “detective” role were led to a balcony in the upper floor, which overlooks the
entire multi-purpose room. From this vantage point they could see the environment from a bird’s-eye view. Here,
along with two researchers, they were asked to try to work out what the environment looked like, any possible
creatures that could inhabit it and what their peers should do in their exploration. After 15 minutes children swapped
roles; those acting as “explorers” became “detectives” and vice versa. This activity was planned to observe children’s
spontaneous interactions with the digital environment and with each other, and to analyze the kind of interpretations
evoked.
2) The drawing activity
After finishing the exploring activity, children were conducted to a part of the room with tables, chairs and
crafting materials (pencils, markers, scissors, magazines, different types of paper, etc.). They were debriefed about
the previous activity and asked to draw what they would like to see under the fog. Children were invited to produce
drawings on white paper, where only a circular shape, representing the digital environment, was provided. To
13
produce their drawings, children could freely choose between the different materials. After finishing the drawings,
we asked children to describe the drawings to their peers.
3.3 The analytical framework
At an analytical level, three main concepts derived from socio-semiotic and multimodal analysis were
employed to frame the research: the situated and embodied meaning-making, the relevance of the interpreter’s
interests and the notion of motivated sign. The goal of the analysis was to get insights on two fundamental aspects.
First, we wanted to analyze how children naturally interact with the environment and which aspects grab their
attention. This analysis would facilitate the definition of design solutions capable of engaging children in the
exploration. Second, we wanted to explore which contents could motivate children. This research would facilitate
the design of valuable situations for the children to become sufficiently motivated to communicate, thus promoting
social interaction between users.
To analyze the data, one researcher first transcribed the videos according to a narrative style, in order to
have an overall view of the experience. After that, she performed specific transcriptions, using different formats, for
each unit of analysis (e.g. different forms of graphical transcriptions). This approach was based on the method
proposed by Flewitt (2009) to analyze multimodal data.
3.3.1. Situated and embodied meaning making
Multimodal analysis stems from the acknowledgement that people use multiple resources to construct
meaning. Thus, it allows the analysis of the different semiotic resources that humans employ to make meaning and
the analysis of how these resources interact with each other (Streeck et al., 2011). By taking into account the
specificities of the analyzed environment and of the proposed PD activities, we focused our analysis on the
following embodied resources:
a) Children
s exploration of the space (their position, paths, pauses and relative speed),
b) Movements of the butterfly net (the position of the butterfly net in relation with the child and the pattern
of movements performed with it),
c) Gaze direction.
14
The choice of focusing on these aspects was motivated by some fundamental considerations. Firstly, to
understand embodiment the unit of analysis should be able to encompass the relationship between the agents and
their environment (Clark, 1998). Secondly, the physical nature of the experience requires to carefully considering
how users engage with it by using their bodies, the space and material artifacts.
The analysis of these aspects was performed on the video recordings of the exploratory activity.
Specifically, we analyzed the first minute of the children’s interaction while holding the net and exploring the
environment. We chose to perform a fine-grain analysis of this first stage of exploration only, in order to gather
insights about which aspects immediately called their attention and curiosity, and which aid in framing their initial
impressions and interpretations of the environment. This transcription was carried out through repetitive frame-by-
frame visualization of videos. Children
s exploration of the space and net movements were transcribed in a map-like
format for each child. Gaze direction instead, was annotated in a written format during the video observation.
3.3.2. The relevance of interpreter’s interests and focus of attention
According to the semiotic theory proposed by Kress (2010) communication follows this sequence: A
message is sent
!
The attention of the receiver engages her with some aspects of the message
!
Some aspects of
the message are selected according to her interests
!
These aspects are framed and interpreted. This flow suggests
the central role of the interests and focus of attention of the receiver in the selection, framing, interpretation and
understanding of a message.
This concept, applied to digital technologies, implies considering the designed experience as a message that
needs to be interpreted by the users. This approach could be particularly valuable in Full-Body Interaction, as these
settings are characterized by a plethora of stimuli, all off which can potentially attract the interest of the users. At the
same time, it can be suitable when working with ASD children since, by relying on embodied resources, it
overcomes the barrier of verbal communication. In the context of PD, we propose that the multimodal analysis of
the users’ focus of attention can provide the basis for grasping their understanding of the system and can allow for
identifying relevant insights to guide design refinements.
At a methodological level, we evaluated children’s focus of attention by combining the analysis of: (1) their
paths in the interactive space and (2) the pictures they took during the exploration. These techniques were based on
literature on the analysis of users’ engagement and interests. The tracking of users has been widely employed for
15
analyzing their engagement with physical and digital environments. Eye-tracking systems (e.g. in websites) or
tracking of users physical exploration (e.g. in museums) have shown to be an effective tool to capture areas of
interest and exploratory behaviors (Lorigo et al., 2008; Serrell, 1997). At the same time, pictures taken by children
have been widely employed in social sciences to research on children’s perspective and experiences (Einarsdottir,
2005; Harper, 2002; Jorgenson and Sullivan, 2009). Moreover, the triangulation of these techniques can offer a
wider panorama on what aspects of the digital environment capture (or do not capture) the attention of the users,
thus providing guidelines to understand their interests.
Specifically, to perform this analysis we traced the graphical transcriptions of all childrens paths (described
in section 3.1.1) over the image of the environment. This graphical solution allowed us to obtain a representation of
their overall traversal of the space. At the same time, the pictures taken by the children during the course of the
exploratory activity were analyzed using a topological approach. In other words, the pictures were mapped onto the
image of the environment, so as to obtain the correspondence with the area depicted in the picture. As a result we
produced two maps, one summarizing the paths (Figure 3) and the other showing the position of the pictures (Figure
4). Furthermore, the pictures were analyzed according to a grounded approach, understood as the process to
inductively define codes that group the pictures into categories that describe what was depicted.
3.3.3 The notion of motivated sign
According to multimodal social semiotics, when somebody makes a sign, they are projecting their
individual interests into the world by making choices from the available resources. As a consequence, “the inner
constitution of a sign reveals the interests of the sign maker” (Kress, 2010).
This idea has direct implications for the evaluation of user-generated artifacts during PD activities. In
particular, we suggest that the analysis of resources and cultural references employed by children can reveal their
interests and offer relevant insights for deepening on their expectations about the system. Furthermore, this analysis
can offer relevant guidelines to improve design and to make it more appealing and engaging for children.
To perform this analysis we focused on the following activities: childrens explanations during the
exploratory activity, children’s drawings and the narratives accompanying these drawings. This approach was
grounded on research showing that drawings and related narratives represent a valid research method to access
16
young children’s views and experiences (Clark, 2005; Cox, 2005). This evaluation was mainly performed through
grounded content analysis, understood as a methodological approach to inductively derive meaning from data. Its
goal was to identify resources and cultural references employed by children.
4. Results
4.1 Situated and Embodied meaning making
The analysis of embodied meaning making was derived from the maps produced to describe children
s
exploration of the space and the movement of their net (Figure 3). In the following sections we provide a detailed
analysis of these different aspects.
Figure 3. Maps showing the childrens exploration of the space(blue line), the movement of their net
(red line) and salient actions (square inserts)
4.1.1 Children
s exploration of the space
For the analysis of children’s exploration of the space we transcribed the child’s position, his path, the
pauses and the relative speed in each individual map. The maps of children’s paths (blue line in Figure 3) reveal that
the children tended to move in similar ways. Specifically, they either remained on the edge of the environment,
17
where they tended to follow circular paths, or they performed straight trajectories to cross the space through its
center.
However, different patterns of movement were present. While child A tended to move forward almost all
the time, children C and D preferred to explore the same areas several times by repeating similar paths. Child B,
instead, tended to move in a straight path and alternated forward and backward movements aimed at passing through
the same path he had just walked through.
The differences in their paths indicate different exploratory behaviors. While the first child seemed to
prefer to cover the largest possible space during a short lapse of time, the other children showed a more fine-grained
approach, carefully and deeply exploring a specific area. These tendencies were partially confirmed by the pace of
their exploration (speed and pauses).
All children experimented with and alternated different walking speeds and they all performed some pauses
during their exploration. Pauses were associated with: 1) changes in direction, 2) exploration of the area by standing
still and moving the net, 3) changes in the movement pattern of the net and 4) repetitive behaviors. In particular,
child D stood for more than 20 seconds on the same position, outside the edge of the projected play area, performing
repetitive actions with the net. In the context of autism, repetitive behaviors are associated with self-stimulation,
understood as patterns of movements that are performed to cope with over-excitement. This finding, thus, suggested
the need for designing improvements to reduce this kind of behavior. At the same time we found other important
points related to interaction design. Considering that one of the project’s requirements was to offer affordances that
promote exploration (see section 3.1), the following questions were raised: What happens when the child stands in
the same position for a long time? What happens when he goes back and forth several times through the same area?
Could the system recognize and react to repetitive actions and how could it do so? Should the system reward
differentiation and variability in the exploratory paths? How could this reward be formalized?
4.1.2 Movements of the butterfly net
To analyze the movements that the children performed with the net, we focused on the position of the net in
relation to their bodies (red line in Figure 3) and on the patterns of movement that they enacted with it. All children
tended to maintain the net in front of them, by moving it horizontally left and right. However, three children
18
performed some changes in their patterns of movement of the net (A= 4 changes; D= 6 changes; C= 4 changes),
while child B did not perform any changes and just slightly modified the inclination of the net.
Explored patterns of movement were related to performing semicircular movements on a side or in front of
the body, or dragging the net on the floor and using it to draw figures in the fog. The latter behavior suggests an
interest towards exploring the changes in the visual appearance of the fog when it is “opened” by the net. It thus
suggests eventual design refinements related to the variability of the fog’s visual appearance in order to promote
exploration. Taking into account the goals defined for the design of the system, the high variability in performed
patterns of movement elicits relevant questions for design and for the definition of technical requirements: How does
the system react to different kinds of movement? Does the system reward variability and exploration of the net’s
movements? Which is the best technical solution for facilitating tracking the net’s position while simultaneously
promoting exploration?
4.1.3 Gaze Direction
During the first minute of interaction all children’s gazes were completely focused on the floor, where the
digital environment was projected. No shifts in focus were reported, neither looking at the surroundings nor at their
peers. This observation suggests children’s interest in the digital environment and coherently relates to autism
literature, which shows that ASD subjects tend to avoid making eye-contact and have a poor interest toward their
peers (Phillips et al., 1992; Zeeland et al., 2010).
However it also points out some relevant design considerations. Showing interest towards other people and
making eye contact are important learning goals for ASD children. Furthermore, since one of the requirements of the
project was to promote social interaction, future design refinements should address proposals for facilitating
functional forms of social interaction (e.g. performing eye-contact when communicating with somebody). Possible
research paths can address questions related to: Which physical affordances can invite the user to look at their
peers? Which game mechanics can promote eye-contact between users? Potential sources of inspiration can be
found in the exploration of novel physical interfaces and in the “head-up games” paradigm (Soute et al., 2009),
which proposes design solutions to liberate players from facing down to attend to screen-based interactions.
19
4.2 Childrens Focus of Attention
The analysis of childrens focus of attention was derived from the maps produced to describe their overall
traversal of the space and the topographical arrangement of the pictures that they took during the activity (Figure 4).
Figure 4. a) Representation of overall displacements (left); b) Map of pictures taken by the children (right)
From the map of the overall traversal of the space, we observe that almost all children concentrated their
exploration on specific areas. Children’s movements focused mainly on the borders between different types of
environments (e.g. borders between a blue area and a green area). These observations are confirmed by the
topological analysis of the distribution of the pictures they took (Figure 4) where we can see that the pictures are
concentrated mainly on border areas. Content analysis of the 56 pictures showed that roughly 45% depict borders
between two different terrains, 35% include the virtual pointer and the others were distributed between images of the
net, shadows and plain areas. Within that, it is particularly relevant to notice that plain areas were depicted only in
9% of the pictures, even though they constituted the biggest proportion of the underlying environment. At the same
time, the children did not take any pictures of their peers or of the surroundings beyond the projected environment.
These findings suggest that children tended to direct their interests toward detailed and liminal areas, where
some salient visual changes were present. These aspects can be interpreted according to the role of salience in
facilitating perception (Price & Scaife, 2002) as well as in the light of ASD children’s interest for detailed images
(Happé & Frith, 2006) and in relation to the evocative power of liminal areas. While exploring the environment,
children were engaged with trying to discover what was hidden under the fog. Thus, areas that depict borders and
visual changes can be much more informative than plain areas.
20
At a design level, this analysis suggests some important reflections. First of all, it strengthens the
previously identified need for defining strategies that allow children to gain interest in other users instead of
focusing only on the environment. Secondly, it can provide relevant insights for guiding the design of the visual
appearance of the environment, for instance through the spatial distribution of different kinds of salient features to
promote children’s exploration or to guide them toward certain locations.
4.3 Analysis of children’s productions
To analyze children’s productions, we used three sources: the explanations given by “detectives” during the
exploratory activity, the drawings made by children and their associated narratives.
4.3.1 Children’s explanations of the FUBILE
During the exploratory activity, children assigned to the “detective” role were situated on a balcony from
which they could see the environment from a birds-eye view. They were instructed to work out what was
represented in the digital environment and what their peers, who were exploring, should do. Here, two researchers
asked children questions related to their interpretations of the environment. We used content analysis to analyze
children’s answers.
During this activity all children dedicated full attention towards looking at the environment. Interpretations
were focused in two main directions. Three children understood it as a geographical map representing a large space
such as a continent or a state. In contrast, child B suggested that there was a figurative image (a bat) hidden under
the fog, which had to be discovered.
At an interaction level, all children related the activity performed by their peers with the act of hunting or
collecting. Interestingly, all children explained it according to a strictly functional perspective (e.g. “You have to
collect animals to make points”). These functional explanations suggest the tendency to understand the environment
according to a goal-oriented and reward-orientated approach, which can probably be attributed to their gaming
culture.
At a design level, the video gaming culture posits relevant questions on the need for balancing expectations
derived from gaming experience with educational requirements. Furthermore, the goal-oriented approach suggests
21
that the presence of a narrative “mission” structure could facilitate children’s understanding of the experience. This
consideration is consistent with ASD literature, which points out the difficulties of ASD children in dealing with
open-ended and ill-structured problems and their preference for well-structured and goal-directed situations
(Strickland, 1997). However, from an interaction design perspective, relevant research paths could explore whether
the difficulties of open-ended situations could be fruitful for promoting social interaction, by requiring the child to
ask for external collaboration. Nonetheless, this difficulty level should be carefully balanced to avoid overwhelming
situations.
4.3.2 Children’s drawings
In the drawing activity children were very engaged and put a lot of effort and time in refining their
drawings. Unfortunately, due to the position of the camera, it was not possible to perform a fine-grained analysis of
the drawing process. Thus, only the final drawings and accompanying explanations were analyzed.
The analysis of children’s drawings points out their preferences for a rich and detailed natural environment.
Following the ideas they explained in the exploratory activity, one child depicted a bat, while the others drew
landscapes. Within that, two children depicted highly realistic environments where natural elements dominated the
scene (i.e. wood, sea, mountains, etc.) and human elements were minimized, mainly representing anachronistic
settings (e.g. a temple, a coliseum). The third child also proposed some fantasy elements (e.g. “a cotton candy
valley”), however this lack of realism was not well accepted by the other children, who criticized it because “that is
fantasy”.
At a formal level, drawings were composed using mixed techniques, mainly combining drawing with
collage. The use of collage was mainly due for economy of work. Children were interested in representing detailed
and dense natural areas: thus, collage using magazine pictures gave them a quick way to include highly detailed
representations with less effort (e.g. foliage). In this context, it is relevant to notice that children strongly preferred
natural and detailed pictures and rejected some images because “it has too few leaves”. This need for realism and
detail can inform the design of the visual appearance of the environment. However, further research should address
how detailed visual representations may affect exploration.
At the level of cultural referents, children’s proposals were mainly grounded on their gaming and
audiovisual culture, which was shown to have a high priming effect on children’s expectations.
22
5. Discussion
Our study confirms previous research on the suitability of multimodal analysis for the evaluation of digital
technologies (Jewitt, 2013) and widens its application to design processes that are based on PD workshops. Within
this context, we suggest that multimodal analysis can offer both direct and indirect benefits to PD.
At a methodological level, multimodal analysis has shown to be suitable to address the challenges related to
analyzing PD by systematically attending to the multiple resources and by providing solid empirical data upon
which inferences may be grounded. These benefits contributed to offer guidelines to orient the design process. At
the same time, the outcomes proceeding from the analysis served as a fruitful ground to facilitate the communication
between experts, designers and researchers.
5.1 A multimodal approach to analyze PD
During Participatory Design workshops participants may offer a huge diversity of possible contributions,
choices and interpretations. In this context, our proposal to combine multimodal analysis with PD activites allowed
us to track children’s interpretations across different modes. We do not claim that this approach allowed us to elicit
a higher number of ideas from children, but rather, that by focusing on multiple resources it allowed us to enrich our
observations on the set of contributions that actually influence future design choices. In this context, we consider
that the observation of embodied and situated meaning making, children
s focus of attention and children
s
productions actually provides relevant guides for the analysis of PD workshops.
On the one hand, we suggest that the analysis of aspects such as childrens exploration of the space, their
interaction with the material artifact (the butterfly net) and their focus of attention, may offer some important
contributions. First, since it requires researchers to move away from analyzing intentional communication only, it
allows focusing on often-neglected aspects such as physicality. This allows paying attention to nuances of meaning
and behavior that may otherwise pass unnoticed and enables insights that would have not been otherwise accessible.
Second, in going beyond the limits of analyzing only communicational prompts, the analysis enhances the richness
of the users’ contributions that can be obtained from PD. This allows the body and the space to be included in the
research related to embodied interaction. This aspect does not only have a practical methodological value, but also
23
presents some important epistemological implications, since it conceptualizes the user from a holistic and
encompassing perspective. Third, the analysis of multimodal resources can be crucial especially when working with
populations that may have difficulties in expressing and communicating their interests, e.g. children with special
needs. This aspect addresses the ethical and pragmatic concerns related to involving a population with special needs
in the design process. Finally, the use of a fine-grain approach to analyze these aspects allowed gathering robust data
upon which specific inferences on future design iterations and research can be grounded.
On the other hand, the employed creative activities (e.g. drawings) allowed children to project and
materialize their own interests and representations in the production of artifacts (Kress, 2010), hence providing
researchers with rich data for investigating their interpretations. Thus, combining the observation of embodied and
situated meaning making with the analysis of children
s productions, showed to be an effective approach to deepen
on the meanings that children attribute to their actions and their interpretations of the experience.
To sum up, the aspects proposed in the analytical framework (section 3.3) may be useful as guidelines for
researchers involved in design and assessment in this field. Nonetheless, the situated and context-specific nature of
this framework requires that researchers carefully tailor their tools and approaches to the characteristics and concrete
features of their specific contexts and applications.
5.2 Multimodal Analysis and PD as tools to guide the design process
During physical interaction with the environment, children focused on exploring and photographing
detailed border areas. This preference is consistent with their choices during the drawing activity, where they mainly
elaborated highly detailed and realistic representations of geographic landscapes. Thus, from the perspective of
defining the visual appearance of the virtual world, these findings suggest that realistic and detailed representations
may be adequate to capture children’s attention and foster exploration behaviors.
On the other hand, we used two different approaches to elicit children explanations: the first during the
detective activity and the second after the drawing activity. Their explanations during the detective activity were
more focused toward describing the space in terms of actions (“you need to capture animals”). Instead, their
explanations of the drawings were mainly in a map-like format (“here we have a forest, there a river...”). This
difference points out how the two activities, by offering different affordances, have elicited different ways of
describing the space: one in terms of the actions of the user and the other in terms of the coexistence relations
24
between objects (De Certeau, 1984). Hence, in the context of the workshop, the shift between different modes (e.g.
exploring, explaining, and drawing) and the use of multimodal analysis, allowed identification of relevant esthetic as
well as functional interests.
In relation to the latter aspect, children were mainly interested in a space that is “actionable” with video
game-like tasks. The priming role of video games was evident both in their explanations and in the elements in their
drawings. The influence of video game culture may suggest possible design solutions to foster children’s interest in
the experience and provide hints to define motivators of social interaction. However, it also points out challenges
related to properly defining the educational experience. For instance, these children were mainly used to games in
which players compete and fight against enemies or other players. However, these common game mechanics were in
conflict with the very educational goals of the experience. As a consequence, when designing serious games,
relevant research challenges should aim for a balance between children’s expectations on “what a game is” and the
values or educational goals that need to be achieved. Involving children in the process and understanding their
interests, forms of usage, values and cultural references, constitutes an essential starting point. At the same time,
strategies to balance knowledge and requirements from different stakeholders constitute principal contributions to
the design process. Nonetheless, in order to properly address this challenge, further research is still needed. We
suggest that future research should address the analysis of the construction of specific socio-cultural values and the
ethical reflection on the dilemma between reproducing them or diverging from them.
In summary, the employment of multimodal analysis proved to be an effective and coherent approach for
capturing, analyzing and interpreting a wide range of users’ contributions in a PD workshop with ASD children.
This helped us gain insights related to precise aspects of interaction design, technical requirements, content
definition and educational goals, hence providing solid guides for design refinements.
5.3 Grounding a multidisciplinary dialogue
During the design process carried out by our multidisciplinary team, the outcomes obtained from the
multimodal analysis provided a useful bedrock upon which to build dialogue. This process was carried out by
framing and linking back the outcomes of the analysis with the projects requirements. This connection was shaped
through the formulation of specific design questions. The presentation of the empirical data, together with the raised
design questions, covered a fundamental social and maieutic function by serving as a starting point to guide
25
communication and foster reflection between multiple stakeholders (i.e. mental health professionals, designers, etc.).
This aspect is particularly important in multidisciplinary teams where the different perspectives, worldviews and
vocabularies may constitute a barrier for the proper flow of dialogue (Frauenberger et al., 2014; Mora et al., 2014).
We thus suggest that multimodal analysis, by providing a broad, robust and coherent analysis of PD, can
facilitate the orchestration of multiple perspectives for the definition of serious games features. This encompasses
both approaches to include children’s contributions as well as to facilitate dialogue in a multidisciplinary team. At
the same time, its maieutic function does not only allow refining of design and facilitating of interdisciplinary
communication, but it also opens relevant paths for further studies.
5.4 Limitations and future works
As this paper is based on an exploratory case study, a number of relevant improvements for future work are
apparent. Firstly, improvements should address technical issues aimed toward facilitating data recording. A careful
consideration of the disposition and quality of the recording devices becomes of crucial importance. Furthermore,
since the proposed approach requires a considerable time-consuming effort which may be unsuitable for certain
contexts, automatic systems for data logging and analysis should be considered (e.g. in our case for tracking user
and net position). We are currently working on the development of a software tool to help in this regard.
Secondly, relevant research and design possibilities could address the combination of this method with
iterative design approaches, such as Design-based research (Barab & Squire, 2004; Wang & Hannafin, 2005). In this
context, multimodal analysis could provide analytical tools to guide and orient design iterations and evaluation.
Thirdly, relevant considerations should focus on defining fruitful relations between proposed PD activities
and analysis. In our case, for instance, both the exploratory activity and the pictures proved to be particularly
suitable for this kind of analysis. On the other hand, the drawing activity presented some methodological
weaknesses in gaining deeper insights on the use of multimodal resources. This shortcoming points to how the
design of the elicitation activities should be informed both by the specific research goals as well as by the analytical
framework we want to apply.
Fourth, the proposed framework does not aim at being a prescriptive general-purpose tool, but a flexible
and situated approach, which should be adapted and reconfigured according to the specificities of different research
contexts and projects.
26
Finally, in the presented analysis, the reduced number of users and the relatively short interaction time that
was analyzed may have offered only a limited perspective. Thus, the design refinements that we have identified and
discussed should be implemented and evaluated in future iterative studies.
6. Conclusions
We have presented an exploratory case study aimed at applying multimodal analysis to a PD workshop
with autistic children. We have applied the multimodal analysis to enrich the analysis of childrens contributions and
enhance communication of the results of the PD workshop to the rest of the interdisciplinary team. Specifically, the
analysis has been structured according to three main concepts derived from socio-semiotics: situated and embodied
meaning making, interpreter interests and attention focus, and motivated sign. We have applied these concepts to
the analysis of children’s exploration of physical and virtual interaction space, to their explanations and handcrafted
productions. Within that, different modal resources have been analyzed.
The results of the study showed that multimodal analysis constitutes an effective and coherent method to
capture, analyze and interpret users’ contributions during a PD workshop. Through this approach we managed to
overcome communication challenges and obtain insights that otherwise would not have been accessible by research
methods focused only on intentional communicational prompts. Major insights were related to interaction design,
technical requirements, content definition and alignment with educational goals. At the same time, by uncovering
unsolved design aspects, this approach can facilitate the orchestration of multiple perspectives in the refinement of a
design proposal.
7. Acknowledgements
This project has been funded through the competitive grant RecerCaixa 2013 from the Obra Social la
Caixa, Barcelona. We would like to acknowledge the four ASD children that collaborated with us in the PD and
their families. We are also very thankful for Sara Price for her review and feedbacks during the paper preparation.
8. References
Bachelard, G. (2002). The formation of the scientific mind. Manchester: Clinamen Press.
27
Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning
Sciences, 13(1), 1–14.
Ben-Sasson, A., Lamash, L., & Gal, E. (2013). To enforce or not to enforce? The use of collaborative interfaces to
promote social skills in children with high functioning autism spectrum disorder. Autism, 17(5), 608–622.
Benton, L., Vasalou, A., Khaled, R., Johnson, H., & Gooch, D. (2014). Diversity for Design: A Framework for
Involving Neurodiverse Children in the Technology Design Process. In Proceedings of the 2014 annual
conference on Human factors in computing systems (pp. 3747–3756). New York, NY, USA: ACM Press.
Bertacchini, F., Bilotta, E., Gabriele, L., Vizueta, D., Pantano, P., Tavernise, A., Vena, S., Valenti, A. (2013). An
emotional learning environment for subjects with Autism Spectrum Disorder, In Proceedings of the
Interactive ollaborative Learning 2013. 653–659.
Bezemer, J. (2008). Displaying orientation in the classroom: Students’ multimodal responses to teacher instructions.
Linguistics and Education, 19(2), 166–178.
Boyd, B. A., Hume, K., McBee, M. T., Alessandri, M., Gutierrez, A., & Johnson, L. S. (2014). Comparative efficacy
of LEAP, TEACCH and non-model-specific special education programs for preschoolers with autism
spectrum disorders. Journal of Autism and Developmental Disorders, 44(2), 366–368.
Church, R. B. (1999). Using gesture and speech to capture transitions in learning. Cognitive Development, 14(2),
313–342.Charlton, B., Williams R. L., McLaughlin, T. F. (2005). Educational games: A technique to
accelerate the acquisition of reading skills of children with learning disabilities. Int. Journal of Special
Education, 20 (2), 66-72.
Clark, A. (1998). Embodiment and the Philosophy of Mind. Royal Institute of Philosophy Supplement, 43, 35-51
Clark, A. (2005). Listening to and involving young children: A review of research and practice. Early Child
Development and Care, 175(6), 489–505.
Cox, S. (2005). Intention and meaning in young children’s drawing. International Journal of Art & Design
Education, 24(2), 115–125.
Crescenzi, L., Jewitt, C., & Price, S. (2014). The role of touch in preschool children’s learning using iPad versus
paper interaction. Australian Journal of Language & Literacy, 37(2), 86–95.
28
de Certeau, M. (1984) The Practice of Everyday Life. Berkeley: University of California Press.
Derboven, J., Van Mechelen, M., Slegers, K. (2015). Multimodal Analysis in Participatory Design with Children: A
Primary School Case Study. In Proceedings of the 33rd Annual ACM Conference on Human Factors in
Computing Systems (CHI '15). ACM, New York, NY, USA, 2825-2828
Deterding, S., Khaled, R., Nacke, L.E., Dixon, D. 2011. Gamification: Toward a Definition. In CHI 2011
Gamification Workshop Proceedings, Vancouver, BC, Canada.
Druin, A. (2002). The Role of Children in the Design of New Technology. Behaviour and Information Technology,
21(1), 1–25.
Einarsdottir, J. (2005). Playschool in pictures: Children’s photographs as a research method. Early Child
Development and Care, 175(6), 523–541.
Flewitt, R. S., Hampel, R., Hauck, M., & Lancaster, L. (2009). What are multimodal data and transcription? In C.
Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 28–39). London, UK: Routledge.
Frauenberger, C., Good, J., Alcorn, A., & Pain, H. (2012). Supporting the design contributions of children with
autism spectrum conditions. Proceedings of the 11th International Conference on Interaction Design and
Children - IDC ’12, 134. doi:10.1145/2307096.2307112
Frauenberger, C., Good, J., Alcorn, A., & Pain, H. (2013). Conversing through and about technologies: Design
critique as an opportunity to engage children with autism and broaden research(er) perspectives. International
Journal of Child-Computer Interaction, 1(2), 38–49. doi:10.1016/j.ijcci.2013.02.001
Frauenberger, C., Good, J., Fitzpatrick, G., & Sejer Iversen, O. (2014). In pursuit of rigour & accountability in
participatory design. International Journal of Human-Computer Studies, 74, 93–106.
doi:10.1016/j.ijhcs.2014.09.004
Frauenberger, C., Good, J., & Keay-Bright, W. (2011). Designing technology for children with special needs:
bridging perspectives through participatory design. CoDesign, 7(1), 1–28.
doi:10.1080/15710882.2011.587013
29
Fuentes-biggi, J., Ferrari-arroyo, M. J., Boada-muñoz, L., Touriño-aguilera, E., Martos-pérez, J., Paz, M. P., …
Sanidad, M. De. (2006). Guía de buena práctica para el tratamiento de los trastornos del espectro autista.
Revista de Neurología, 43(7), 425–438.
Giaccardi, E., Paredes, P., Díaz, P., & Alvarado, D. (2012). Embodied Narratives!: A Performative Co-Design
Technique. In Proceedings of the Designing Interactive Systems Conference (pp. 1–10). New York, NY, USA:
ACM.
Goh, D. H., Ang, R. P., & Tan, H. C. (2008). Strategies for designing effective psychotherapeutic gaming
interventions for children and adolescents. Computers in Human Behavior, 24(5), 2217–2235.
doi:10.1016/j.chb.2007.10.007
Goldin-Meadow, S. (2011). Learning through gesture. Wiley Interdisciplinary Reviews: Cognitive Science, 2(6),
595–607.
Goodwin, C. (2000). Action and embodiment within situated human interaction. Journal of Pragmatics.
Hansen, O.B., Abdurihim, A., McCallum, S. (2013). Emotion recognition for mobile devices with a potential use in
serious games for autism spectrum disorder. SGDA 2013. 8101, 1–14. Happé, F., & Frith, U. (2006). The
weak coherence account: detail-focused cognitive style in autism spectrum disorders. Journal of Autism and
Developmental Disorders, 36(1), 5–25.
Harding, S. (1989). Is there a feminist method. In N. Tuana (Ed.) Feminism and science (pp. 18–32). Bloomington,
Ind: Indiana University Press.
Harper, D. (2002). Talking about pictures: A case for photo elicitation. Visual Studies, 17(1), 13–26.
doi:10.1080/14725860220137345
Harrison, S., Sengers, P., & Tatar, D. (2011). Making epistemological trouble: Third-paradigm HCI as successor
science. Interacting with Computers, 23(5), 385–392. doi:10.1016/j.intcom.2011.03.005
Hindmarsh, J., & Pilnick, A. (2002). The tacit order of teamwork: collaboration and embodied conduct in anesthesia.
The Sociological Quarterly, 43(2), 139–164.
Hoepfl, M. C. (1997). Choosing qualitative research: A primer for technology education researchers. Journal of
Technology Education, 9(1), 47–63.
30
Hoque, M. E., Lane, J. K., elKaliouby, R., Goodwin, M. & Picard, R. W. (2009). Exploring Speech Therapy Games
with Children on the Autism Spectrum, In
Proceedings of InterSpeech, Brighton, UK, 1455-1458, ISCA
Hummels, C., Overbeeke, K. C. J., & Klooster, S. (2006). Move to get moved: a search for methods, tools and
knowledge to design for expressive and rich movement-based interaction. Personal and Ubiquitous
Computing, 11(8), 677–690. doi:10.1007/s00779-006-0135-y
Jaworski, A., & Thurlow, C. (2009). Gesture and movement in tourist spaces. In C. Jewitt (Ed.), The Routledge
handbook of multimodal analysis (pp. 253–266). London, UK: Routledge.
Jewitt, C. (2009). Different approaches to multimodality. London: Routledge. In C. Jewitt (Ed.), The Routledge
handbook of multimodal analysis (pp. 28–39). London, UK: Routledge.
Jewitt, C. (2013). Multimodal methods for researching digital technologies. In S. Price & C. Jewitt (Eds.), The
SAGE handbook of digital technology research. (pp. 250–265). Los Angeles, CA, USA: SAGE Publications
Ltd.
Jorgenson, J., & Sullivan, T. (2009). Accessing children’s perspectives through participatory photo interviews.
Qualitative Social Research, 11(1).
Korkiakangas, T., & Rae, J. (2014). The interactional use of eye-gaze in children with autism spectrum disorders.
Interaction Studies, 15(2), 233–259.
Kress, G. (2010). Multimodality. A social semiotic approach to contemporary communication. London, UK:
Routledge.
Lorigo, L., Haridasan, M., Brynjarsdóttir, H., Xia, L., Joachims, T., Gay, G., … Pan, B. (2008). Eye tracking and
online search: Lessons learned and challenges ahead. Journal of the American Society for Information Science
and Technology, 59(7), 1041–1052.
Malinverni, L., Mora Guiard, J., Padillo, V., Hervas, A., & Pares, N. (2014). Participatory Design Strategies to
Enhance the Creative Contribution of Children with Special Needs. In Proceedings of the 2014 Conference on
Interaction Design and Children (pp. 85-94). ACM, New York, NY, USA DOI=10.1145/2593968.2593981
Malinverni, L., & Pares, N. (2014). Malinverni, L., & Pares, N. Learning of Abstract Concepts through Full-Body
Interaction: A Systematic. Educational Technology & Society, 17(4), 100 –116.
31
Mazzone, E., Tikkanen, R., Read, J. C., Iivari, N., & Beale, R. (2012). Integrating children ’ s contributions in the
interaction design process. Internation Journal of Arts and Technology, 5(2), 319–346.
Millen, L., Cobb, S. V. G., Patel, H., & Factors, H. (2010). Participatory design with children with autism.
International Journal on Disability and Human Development., 10(4), 289–294.
Mondada, L. (2011). Understanding as an embodied, situated and sequential achievement in interaction. Journal of
Pragmatics, 43(2), 542–552. doi:10.1016/j.pragma.2010.08.019
Mora, J., Malinverni, L., & Pares, N. (2014). Narrative-Based Elicitation: Orchestrating Contributions from Experts
and Children. In CHI ’14 Extended Abstracts: ACM SIGCHI Conference on Human Factors in Computing
Systems (pp. 1159-1164). ACM, New York, NY, USA. DOI=10.1145/2559206.2581292
Moser, C., Chisik, Y., & Tscheligi, M. (2014). Around the world in 8 workshops: investigating anticipated player
experiences of children. In Proceedings of the first ACM SIGCHI annual symposium on Computer-human
interaction in play (CHI PLAY ’14). (pp. 207–216). New York, NY, USA: ACM.
doi:10.1145/2658537.2658702
Muller, M. J. (2003). Participatory design: the third space in HCI. Human-computer interaction: Development
process, 4235.
Nesset, V., & Large, A. (2004). Children in the information technology design process: A review of theories and
their applications. Library & Information Science Research, 26(2), 140–161. doi:10.1016/j.lisr.2003.12.002
Phillips, W., Baron-Cohen, S., & Rutter, M. (1992). The role of eye contact in goal detection: Evidence from normal
infants and children with autism or mental handicap. Development and Psychopathology, 4(3), 375–383.
Piper, A. M., Brien, E. O., Morris, M. R., & Winograd, T. (2006). SIDES!: A Cooperative Tabletop Computer Game
for Social Skills Development. In Proceedings of the 2006 20th anniversary conference on Computer
supported cooperative work (CSCW ’06) (pp. 1–10).
Prensky, M. (2003). Digital game-based learning. ACM Computers in Entertainment, 1 (1), 1 - 4.
Price, S., & Jewitt, C. (2013). A multimodal approach to examining “embodiment” in tangible learning
environments. In Proceedings of the 7th International Conference on Tangible, Embedded and Embodied
Interaction (TEI ’13). (pp. 43–50).
32
Price, S., & Scaife, M. (2002). Animated Diagrams: How effective are explicit dynamics for learners? In P. Bell, R.
Steven, & T. Satwicz (Eds.), Keeping Learning Complex: The Proceedings of the Fifth International
Conference of the Learning Sciences (pp. 344–351). Mahwah, NJ: Erlbaum.
Resnick, M. (2002). Rethinking Learning in the Digital Age. In The Global Information Technology Report:
Readiness for the Networked World, edited by G. Kirkman. Oxford University Press.
Rogers, Y., & Marsden, G. (2013). Does he take sugar?: moving beyond the rhetoric of
compassion. interactions, 20(4), 48-57.
Scaife, M., & Rogers, Y. (1999). Kids as Informants!: Telling us what we didn ’ t know or confirming what we knew
already!? In A. Druin (Ed.), The design of children’s technology (pp. 1–26). Morgan Kaufmann Publishers
Inc., San Francisco, CA, USA.
Schaper, M., Malinverni, L., & Pares, N. (2014). Participatory Design Methods to Define Educational Goals for
Full-Body Interaction. In Proceedings of the 11th Conference on Advances in Computer Entertainment
Technology (ACE '14). ACM, New York, NY, USA. DOI=10.1145/2663806.2663867
Serrell, B. (1997). Paying attention: The duration and allocation of visitors’ time in museum exhibitions. Curator:
The Museum Journal, 40(2), 108–125.
Soute, I., Markopoulos, P., & Magielse, R. (2009). Head Up Games: combining the best of both worlds by merging
traditional and digital play. Personal and Ubiquitous Computing, 14(5), 435–444. doi:10.1007/s00779-009-
0265-0
Streeck, J., Goodwin, C., & LeBaron, C. (2011). Embodied interaction in the material world: An introduction. In
Streeck, J., Goodwin, C., LeBaron, C. (Eds.). Embodied Interaction: Language and Body in the Material
World ( pp.1-26). Cambridge, UK: Cambridge University Press.
Strickland, D. (1997). Virtual reality for the treatment of autism. Studies in Health Technology and Informatics, 81–
86.
Walsh, G., Druin, A., Guha, M. L., Foss, B., Golub, E., Hatley, L., & Franckel, S. (2010). Layered Elaboration!: A
New Technique for Co-Design with Children. In Proceedings of the SIGCHI Conference on Human Factors
in Computing Systems (pp. 1237– 1240). New York, NY, USA: ACM.
33
Wang, F., & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments.
Educational Technology Research and Development, 53(4), 5–23.
Whalen, C., Moss, D., Ilan, A. B., Vaupel, M., Fielding, P., MacDonald, K., Cernich, S., & Symon, J. (2010).
Efficacy of TeachTown: Basics computer-assisted intervention for the Intensive Comprehensive Autism
Program in Los Angeles School District. Autism, 14, 179–189.
Yan, F. (2011). A SUNNY DAY: Ann and Ron’s World an iPad Application for Children with Autism. In Serious
Games Development and Applications (pp. 129–138). Springer Berlin Heidelberg.
Zeeland, S. V., Ashley, A., Dapretto, M., & Ghahremani, D. G., Poldrack, R. A., Bookheimer, S. Y. (2010). Reward
processing in autism. Autism Research, 3(2), 53–67.
34
VITAE
Laura Malinverni
Laura Malinverni is a designer, artist and researcher in the field of education, art and new
technologies. She is trained in Fine Art & Design, Cognitive Science and Art Therapy.
Currently, she is a PhD student in the ICT Department of Universitat Pompeu Fabra. Her
research focuses on Interaction Design and Children and particularly on embodied interaction, Participatory Design
and socio-emotional aspects of learning. She has been organizing and teaching several workshops both with children
and adults and worked in the field of formal and informal education. As artist her works have been exposed in
several international events and exhibitions.
Joan Mora-Guiard
Joan Mora Guiard is a PhD Candidate in Universitat Pompeu Fabra. He is doing research
on the application of Full-Body Interactive Systems for interventions for children in the
Autism Spectrum Disorder. He has been working in the development of museology and
leisure interactive installations for different museums and events around Europe.
Narcis Pares
Narcis Pares is a Tenure Associate Professor in the ICT Department of Universitat
Pompeu Fabra (Barcelona, Spain). He has an interest in the possibilities of Full-Body
Interaction based on current Embodied Cognition theories in areas such as Learning, Play
and Special Needs. This research is undertaken from the standpoint of Interaction Design,
Interactive Communication and Interactive Playgrounds, focusing attention mainly on interaction for children and
35
using non-invasive technologies. He is senior member of the Cognitive Media Technologies Group and leader of the
Full-Body Interaction Lab.
36
