Affective computing, emotional development, and autism

Full text

CHAPTER
516
39
Introduction
Aective Computing and Child
Development
Children’s development is a fertile application
of aective computing. e nonverbal emotional
communication of children and infants may be less
impacted by social display rules than the commu-
nication of older individuals, thus oering a rich
environment for the automated detection and mod-
eling of emotion. Substantively, early dyadic inter-
action between infants and parents oers a model
for understanding the underpinnings of nonverbal
communication throughout the lifespan. ese
interactions, for example, may lay the basis for the
development of turn-taking and mutual smiling
that are fundamental to later nonverbal communi-
cation (Messinger, Ruvolo, Ekas,& Fogel, 2010).
At the same time, the child’s development aects
the adult he or she will become. Interventions based
in aective computing that help children develop
optimally have the potential to benet society in
the long term. roughout, whenever appropriate,
we discuss how the reviewed studies of detection
and modeling of emotions have contributed to our
understanding of emotional development in chil-
dren with ASD.
Aective Computing and the Development
of Autism Spectrum Disorders
Disordered development can provide insights
into typical development. is chapter discusses
the detection and modeling of emotion—and the
application of interventions grounded in aective
computing—in children with autism spectrum dis-
orders (ASDs) and their high-risk siblings. Autism
spectrum disorders are pervasive disorders of social
Abstract











Key Words: 


and
Aective Computing, Emotional
Development, and Autism
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 516 7/18/2014 1:21:49 PM

Messinger, Duvivier, Warren, Mahoor, Baker, Warlaumont, Ruvolo 517
communication and impact a broad range of non-
verbal (as well as verbal) interactive skills (American
Psychiatric Association, 2000). Because the symp-
toms of these developmental disorders emerge
before 3years of age, ASDs provide a window
into early disturbances of nonverbal social inter-
action. In addition, the younger siblings of chil-
dren with an ASD—high-risk siblings—can oer
a prospective view of the development of ASDs
and related symptoms. Approximately one-fth
of these ASD siblings will develop an ASD and
another fth will exhibit ASD-related symptoms
by 3years of age that are below the threshold for a
clinical diagnosis (Boelte& Poustka, 2003; Bolton,
Pickles, Murphy,& Rutter, 1998; Constantino
etal., 2006; Messinger etal., 2013; Murphy etal.,
2000; Ozono etal., 2011; Szatmari etal., 2000;
Wassink, Brzustowicz, Bartlett,& Szatmari.,
2004). Automated measurement and model-
ing often focuses on high-risk siblings to provide
objective data on the development of ASD-related
symptoms.
Chapter Overview
In a developmental context, aective comput-
ing involves the use of computer software to detect
behavioral signs of emotions and model emotional
functioning and communication and the construc-
tion of software and hardware agents that interact
with children. e chapter begins with a review of
automated measurement of facial action and the
application of those measures to better understand
early emotion expression. Emotional communi-
cation is complex, and the chapter then reviews
time-series and machine-learning approaches to
modeling emotional communication in early inter-
action, which includes comparisons between typi-
cally developing children and children with ASDs.
Next, we review automated approaches to emotion
detection—and to the identication of ASDs—
from children’s vocalizations, and we discuss eorts
to model the vocal signal using graph-based and
time-series approaches. e nal measurement
section reviews new approaches to the collection
of electrophysiological data (electrodermal activa-
tion [EDA]), focusing on eorts in children with
ASD. Finally, we review translational applications
of aective computing in two areas that have shown
promise in helping children with ASD develop
skills in the areas of emotional development and
social communication:embodied conversational
agents (ECAs) and robotics. e chapter ends
with a critical discussion of accomplishments and
opportunities for advancement in aective comput-
ing eorts with children.
Automated Measurement of Emotional
Behavior
Automated Facial Measurement
e face is central to the communication of emo-
tion from infancy through old age. However, man-
ual measurement of facial expression is laborious
and resource-intensive (Cohn& Kanade, 2007).
As a consequence, much more is known about the
perception of facial expressions than of the produc-
tion of facial expressions. Software-based automated
measurement oers the possibility of ecient,
objective portraits of facial expression and emotion
communication. Here, we describe a methodologi-
cal framework for the automated measurement of
facial expression in infants and their parents during
early interaction.
A growing body of research on infant–parent
interaction uses automated measurement based on
the facial action coding system (FACS) (Ekman&
Friesen, 1992; Ekman, Friesen,& Hager, 2002)
and its application to infants (BabyFACS) (Oster,
2006). FACS is a comprehensive manual system for
recording anatomically based appearance changes
in the form of facial action units (AUs; Lucey,
Ashraf,& Cohn, 2007). To better understand the
dynamics of expression and emotional communica-
tion, the strength of key AUs is measured using an
intensity metric that species whether a facial action
its present and, if present, its strength from mini-
mal to maximal using FACS criteria (Mahoor etal.,
2008). Objective measurement of facial expression
intensity allows for time-series modeling of interac-
tive inuence.
A commonly used automated measurement
pipeline combines active appearance and shape
models (AASMs) and support vector machines
(SVMs) (Messinger etal., 2012). Active appear-
ance and shape models are used to detect and track
facial movement (see Figure 39.1). e shape com-
ponent of the AASM unites the two-dimensional
representations of the movement of 66 vertices
(Baker, Matthews,& Schneider, 2004; Cohn&
Kanade, 2007). Mouth opening can be measured
as the vertical distance between the upper and
lower lips in the shape component of the AASM.
e appearance component of the AASM con-
tains the grayscale values for each pixel contained
in the modeled face. Appearance is the grayscale
texture within the region dened by the mesh. In
the research reported here, nonlinear manifold
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 517 7/18/2014 1:21:49 PM

518 Affective Computing, Emotional Development, and Autism
learning (Belkin& Niyogi, 2003) was used to
reduce the dimensionality of the appearance and
shape data to produce a set of variables that are
used to train SVMs. Support vector machines
are machine learning classiers that were used to
determine whether the AU in question was pres-
ent and, if present, its intensity level. To make
this assignment, a one-against-one classication
strategy was used (each intensity level was pitted
against each of the others) (Chang& Lin, 2001;
Mahoor etal., 2008).
Emotion Measurement via
Continuous Ratings
Here, we describe a method for collecting con-
tinuous ratings of emotion constructs in time
that can be modeled in their own right and used
to validate automated measurements of emo-
tional behavior. In the automated facial expres-
sion measurement, expert manual measurement
of facial actions’ levels of cross-system (automated
vs. manual) reliability are typically comparable to
standard interobserver (manual vs. manual) reliabil-
ity. However, intersystem agreement speaks to the
validity of the automated measurements but not to
the emotional meaning of the underlying behaviors.
One approach to validating automated measure-
ments of the face as indices of emotion intensity are
continuous ratings made by third-party observers
(http:// measurement.psy.miami.edu/).
Continuous emotion measurement is similar to
the aect rating dial in which participants in an
emotional experience can provide a continuous
report on their own aective state (Gottman&
Levenson, 1985; Levenson& Gottman, 1983;
Ruef& Levenson, 2007). In the research described
here, however, continuous ratings were made by
observes who moved a joystick to indicate the
aective valence they perceived in an interacting
infant or parent. e ratings of multiple indepen-
dent observers were united into a mean index of
perceived emotional valence (Waldinger, Schulz,
Hauser, Allen,& Crowell, 2004). Continuous
nonexpert ratings have strong face validity because
they reect a precise, easily interpretable descrip-
tion of a construct such as positive (“joy, happi-
ness, and pleasure”) or negative emotion (“anger,
sadness, and distress”).
Applying Automated and Other
Measurement to Early Emotion Expression
THE CASE OF SMILING
Automated measurement of the intensity of
smiling has yielded insights into early positive
emotion. Although infant smiles occur frequently
in social interactions and appear to index positive
emotion, adult smiles occur in a range of contexts,
not all of which are associated with positive emo-
tion. is has led some investigators to propose that
a particular type of smiling, Duchenne smiling, is
uniquely associated with the expression of positive
emotion whereas other smiles do not reect positive
emotion (Ekman& Friesen, 1982). In Duchenne
smiling, the smiling action around the mouth—
produced by zygomaticus major (AU12)—is
Input video
with tracking
Ane
warp
Feature
extraction
(SIFT)
AU
detection
AU 12 (0.00)
Fig.39.1 Facial measurement. From top to bottom:Input video
with overlaid shape model, ane warp to control for orientation
and size, extracted features, and action unit (AU) detection with
respect to support vector machine threshold and ground truth
(manual facial action coding system [FACS] coding).
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 518 7/18/2014 1:21:50 PM

Messinger, Duvivier, Warren, Mahoor, Baker, Warlaumont, Ruvolo 519
complemented by eye constriction produced by
the muscles around the eyes, the orbicularis oculi
and pars orbitalis (AU6). Anatomically, however,
smiling and eye constriction are not yes/no occur-
rences but reect a continuum of muscular activa-
tion (Williams, Warick, Dyson,& Bannister etal.,
1989). Automated measurement of the intensity of
these two actions could indicate whether there is a
continuum of Duchenne smiling.
A CONTINUUM OF DUCHENNE SMILING
Automated measurement of the intensity of smil-
ing and eye constriction indicated that smiling was
a continuous signal (Messinger, Mahoor, Chow,&
Cohn, 2009; Messinger, Mattson, Mahoor,&
Cohn, 2012). Infant smile strength and eye con-
striction intensities were highly correlated and were
moderately associated with degree of mouth open-
ing. Mouth opening is another continuous signal
that frequently occurs with smiling, where it may
index states of high positive arousal such as laugh-
ing. Mothers exhibited similar associations between
smiling and eye constriction intensity, whereas links
to mouth opening were less strong. In essence, there
did not seem to be dierent “types” of smiling—
for example, Duchenne and non-Duchenne—dur-
ing infant–mother interactions (Messinger, Cassel,
Acosta, Ambadar,& Cohn, 2008). Rather, associa-
tions between smiling and eye constrictions revealed
by automated measurement made it more appro-
priate to ask a quantitative question:“How much
Duchenne smiling is being displayed?” or, even
more simply, “How much smiling is present?”
A GRAMMAR OF EARLY FACIAL EXPRESSION
Automated measurements of facial expres-
sions and continuous ratings of aect have yielded
insights into similarities between early positive and
negative emotion. Infants exhibit a tremendous
range of aective expression, from intense smiles
to intense cry-face expressions. e cry-face expres-
sion—and not expressions of discrete negative emo-
tion such as sadness and anger—is the preeminent
index of negative emotion in the infant.
Since Darwin and Duchenne de Boulogne, inves-
tigators have asked how individual facial actions
combine to convey emotional meaning (Darwin,
1872/1998; Duchenne, 1990/1862; Frank,
Ekman,& Friesen, 1993). Darwin, in particular,
suggested that a given facial action—to wit, eye con-
striction—might be associated not only with intense
positive aect but with intense negative aect as
well. Ratings of still photographs suggested that eye
constriction and mouth opening index the intensity
of both positive and negative infant facial expressions
(Bolzani-Dinehart etal., 2005). However, auto-
mated measurements—complemented by continu-
ous ratings of emotion—were required to determine
whether this association was present in dynamically
unfolding, real-time behavior.
Messinger etal. (2012) used automated mea-
surements of infants and parents in the face-to-
face/still-face (FFSF) procedure to examine these
associations. When infants smiled—as noted ear-
lier—the intensity of the smile, the intensity of
eye constriction, and the degree of mouth open-
ing were all associated. In parallel fashion, when
infants engaged in cry-face expressions, the inten-
sity of eye constriction and the degree of mouth
opening were also associated (see Figure 39.2A).
at is, automated measurement revealed simi-
lar signatures of facial intensity in both positive
and negative expressions. In both smile and cry-
face expressions, degree of eye constriction inten-
sity and mouth opening predicted the absolute
intensity of continuously rated emotional valence
(see Figure 39.2B). at is, pairing automated
measurement and continuous ratings indicated
a parsimony in the expression of early negative
and positive emotion that was rst suggested by
Darwin. Automated measurement and continuous
emotional ratings can be used to understand not
only emotional expression but—through model-
ing of interaction—emotional communication.
Modeling Emotional Communication
Here, we review windowed cross-correlations,
advances in time-series modeling, and machine learning
approaches to modeling dyadic emotional communi-
cation. Fundamental questions in infant–parent com-
munication concern the inuence of each partner on
the other. Previous research indicates that the degree to
which parents match the aective states of their infants
predicts subsequent self-control, internalization of
social norms, and cognitive performance (Feldman&
Greenbaum, 1997; Feldman, Greenbaum,& Yirmiya,
1999; Feldman, Greenbaum, Yirmiya,& Mayes,
1996; Kochanska, 2002; Kochanska, Forman,& Coy,
1999; Kochanska& Murray, 2000). Yet it is not clear
that the degree to which one partner responds to the
other—or the degree to which both partners are syn-
chronous with one another—is stable over the course
of several minutes. Both automated measurement and
continuous emotion rating have been used to ascer-
tain the temporal stability of measures of interactive
responsivity.
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 519 7/18/2014 1:21:51 PM

520 Affective Computing, Emotional Development, and Autism
WINDOWED CROSS-CORRELATIONS AND
TIME-VARYING CHANGES IN INTERACTION
Automated measurement of Duchenne smil-
ing intensity illustrated apparent variability in
interactive synchrony in two infant–mother dyads
engaged in face-to-face play (Messinger etal.,
2009). Dierences in interaction existed between
the two dyads and in the microstructure of inter-
action within these segments (see Figure 39.3). At
the dyad level, there were dierences in tempo,
with one dyad’s interactions being faster paced
than the other’s. Within dyads, the microstructure
of coordination was examined using windowed
cross-correlations of sliding 3-second epochs of
interaction (Boker, Rotondo, Xu,& King, 2002).
e midline of the rectangular plot in Figure 39.3
indicates the changing levels of zero-order correla-
tion of Duchenne smiling intensity over time. e
varying associations produced by windowed cross-
correlations of automated measurement indicate
continuous changes in the degree of dyadic syn-
chrony over the course of interaction. is chang-
ing pattern suggests that disruptions and repairs
of emotional synchrony—a potential predictor of
social resiliency—are a common feature of infant–
mother interactions (Schore, 1994; Tronick&
Cohn, 1989).
TIME-SERIES MODELS CHARACTERIZING
DYNAMIC CHANGES IN THE STRENGTH OF
INTERACTION
Descriptions of temporal changes in synchrony
are not a statistical demonstration of time-varying
changes in interaction dynamics. To address this
issue, statistical modeling of time-varying changes
in interactive inuence was carried out using
nonexpert ratings of aective valence (Chow,
Haltigan,& Messinger, 2010). Infants and parents
were observed in the FFSF procedure in order to
present infants with the stressor of parental nonre-
sponsivity. In the FFSF, a naturalistic face-to-face
interaction is disrupted by the still-face, in which
the parent is asked not to initiate or respond to the
infant, and ends with a 3-minute reunion in which
the parent re-engages with the infant (Adamson&
Frick, 2003; Bendersky& Lewis, 1998; Cohn,
Campbell,& Ross, 1991; Delgado, Messinger,&
Yale., 2002; Matias& Cohn, 1993; Tronick,
(a) (b)
Eye constriction
Smile
Cry-face
Smile/Cry-face
Emotion
intensity
ratings
R2 = 0.41****
Mouth opening
Eye constriction
Eye constriction
Mouth opening
Mouth opening
r = 0.42****
r = 0.52****
rp = 0.27****
r = 0.53****
r
p
= 0.24****
r = 0.34***
r
p
= 0.14*
r = 0.30**
r = 0.43****
r
p
= 0.27***
r = 0.29*
r
p
= 0.20*
r = 0.55***
r
p
= 0.46***
r = 0.48***
r
p
= 0.40***
Fig.39.2 (A)e intensity of eye constriction and mouth opening are associated with the intensity of both infant smiles and cry-face
expressions. Overall (r) and partial correlations (rp) between the intensity of smiles, eye constriction, and mouth opening and between
the intensity of cry-faces, eye constriction, and mouth opening. Frames of video in which neither smiles nor cry-faces occurred (zero val-
ues) were randomly divided between the smile and cry-face correlation sets to maintain independence. (B)Eye constriction and mouth
opening intensity predict aective valence (emotion intensity) ratings during both smile and cry-face expressions. R2, r, and rp from
regressing aective valence ratings on the intensity of smile/cry-faces, eye constriction, and mouth opening. All statistics represent mean
values across infants. p values reect two-tailed, one-sample t tests of those values:* p <.05. ** p <.01. ***_p <.001. **** p <.0001.
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 520 7/18/2014 1:21:51 PM

Messinger, Duvivier, Warren, Mahoor, Baker, Warlaumont, Ruvolo 521
Als, Adamson, Wise,& Brazelton, 1978; Yale,
Messinger,& Cobo-Lewis, 2003).
A stochastic regression approach applied in the
context of a time-series analysis allowed the inves-
tigators to test whether interactive inuence itself
changed dynamically over time. ese analyses
address the longstanding problem of nonstationar-
ity in time-series by modeling changes in interac-
tive inuence (Boker etal., 2002; Newtson, 1993).
During face-to-face interaction, and particularly dur-
ing the reunion episode following the still-face per-
turbation, the strength of interactive inuence varied
with time. e nding of changes in the dynamics of
interaction suggests new avenues of research in sta-
tistical modeling of dyadic interaction. Applications
include not only infant–parent interaction, but
dyadic interchanges involving children, adults, and,
potentially, software agents and robots.
MODELING DYNAMICS AMONG ASD SIBLINGS
Bivariate time-series models with random eects
have been used to document ASD-related dierences
in temporal processes (Chow etal., 2010). ese
time-series models incorporated siblings at high risk
for an ASD in order to address potential decits in
emotional expressivity and reciprocal social interac-
tion among these ASD siblings (Baker, Haltigan,
Brewster, Jaccard,& Messinger, 2010; Cassel etal.,
2007; Constantino etal., 2003; Yirmiya etal., 2006).
No risk-related dierences in interactive inuence
were apparent, but dierences in self-regulation
emerged (Chow etal., 2010). Infant siblings of
children with ASDs (ASD-sibs) exhibited higher
levels of self-regulation—indexed by lower values
of autoregression variance parameters—than com-
parison infants. is tendency of ASD-sibs to exhibit
less variability in their self-regulatory dynamics than
comparable control siblings (COMP-sibs) was evi-
dent during the still-face and reunion, suggesting
that ASD-sibs were less emotionally perturbed by the
still-face than were other infants (Chow etal., 2010).
Machine Learning Approaches to
Modeling Dyadic Interaction
Machine learning approaches can be used not
only to measure emotional signals but to model
emotional communication and social interac-
tion more broadly. Machine learning draws on
0.35
0.47 0.42 0.28 0.58
0102030405060708090 100 110 120 130 140 150 160 170 180
Dyad A
Seconds
Dyad B
Facial actions & ticking
0.50 0.36 0.21
1.0
1.0
0
−1.0
0
−1.0
0
1
2
3
4
5
0
1
2
3
4
5
Infant nonsmiling
Tickle
Infant smiling
Mother smiling
Tickle
Infant smiling
Mother smiling
Fig.39.3 Automated measurements of the intensity of infant and mother smiling activity plotted over successive seconds of interaction.
is is Duchenne smiling activity, the mean of smile strength and eye constriction intensity. Correlations between infant and mother
smiling activity are displayed below each segment of interaction. Above each segment of interaction is a plot of the windowed cross-
correlations between infant and mother smiling activity. As seen in the color bar to the right of the plots, high positive correlations are
deep red, null correlations are pale green, and high negative correlations are deep blue. e horizontal midline of these plots indicates
the zero-order correlation between infant and mother smiling activity. e correlations are calculated for successive 3-second segments
of interaction. e plots also indicate the associations of one partner’s current smiling activity with the subsequent activity of the other
partner. Area above the midline indicates the correlation of current infant activity with subsequent mother smiling activity. Area beneath
the midline indicates the reverse. Reprinted from Infancy.
AQ: Please
confirm
this color
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 521 7/18/2014 1:21:52 PM

522 Affective Computing, Emotional Development, and Autism
algorithms and theory from a wide range of disci-
plines including Bayesian statistics, approximation
algorithms, numerical optimization, and stochastic
optimal control, providing a rich toolbox applicable
to the study of interaction and development. At its
core, machine learning is concerned with develop-
ing computational algorithms to learn from data.
Of particular relevance is discovering underlying
structural relationships in interaction and making
predictions about the development of these pat-
terns. Using entropy as a dependent measure, for
example, researchers found that infant behavior
was most predictable (most self-similar over time)
during the still-face episode of the FFSF but least
predictable in the reunion episode, during which
infants may exhibit high levels of both positive
and negative aect (Montirosso, Riccardi, Molteni,
Borgatti,& Reni, 2010).
Researchers have used machine learning meth-
ods to characterize the development of interactive
behavior between mothers and infants both at the
level of weekly sessions and at the level of specic
interactive contexts in a longitudinal dataset cov-
ering the rst 6months of life (Messinger etal.,
2010). e researchers rst asked whether weekly
sessions of infant–mother face-to-face interaction
become more similar to each other—and so more
predictable to each partner—over developmental
time (Messinger etal., 2010). Sessions were char-
acterized with respect to infant, mother, and dyadic
smiling states (e.g., mutual smiling). Similarity met-
rics explored included not only the mean and vari-
ance of these parameters but the entire distribution
of values. Asimilarity metric (the Bhattacharyya
coecient) was computed over a dyad’s consecu-
tive interactive sessions. Over a range of measures,
there were increases with age in the similarity of
models describing consecutive interactions sessions.
is suggests that the consistency—and thus pre-
dictability—of interaction patterns increases with
development. ese ndings suggest the potential
of machine learning for describing how repeated
interactions between infant and parent produce sta-
ble dyadic dierences that contribute to personality
development.
e researchers next focused on those factors
that inuenced the predictability of infant smiling
within specic interactive contexts and asked how
that predictability changed with development (see
Figure 39.4). at is, they predicted the timing of
the infant’s next social action based on the current
state of the interaction. To do so, they built a model
predicting when the infant would initiate or ter-
minate a smile given the current state of the dyad
(whether the infant and the mother were each cur-
rently smiling and which of the partners had smiled
Infant smile transitions
Infant smile initiations
Mother
not smiling
−1
−2
−3
Predictability
−4
−5
123
Mother
smiling
Mother
not smiling
Mother
smiling
Infant smile terminations
123123123
Mother most recent
Infant most recent
Fig.39.4 Predictability (reverse-signed entropy) of infant smiling actions in multiple contexts. Each panel describes the predictability
of a given infant action in a given context (e.g., infant smile initiation while mother is not smiling in the hand panel of the top left graph)
both when the infant acted most recently (infant last) and when the mother acted most recently (mother last). Predictability is described
with respect to infant age categories:4–10 weeks (1–2.5months), 11–17 weeks (2.5–4months), and 18–24 weeks (4.5–6months).
Figure component reprinted from Neural Networks.
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 522 7/18/2014 1:21:53 PM

Messinger, Duvivier, Warren, Mahoor, Baker, Warlaumont, Ruvolo 523
or stopped smiling most recently) and the infant’s
age. e researchers assessed predictability by mea-
suring the entropy of the probability distribution of
the time until the infant’s next action. Entropy is
the inverse of predictability, such that more entropic
distributions are more dicult to predict.
Infant smile initiations become more predictable
(less entropic) with development, whereas infant
smile terminations become less predictable with
age. at is, infant smiling became a more stable
state with development. Both infant smile initia-
tions and terminations were more predictable if the
infant—rather than the mother—had last changed
his or her smiling state. Overall, then, infants were
most predictable when their last action had created
the dyadic conditions in which they were acting.
us, parents who smile to elicit an infant smile
may, paradoxically, lessen the predictability of that
smile occurring. e results point to the potential
of machine learning approaches to produce insights
into real-time emotional communication and devel-
opment, a theme that we next explore with respect
to infant vocalizations.
Automated Measurement of Emotion
inVocalizations
e majority of work on the automated detec-
tion of infant emotion from vocalizations has
focused on infant cries, whereas the detection of
other emotional characteristics of child vocaliza-
tions is less frequent. Infant crying is a ubiquitous
signal of distress that develops into a more varie-
gated expression of negative emotion in the rst
year of life (Gustafson& Green, 1991). Researchers
have distinguished among the communicative
functions of infant cries and other vocalizations
(Fuller, 1991; Petroni, Malowany, Johnston,&
Stevens, 1995). Petroni etal. classied cries as
pain/distress cries or other using a neural network
approach, whereas Fuller (1991) classied cries
as pain-induced, hunger-related, or fussy using
discriminant function analysis. Arobotics group
used low-level auditory features to achieve both
cry detection (Ruvolo& Movellan, 2008) and the
classication of both crying and playing/singing
from ambient sound in a preschool environment
(Ruvolo, Fasel,& Movellan, 2008). More generally,
researchers have used partial least squares regression
to classify child sounds according to child mood and
energy level (Yuditskaya, 2010) and achieved some
success using a least squares minimum distance
classier to distinguish between infant vocalizations
that mothers’ interpreted as more emotional and
more communicative (Papaeliou, Minadakis,&
Cavouras, 2002). Overall, automated identication
and characterization of cries is a more mature area
of research than is classication of other features of
child emotional vocalizations.
AUTOMATED MEASUREMENT
OF VOCALIZATIONS AND ASD
ere is evidence for dierences between the
vocalization of children with ASD, their high-risk
siblings, and the vocalizations of low-risk, typi-
cally developing infants (Paul, Fuerst, Ramsay,
Chawarska,& Klin, 2011; Sheinkopf, Iverson,
Rinaldi,& Lester, 2012; Sheinkopf, Mundy,
Oller,& Steens, 2000). e cries of infant
high-risk ASD siblings tend to have a higher funda-
mental frequency than those of other children, and
it appears that siblings who will go on to an ASD
diagnosis have among the highest pitched cries.
Although automated vocalization research typically
uses samples of relatively short duration, the LENA
system identies child and adult speech characteris-
tics during day-long naturalistic audio recordings.
Oller etal. (2010) used LENA to distinguish among
typically developing children, children with an
ASD, and children with a non-ASD developmental
delay based on acoustic features of their vocaliza-
tions (Oller, Yale,& Delgado, 1997). e LENA
system includes a cry and a laugh detector, although
only the reliability of detection of speech-related
child vocalizations versus non–speech-related vocal-
izations (including cries, laughter, and vegetative
sounds) has been established (Xu, Yapanel,& Gray,
2009). It remains to be seen whether automated
detection of emotional features of vocalization—or
more general acoustic features of vocalizations—
could be used for the prospective classication of
ASD. As in facial measurement, audio measure-
ments have also led to new advances in the model-
ing of emotional signals in the audio domain.
DEVELOPMENTAL PREDICTIONS FROM
MODELED VOCALIZATION
In a seminal longitudinal study, researchers Jae,
Beebe, Feldstein, Crown, and Jasnow (2001) imple-
mented automated measurement of the timing of
infant and adult vocalizations during infant–par-
ent and infant–stranger interactions at 4months
of age (Feldstein etal., 1993). Time-series analy-
sesofinteractive patterns indicated that the quan-
tity of infant vocal interruptions was predicted
by the immediately previous quantity of previous
mother interruptions, a demonstration of what
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 523 7/18/2014 1:21:53 PM

524 Affective Computing, Emotional Development, and Autism
the researchers term coordinated interpersonal tim-
ing. Overall, higher levels of coordinated inter-
personal timing at 4months were associated with
a predilection toward disorganized attachment at
12months, whereas secure attachment was asso-
ciated with mid-range pattern levels of interactive
inuence timing. e results point to curvilinear
patterns in development, which suggests the impor-
tance of nonlinear modeling in understanding vocal
interaction.
Modeling Vocal Interactions with Cross-
Recurrence Quantication Analysis
Cross-recurrence quantication analysis (CRQA)
and recurrence quantication analysis (RQA) are
promising visual approaches to the analysis of inter-
actions. e analyses document patterns within
time-series data that either recur within a single time
series (RQA) or are coordinated across two separate
time series (CRQA). Recurrence quantication anal-
ysis is a recurrence plot in which a single time-series
is represented in a 2-D plot, with time increasing
along both the x-axis and y-axis. In most approaches,
a pixel is lled in when the value of the time-series at
the x-axis time point matches (or comes within some
threshold of similarity to) the value of the time-series
at the y-axis time point. Other pixels are not lled
in. Diagonal lines in the recurrence plot indicate
recurring sequences of values in the time-series
(Webber& Zbilut, 2005). Cross-recurrence quan-
tication analysis begins with a cross-recurrence plot
that compares the values of two time-series—such as
those produced by two conversation partners—with
one time-series being represented along the x-axis
and one time-series being represented along the
y-axis. e cross-recurrence allows for the creation of
a diagonal cross-recurrence prole, which shows the
degree of coordination between the two time-series
at each of a range of lags (Dale, Warlaumont,&
Richardson, 2011).
Although researchers have used RQA and
CRQA to analyze heart rate coordination among
groups of individuals (Konvalinka etal., 2011),
these approaches are typically applied to the analy-
sis of dyadic communication—often in the vocal
modality—and have been used to characterize the
interactions of children with an ASD. Focusing on
mother and infant gaze data during a reunion epi-
sode of a still-face procedure, researchers derived
a “trapping time” metric from the lengths of ver-
tical lines in an RQA plot that indexed the ex-
ibility of gaze interactions between child and
mother (de Graag, Cox, Hasselman, Jansen,&
de Weerth, 2012). Cross-recurrence quantica-
tion analysis can also be applied to mother–infant
acoustic coordination, such as pitch coordination
(Buder, Warlaumont, Oller,& Chorna, 2010).
Warlaumont, Oller, Dale, Richards, Gilkerson,
and Xu (2010) found that there was less vocal
interaction between children with ASD and adults
(reected in the height of the diagonal cross recur-
rence prole) and that, in cross-recurrence plots
across a variety of lags (Warlaumont etal., 2010),
the ratio of child leading to adult following was
smaller in dyads including a child with ASD.
Taken together, this literature suggests that RQA
and CRQA can be usefully applied to the study of
emotional and behavioral coordination dynamics
between children and caregivers and, in some cases,
can reveal dierences between typically developing
children and children with ASD.
Electrodermal Activity, Measurement, and
Applications to ASD
In addition to facial and vocal signals, physi-
ological indices of arousal are key to understand-
ing emotional dynamics in both typically children
and children with developmental disorders such as
autism. Electrodermal activity is measured by skin
conductance and can serve as an index of sympathetic
nervous system arousal. As such, it can provide a rea-
sonable physiologic index of children’s emotional
responses and regulation, providing information on
baseline arousal (tonic EDA), reactions to events
(phasic EDA), and subsequent return to baseline
(recovery or habituation) (Benedek& Kaernbach,
2010; Rogers& Ozono, 2005). In non-ASD sam-
ples, there is evidence that higher EDA may be linked
to more internalizing problems in children, whereas
lower EDA may convey risk for externalizing behav-
iors (El-Sheikh& Erath, 2011). Complicating asso-
ciations between EDA and child outcomes, however,
is evidence that it is involved with and predicted by
interactive eects involving various biological (e.g.,
the long allele of the 5-HTTLPR serotonin genetic
variant) and environmental factors (e.g., harsh par-
enting; El-Sheikh, Keiley, Erath,& Dyer, 2013;
Erath, El-Sheikh, Hinnant,& Cummings, 2011;
Gilissen, Bakermans-Kranenburg, Ijzendoorn,&
Linting, 2008).
ELECTRODERMAL ACTIVATION IN CHILDREN
WITH ASD
e measurement of EDA can provide informa-
tion regarding the form and correlates of individual
dierences in children with ASD. Recent trends
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 524 7/18/2014 1:21:53 PM

Messinger, Duvivier, Warren, Mahoor, Baker, Warlaumont, Ruvolo 525
emphasize the need to understand heterogeneity in
ASD from a social-cognitive perspective (Mundy,
Henderson, Inge,& Coman, 2007), and the same
is true for emotion and its regulation (Mazefsky,
Pelphrey,& Dahl, 2012). Mazefsky and colleagues
have argued cogently for the benets of integrat-
ing traditional autism emotion research with emo-
tion regulation frameworks more widely applied
to normative populations. Such an integration
would require that EDA patterns be tied to chil-
dren’s behavioral responses, emotional expressions,
regulation “strategies,” broader functioning, and/
or to other internal and external correlates (Cole,
Martin,& Dennis, 2004).
As an index of sympathetic nervous system
arousal, EDA has been of longstanding interest
to ASD researchers examining sensory dysfunc-
tion in these children. Despite the increased pres-
ence of sensory-related behaviors in ASD, the
extant literature on sensory dysfunction has not
supported propositions that children with ASD
exhibit atypical general arousal or hyperarousal
reactions, with the little evidence for group dif-
ferences suggesting reduced reactivity to certain
stimuli (Rogers& Ozono, 2005). In reaction,
researchers have proposed that group dierences in
EDA may be obscured by the presence of distinct
subgroups of children with ASD who exhibit pat-
terns of either high or very low arousal (Hirstein,
Iversen,& Ramachandran, 2001; Schoen, Miller,
Brett-Green,& Hepburn, 2008).
Traditional electrodermal measurement tends to
be more dicult for children than for adults due to
diculties with the application and tolerance of the
sensors (Fowles& Fowles, 2007). Moreover, chil-
dren with ASD may have diculties with compre-
hension, high sensory discomfort, and behavioral
noncompliance that represent challenges to the fea-
sibility of traditional EDA measurement. Arecent
development is wireless wearable wrist sensors that
approximate the size and appearance of a watch
(Poh etal., 2012; Poh, Swenson,& Picard, 2010)
and can be worn continuously during naturalistic
laboratory tasks, thus facilitating the integration
of EDA data with behavioral observations of emo-
tion. Apilot study, for example, is currently being
conducted of children with ASD in which the wrist
sensors are used to track arousal across a series of
naturalistic and structured parent–child and child-
alone laboratory tasks (Baker, Fenning, Howland,&
Murakami, 2014). In selected EDA data tasks for
two early participants (see Figure 39.5), one child
0
Wait task Free
play
Clean up Problem
solving
Physical
frustration
Task
Video
frustration
Break
outside
Cognitive
test
ADOS
1
2
3
0
15:41:00
15:43:05
15:45:10
15:47:15
15:49:20
15:51:25
15:53:30
15:55:35
15:57:40
15:59:45
16:01:50
16:03:55
16:06:00
16:08:05
Time
Time
16:10:10
16:12:15
16:14:20
16:16:25
16:18:30
16:20:35
16:22:40
16:24:47
16:26:52
16:28:57
16:31:02
16:33:07
09:42.0
0
0.04
0.08
EDA
0.12
0.16
0.2
11:47.0
13:52.0
15:57.0
18:02.0
20:07.0
22:12.0
24:17.0
26:22.0
28:27.0
30:32.0
32:37.0
34:42.0
36:47.0
38:52.0
40:57.0
43:02.0
45:07.0
47:12.0
49:17.0
51:22.0
53:27.0
55:32.0
57:37.0
1
2
3
4
5
EDA
6
7
8
9
10
EDA in microsiemens
4
5
6
Fig.39.5 Electrodemal activity (EDA) measurements for two children. e large plot visualizes EDA across laboratory tasks, whereas
specic measurements for each child within the Autism Diagnostic Observation Schedule (ADOS) task are inset. Of note, the phasic
peak in EDA for the child in the blue inset occurred when the examiner asked the child about uncomfortable emotions and problematic
peer interactions.
AQ: Please
confirm
this color
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 525 7/18/2014 1:21:54 PM

526 Affective Computing, Emotional Development, and Autism
appears to be exhibiting more typical EDA levels
whereas the prole of the other child appears more
consistent with the underaroused group discussed
in the literature (Hirstein etal., 2001). More gener-
ally, the potential for extended use of such sensors
would allow for measurement of EDA in children
with ASD during completely natural daily activities
in the home, school, and community. Continuously
collected acquisition of EDA measurements in
naturalistic settings has the potential to spur new
research initiatives that parallel similar initiatives in
vocalization research sparked by continuous record-
ing of vocalization data through the LENA system.
Translational Applications of Aective
Computing to Children with ASD
In addition to advances in emotion recognition
and modeling, aective computing approaches can
also be used to model a system’s “emotional response”
to a user and to express emotion via embodied con-
versational agents or robots (Graesser& D’Mello,
2011; Picard, 1997). Children with ASD have spe-
cial challenges in the areas of social communication,
social interaction, and stereotyped behaviors. From
an aective perspective, children with ASD often
have diculty recognizing emotions in others and
sharing enjoyment, interests, or accomplishments,
as well as in interpreting facial cues to decode emo-
tion expression. Many children with ASD also
display a preference for sameness and routines, indi-
cating that the uniform, predictable interactions
oered by translational applications such as embod-
ied conversational agents and robotics may also be
particularly benecial for these children. is sec-
tion reviews recent studies on translational applica-
tions to facilitate the socioemotional development
of children (including children with ASD) through
the use of agents and robots.
Embodied Conversational Agents
Embodied conversational agents are software-
based automata with varying degrees of autonomy
that can be used to assist children in emotional or
other tasks. Agents are represented with a human
audiovisual form whose appearance ranges from
cartoon-like to photographic. Typically, develop-
ing children appear to communicate as much with
an embodied conversational agent as with a human
psychologist using the same script (Black, Flores,
Mower, Narayanan,& Williams., 2010), make sim-
ilar nonverbal gestures with both, and smile more
often and dget less when interacting with an agent
than with a psychologist (Mower, Black, Flores,
Williams,& Narayan, 2011). Agents have primarily
been geared toward improving the academic perfor-
mance of intelligent tutoring systems (ITS) within
typically developing children domains (Graesser,
Chipman, Haynes,& Olney, 2005; Lane, Noren,
Auerbach, Birch,& Swartout, 2011) and tend to
focus on cognitive aspects of learning, to the neglect
of emotional dimensions of learning.
Recent decades have seen increased recognition
of the interplay between emotions and learning
and of the centrality of the role of emotions to
learning (Cicchetti& Sroufe, 1976; Graesser&
D’Mello, 2011; Kort, Reilly,& Picard, 2001).
Findings from the growing literature on emo-
tions and computing suggest that a broader array
of emotions are relevant to learning than those
mentioned in discrete theories of emotion, and
learners often report negative emotions such as
frustration, confusion, and boredom, some of
which facilitate, rather than hinder, deep learn-
ing (Graesser& D’Mello, 2011). Partially as a
result, many ITSs are increasingly incorporating
aect-based agents (e.g., Mao& Li, 2010) in a
range of tutoring systems, including more tradi-
tional academic applications (e.g., Arroyo, Woolf,
Royer,& Tai, 2009). An example is Aective
Auto-Tutor, arguably the rst fully automated,
aect-aware dialogue-based ITS for computer
literacy (D’Mello& Graesser, 2013). is aec-
tive tutoring system was designed to detect stu-
dents’ emotions and use this information to guide
response selection to help children regulate their
emotions during learning (D’Mello& Graesser,
2012). e tutor led to better learning outcomes
than its non–aect-aware equivalent counterpart,
particularly for novice students with low domain
knowledge.
Agent-based intervention systems can also
directly target emotional responsiveness by eliciting
empathy to help the learner practice experiencing
and expressing dierent target emotional states.
FearNot! (Fun with Empathic Agents to Achieve
Novel Outcomes in Teaching) is a prime example
of an agent-based system used to elicit emotion and
teach typically developing children regulation and
coping skills related to bullying prevention (Paiva
etal., 2004). FearNot! taught dierent coping strat-
egies to children using three aect-based agents:a
bully, a victim, and a narrator. Children, for exam-
ple, acted as an invisible friend to the victim agent.
ey watch the victim agent interact with the bully,
have a private conversation with the victim agent
about what happened—where they oer coping
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 526 7/18/2014 1:21:54 PM

Messinger, Duvivier, Warren, Mahoor, Baker, Warlaumont, Ruvolo 527
strategies that the agent might accept or refuse—
and then watch the outcome of the agent’s chosen
coping strategy. FearNot! agents were autonomous,
with a complex architecture guiding their behav-
ioral decisions, including a model of the world rep-
resenting the agent’s own emotions as well as those
of others (based on agent appraisals). Agents had
a parameter-based personality including role-based
(e.g., victim or bully) thresholds for experiencing
dierent emotions, speed of decay for dierent emo-
tions, and a function for recalculating the intensity
of equivalent emotions. Agents also had an action
selection module, which included unplanned action
tendencies based on the agent’s role and personality
(e.g., in the victim role, the agent would cry if bul-
lied, but did not know it would cry). e ecacy of
these empathy-eliciting agents was examined empir-
ically with 52 children aged 8–12 and appeared suc-
cessful:86% of children felt empathy for an agent
(usually the victim), and 72% felt angry (usually
with the bully). FearNot! oers a prime example of a
future direction for using agents to target important
core emotional skills for children that might also be
applied to children with ASD (Paiva etal., 2004).
AGENTS AND CHILDREN WITH ASD
As with typically developing children, embodied
conversational agents can facilitate academic learn-
ing among children with ASD (Bosseler& Massaro,
2003). Increased learning in systems that incorporate
an embodied agent (an animated face) versus disem-
bodied voice-based teaching, for example, have been
found in children with ASD (Massaro& Bosseler,
2006). Agents also have the potential to help chil-
dren with ASD learn to recognize emotions in others
and in themselves. Rachel is an example of pedagogi-
cal emotional coach that collects multimodal data
from children with ASD as they engage in emotion
recognition and emotion storytelling tasks using a
“person-in-the-loop” paradigm in which children
interact with the agent and the system is guided in
real time by a therapist, unbeknownst to the child
(Mower etal., 2011). Support vector machine clas-
sication indicated that children’s speech patterns
were not distinguishable between parent and Rachel,
suggesting that Rachel is able to elicit ecologically
valid interactions from children with ASD in the
context of emotional learning.
Despite these promising eorts, there is sub-
stantial untapped potential in the use of embodied
conversational agent applications for children with
ASD. To facilitate self-recognition and expression
of emotion, systems might detect facial expressions
and physiological signals in children with ASD and
prompt them to report on their emotional experi-
ences by matching their emotional experience to
sample emotional faces. Alternately, posing facial
expressions could be integrated into playing an
ongoing game (see Cockburn etal., 2008). In sum-
mary, the main untapped potential in the use of
agents to help children with ASD arguably rests with
matching emerging technological potential to the
core social decits of children with these disorders.
Robots and Autism
An increase in the presence of social robots
around children appears likely (Movellan, Eckhardt,
Virnes,& Rodriguez, 2009; Tanaka, Cicourel,&
Movellan, 2007), although the potential devel-
opmental eects of interactions with these robots
are only beginning to receive attention in the psy-
chological literature (Kahn, Gary,& Shen, 2013).
Several research groups have studied the response
of children with ASD to both humanoid robots
and nonhumanoid toy-like robots in the hope that
these systems will be useful for understanding aec-
tive, communicative, and social dierences seen in
individuals with ASD and to utilize robotic systems
to develop novel interventions and enhance exist-
ing treatments for children with ASD (see Diehl,
Schmitt, Villano,& Crowell, 2012).
Many individuals with ASD show a preference
for robot-like characteristics over nonrobotic toys
(Dautenhahn& Werry, 2004; Robins, Dautenhahn,
Boekhorst,& Billard, 2005) and, in some circum-
stances, respond faster when cued by robotic move-
ment than by human movement (Bird, Leighton,
Press,& Heyes, 2007; Pierno, Mari, Lusher,&
Castiello, 2008). Although these ndings concern
school-aged children and adults, the preference for
very young children with ASD to orient to non-
social contingencies rather than biological motion
suggests that downward extension of this preference
may be particularly promising (Annaz etal., (2012)
Klin, Lin, Gorrindo, Ramsay,& Jones, 2009).
Furthermore, a number of studies have indicated
the advantages of robotic systems over animated
computer characters for skill learning and optimal
engagement, likely due to the capability of robotic
systems to utilize physical motion in a manner not
possible in screen technologies (Bainbridge, Hart,
Kim,& Scassellati, 2011; Leyzberg, Spaulding,
Toneva,& Scassellati, 2012).
Despite this hypothesized advantage, there
have been relatively few systematic and adequately
controlled applications of robotic technology
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 527 7/18/2014 1:21:55 PM

528 Affective Computing, Emotional Development, and Autism
investigating the impact of directed intervention
and feedback approaches (Duquette, Michaud,&
Mercier, 2008; Feil-Seifer& Matarić, 2009;
Goodrich, Colton, Brinton,& Fujiki, 2011; Kim
etal., 2012). Kim and colleagues (2012) demon-
strated that children with ASD spoke more to an
adult confederate when asked by a robot than when
asked by another adult or by a computer. Duquette
and colleagues (2008) found that children paired
with a robot had greater increases in shared atten-
tion than did those paired with a human. Goodrich
and colleagues reported (2011) that a low-dose
robot-assisted ASD exposure with a humanoid
robot yielded enhanced positive child–human
interactions immediately afterward. Feil-Seifer and
Mataric (2009) showed that when a robot acted
contingently during an interaction with a child
with ASD, it had a positive eect on that child’s
social interaction. Although these approaches have
demonstrated the potential and value of robots for
more directed intervention, the majority of robotic
systems studied to date have been unable to perform
autonomous closed-loop interaction. As such, these
platforms have limited applicability to interven-
tion settings necessitating extended and meaningful
adaptive interactions.
By contrast, examples of adaptive robotic
interaction with children with ASD include
proximity-based closed-loop robotic interaction
(Feil-Seifer& Mataric, 2011), haptic interaction
(Amirabdollahian, Robins, Dautenhahn,& Ji,
2011), and adaptive game interactions based on
aective cues inferred from physiological signals
(Liu, Conn, Sarkar,& Stone, 2008). Although these
systems are capable of adaptive interaction, the par-
adigms explored were focused on simple task and
game performance and had little direct relevance to
the core decits of ASD. Recent work has explicitly
focused on realizing co-robotic interaction architec-
ture capable of measuring behavior and adapting
performance in a way that addresses fundamental
early attentional and aective impairments of ASD
(i.e., joint attention skills). Mazzei etal. (2011)
used a combination of hardware, wearable devices,
and software algorithms to measure the aective
states (e.g., eye gaze attention, facial expressions,
vital signals, skin temperature, and EDA signals) of
children with ASD, and these were used for con-
trolling the robot reactions and responses. Bekele
and colleagues (Bekele, etal., 2013a; Bekele etal.,
2013b) studied the development and application
of a humanoid robotic system capable of intelli-
gently administering joint attention prompts and
adaptively responding based on within-system
measurements of gaze and attention. Preschool
children with ASD directed their gaze more fre-
quently toward the humanoid-robot administrator,
were frequently capable of accurately responding to
robot-administered joint attention prompts, and
also looked away from target stimuli at rates com-
parable to typically developing peers. is suggests
that robotic systems endowed with enhancements
for successfully pushing toward correct orienta-
tion to target might be capable of taking advantage
of baseline enhancements in nonsocial attention
preference in order to meaningfully enhance skills
related to coordinated attention.
For eective ASD intervention, innovative
therapeutic approaches using robot systems should
have the ability to perceive the environment and
users’ behaviors, states, and activities. Increasingly,
researchers are attempting to detect and ex-
ibly respond to individually derived, socially, and
disorder-relevant behavioral cues within intelligent
adaptive robotic paradigms for children with ASD.
Systems capable of such adaptation may ultimately
be utilized to promote meaningful change related to
the complex and important social communication
impairments of the disorder itself. However, ques-
tions regarding generalization of skills remain for
the expanding eld of robotic applications for ASD.
Although many are hopeful that sophisticated clini-
cal applications of adaptive robotic technologies
may demonstrate meaningful improvements for
young children with ASD, it is important to note
that it is both unrealistic and unlikely that such
technology will constitute a sucient intervention
paradigm addressing all areas of impairment for
all individuals with the disorder. However, if sys-
tems are able to discern measurable and modiable
aspects of adaptive robotic intervention with mean-
ingful eects on skills important to neurodevelop-
ment, the eld may realize transformative robotic
technologies with pragmatic real-world application
of import.
Conclusion and Discussion of Alternate
Approaches
Overview
Children potentially oer a relatively simple
model for the application of software-based tools
for the automated measurement and modeling of
emotional behaviors. At the same time, the aec-
tive computing tools implemented in software-
and hardware-based nonhuman agents have the
potential to help children—both with and without
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 528 7/18/2014 1:21:55 PM

Messinger, Duvivier, Warren, Mahoor, Baker, Warlaumont, Ruvolo 529
serious developmental and clinical conditions such
as ASD—confront social and emotional problems
that may impact their development. Here, we pres-
ent a critical summary of key issues in the detection
and modeling of emotional behaviors in andthe
implementation of autonomous software and hard-
ware agents designed to help children.
Facial Expressions
e automated detection of infant and parent
facial expressions—paired with continuous ratings
of emotional valence—has yielded insights into
the continuous ow of emotion expression dur-
ing interaction and suggested parallels between
infant positive and negative emotion expression
(Messinger etal., 2009; Messinger etal., 2012).
To date, however, this research has been conducted
with relatively small sample sizes, and the eciency
promised by automated facial measurement has not
been clearly realized. It is also of note that although
substantial research has been conducted on the
detection of emotion signals in infants younger
than 1year of age, there is relatively little research
on facial expressions of emotion in older children.
Developments that may begin to correct this imbal-
ance include plans for the release of (1)a large data-
base of annotated audio and video measurements of
children between 1 and 2years of age (Rehg etal.,
2013); (2)a multilaboratory repository of audio-
visual data on older children collected in multiple
laboratory settings via the Databrary project (http://
databrary.org/); and (3)the availability of publicly
available databases containing child behavior, such
as YouTube.
Vocalizations and Electrodermal
Activation
e automated detection of cry-vocalizations—
a key signal of infant negative emotion—is rela-
tively robust. However, automated dierentiation
between cries on the basis of apparent communica-
tive intent and the classication of emotional signals
other than cries appears to be a more dicult chal-
lenge. However, the advent of systems for day-long
recording of ambient audio in naturalistic settings
and their automated analysis suggests the tremen-
dous potential of aective computing to under-
stand naturalistic behavior in context. Likewise,
continuous measurement of EDA in extended and
naturalistic conditions oers substantial potential
for understanding the time course of arousal in
response to naturalistic stressors among typically
developing children and children with ASD.
Multimodal Fusion
In the research reviewed, visual and vocal
(audio) signals of emotion were measured sepa-
rately. Recently, however, Rehg and colleagues
fused video-based (e.g., smile and gaze-at-examiner
detection) and audio-based measurements (e.g.,
number and fundamental frequency of child speech
segments) to index child engagement (Rehg etal.,
2013). Although such eorts are rare, the impor-
tance of fusing multimedia measurements—
including physiological as well as visual and audio
sensors—cannot be underestimated. Such fusion
oers the possibility of a better understanding of the
emergence of emotional states from the interplay
of their behavioral and physiological constituents
(Calvo, 2010), as well as a better understanding of
children’s emotional interaction and development.
Modeling Advances
Although not commonly used in the analysis of
automated measurements, there have been wide-
spread advances in the modeling of complex com-
municative systems that are important to aective
computing researchers. Time-series approaches can
now be used to assess the communicative inu-
ence of one partner on another (e.g., parent to
infant inuence) across dyads (Beebe etal., 2007).
Additional progress in time-series modeling has
led to the quantication of time-varying changes
in communicative inuence and group-based
dierences in self-regulation (autocorrelation)
(Chow etal., 2010). At the same time, innovative
approaches based in recurrence quantication anal-
ysis and machine learning approaches that quantify
entropy (the predictability of a given action during
communication) are gaining prominence.
What Modeling Approach Is Most
Appropriate?
Generally, time-series approaches are appropri-
ate when a continuous signal such as the intensity
of a facial action is being modeled. e modeling
of discrete emotional signals (e.g., the presence of
a smile) is well-suited to recurrence quantication
analysis and entropy-based approaches. Descriptive
approaches to modeling, such as windowed
cross-correlations, oer an intuitive description
of emotional communication dynamics whereas
approaches based in time-series analyses oer the
ability to conduct inferential testing of hypotheses.
Despite these rules of thumb, however, there is not
yet consensus on which modeling approach is most
appropriate to understanding a given expressive or
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 529 7/18/2014 1:21:55 PM

530 Affective Computing, Emotional Development, and Autism
communicative system. Projected future growth in
automated measurement (e.g., via Kinect) and the
need to understand and control how software- and
hardware-based agents interact suggests that model-
ing may become a more central aspect of aective
computing initiatives with children in the future.
Modeling to Detect Interaction
In the research reviewed, behavior was measured
and then modeled to detect and understand inter-
action. Rehg and colleagues have demonstrated an
alternate approach that involves directly detecting
interaction structures and dened as quasi-periodic
spatiotemporal patterns (Prabhakar, Oh, Wang,
Abowd,& Rehg, 2010; Prabhakar& Rehg, 2012;
Rehg, 2011). Sequencing video into a string of
visual words, they detected patterns in naturalistic
YouTube videos and used supervised learning to
identify instances of adult–child interaction directly
from those videos. is approach highlights the
potential importance of modeling—broadly con-
strued—in the measurement of interaction.
Modeling to Simulate Development
e modeling approaches reviewed are con-
cerned with characterizing communicative systems.
Additional models that simulate interaction and
development have been implemented by Deák and
collaborators (Deák, Fasel,& Movellan, 2001; Fasel,
Deák, Triesch,& Movellan, 2002; Jasso, Triesch,&
Gedeon, 2008; Lewis, Gedeon,& Triesch, 2010;
Triesch, Teuscher, Deák,& Carlson, 2006). Using
a bottom-up perspective, these researchers posit a
set of infant perceptual preferences, the ability to
learn spatiotemporal contingencies, and a rela-
tively structured environment that is based on the
researchers’ coding of observed infant–parent play
with toys. By assigning variable reward values to
gazes at the parent’s face and toys, the researchers
shed light on the basic abilities required for more
complex developmental processes. Modeled pro-
cesses include following a parent’s gaze (responding
to joint attention) and turning toward a parent’s
face when confronted with an unknown object and
responding to the parent’s positive or negative emo-
tional expression (social referencing). is approach
highlights the potential of modeling to contribute
to an understanding of how development occurs in
both typical and atypical (e.g., ASD) cases.
Software Agents
Initial “person-in-the-loop” systems for children
with ASD have targeted emotional competencies
(e.g., Rachel; Mower etal., 2011). More advanced,
agent-based systems intended for typically devel-
oping children detect and respond to learner’s
emotions in real time in teaching an academic con-
tent area (e.g., Aective Auto-Tutor; D’Mello&
Graesser, 2012). Ideally, future applications for
children with and without ASD would synthesize
these features. ese applications could address core
emotional functioning, including both the identi-
cation and the expression of emotion in dynamic
(e.g., dyadic) contexts as targets, while using detec-
tion and user-modeling approaches to detect emo-
tions such as boredom, confusion, and frustration.
Such a synthetic approach could provide automated,
emotion-based feedback to children with ASD—as
is being done to some degree with typically develop-
ing children—during ongoing interactions.
Robots
In comparison with embodied conversational
agents, relatively more research has been conducted
in which hardware-based agents—robots—have
been used to interact and intervene with children
with ASD (Diehl etal., 2012). Children tend to
respond positively to robots, and they oer poten-
tial for facilitating emotional development in chil-
dren with ASD. As with conversational agents, the
greatest area for future development is likely to be
the development of autonomous closed-loop sys-
tems that apply to social-emotional targets of core
importance to children with ASD. In addition, the
extent to which social-emotional skills acquired and
developed via conversational agents and robots are
generalized to social interaction with other children
and adults is not clear. Finally, the degree to which
agent-based interventions can supplement more
established clinical interventions in real-world set-
tings has yet to be addressed.
Ethics and Outcomes in a Changing
World
In addition to scientic concerns, a recent review
suggests that the projected increase in autonomous
agents such as robots presents complex ethical issues
(Kahn etal., 2013). Children are likely to interact
with technologically “smart” entities such as social
robots as play partners but have ultimate control
over these partners. at is, the reciprocity inher-
ent in social relationships with another child does
not exist with robots which, ultimately, can be
turned o. Although children may benet from
many aspects of these interactions, there is concern
that they may generalize their likely objectication
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 530 7/18/2014 1:21:55 PM

Messinger, Duvivier, Warren, Mahoor, Baker, Warlaumont, Ruvolo 531
of the robots to their interactions with other chil-
dren (Kahn etal., 2013). Finally, parental-sensitive
responsivity is a robust predictor of optimal out-
comes (Belsky& Fearon, 2002; NICHD-ECCRN,
2001). It is of some concern, then, that little is
known about the emotional impact of parent-,
child-, and infant-held personal digital assistants
on children’s outcomes. If the potential of aective
computing is to be used for children’s benet, the
ethical, moral, and developmental impact of both
academic and commercial aective computing tools
require continued investigation.
Acknowledgments
e rst author’s contribution to this chapter was
supported by grants from the National Institutes
of Health (R01HD047417& R01GM105004),
the National Science Foundation (DLS 1052736),
Autism Speaks, and the Marino Autism Research
Institute. e authors thank the families who gen-
erously participated in the research described.
References
Adamson, L. B.,& Frick, J. E. (2003). e still face:Ahistory
of a shared experimental paradigm. Infancy, 4(4), 451-474.
American Psychiatric Association (APA). (2000). Diagnostic
and statistical manual of mental disorders (DSM-IV-TR)
Fourth Edition (Text Revision). Washington, DC:American
Psychiatric Association.
Amirabdollahian, F., Robins, B., Dautenhahn, K.,& Ji, Z.
(2011). Investigating tactile event recognition in child-robot
interaction for use in autism therapy. Conference Proceedings
of IEEE Engineering in Medicine& Biology Society, 2011,
5347–5351. doi:10.1109/iembs.2011.6091323
Annaz, D., Campbell, R., Coleman, M., Milne, E.,&
Swettenham, J. (2012). Young children with autism do
not preferentially attend to biological motion. Journal
of Autism and Developmental Disorders. doi:10.1007/
s10803-011-1256-3
Arroyo, I., Woolf, B. P., Royer, J. M.,& Tai, M. (2009). Aective
Gendered Learning Companions. 14th International confer-
ence on Articial Intelligence and Education (AIED 2009) V.
Dimitrova and R. Mizoguchi (eds), IOS Press..
Bainbridge, W. A., Hart, J. W., Kim, E. S.,& Scassellati, B.
(2011). e benets of interactions with physically present
robots over video-displayed agents. International Journal of
Social Robotics, 3(1), 41–52.
Baker, J. K., Fenning, R. M., Howland, M.,& Murakami, C.
(2014). I second that emotion:Concordance and synchrony
in physiological arousal between children with ASD and their
parents. In A. Esbensen (Chair), Expanding research on family
environment:How, who, and when to measure. Symposium
presented at the 47th Annual Gatlinburg Conference on
Intellectual and Developmental Disabilities. Chicago, IL.
Baker, J. K., Haltigan, J. D., Brewster, R., Jaccard, J.,& Messinger,
D. (2010). Non-expert ratings of infant and parent emo-
tion:Concordance with expert coding and relevance to early
autism risk. International Journal of Behavioral Development,
34(1), 88–95. doi:10.1177/0165025409350365
Baker, S., Matthews, I.,& Schneider, J., (2004). Automatic con-
struction of active appearance models as an image coding
problem. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 26(10), 1380-1384.
Beebe, B., Jae, J., Buck, K., Chen, H., Cohen, P., Blatt,
S., . . . Andrews, H. (2007). Six-week postpartum maternal
self-criticism and dependency and 4-month mother-infant
self- and interactive contingencies. Developmental Psychology,
43(6), 1360–1376.
Bekele, E., Swanson, A., Davidson, J., Sarkar, N.,& Warren,
Z. (2013a). Pilot Clinical Application of an Adaptive
Robotic System for Young Children with Autism. Autism:
International Journal of Research and Practice, DOI:
10.1177/1362361313479454. PMID:24104517, 2013.
Bekele, E. T., Lahiri, U., Swanson, A. R., Crittendon, J. A.,
Warren, Z. E.,& Sarkar, N. (2013b). A step towards devel-
oping Adaptive Robot-Mediated Intervention Architecture
(ARIA) for children with autism. Neural Systems and
Rehabilitation Engineering, IEEE Transactions, 21(2), 289–
299. doi:10.1109/TNSRE.2012.2230188
Belkin, M.,& Niyogi, P. (2003). Laplacian Eigenmaps for
dimensionality reduction and data representation. Neural
Computation Archive, 15(6), 1373–1396.
Belsky, J.,& Fearon, R. M.P. (2002). Early attachment secu-
rity, subsequent maternal sensitivity, and later child devel-
opment:Does continuity in development depend upon
continuity of caregiving? Attachment& Human Development,
4(3), 361–387. doi:10.1080/14616730210167267
Bendersky, M.,& Lewis., M. (1998). Arousal modulation in
cocaine-exposed infants. Developmental Psychology, 34(3),
555-564.
Benedek, M.,& Kaernbach, C. (2010). Decomposition of skin
conductance data by means of nonnegative deconvolution.
Psychophysiology, 47(4), 647–658.
Bird, G., Leighton, J., Press, C.,& Heyes, C. (2007). Intact
automatic imitation of human and robot actions in
autism spectrum disorders. Proceedings of the Royal Society
B:Biological Sciences, 274(1628), 3027–3031. doi:10.1098/
rspb.2007.1019
Black, M. P., Flores, E., Mower, E., Narayanan, S. S.,&
Williams, M. (2010). Comparison of child-human and
child-computer interactions for children with ASD. Paper
presented at the International meeting for autism research
(IMFAR), Philadelphia, PA.
Boelte, S.,& Poustka, F. (2003). e recognition of facial aect
in autistic and schizophrenic subjects and their rst-degree
relatives. Psychological Medicine, 33(5), 907–915.
Boker, S. M., Rotondo, J. L., Xu, M.,& King, K. (2002).
Windowed cross-correlation and peak picking for the analy-
sis of variability in the association between behavioral time
series. Psychological Methods, 7(3), 338–355.
Bolton, P., Pickles, A., Murphy, M.,& Rutter, M. (1998).
Autism, aective and other psychiatric disorders:Patterns
of familial aggregation. Psychological Medicine, 28(Mar),
385–395.
Bolzani-Dinehart, L., Messinger, D. S., Acosta, S., Cassel, T.,
Ambadar, Z.,& Cohn, J. (2005). Adult perceptions of posi-
tive and negative infant emotional expressions. Infancy, 8(3),
279–303.
Bosseler, A.,& Massaro, D. W. (2003). Development and evalu-
ation of a computer-animated tutor for vocabulary and lan-
guage learning in children with autism. Journal of Autism and
Developmental Disorders, 33(6), 653–672.
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 531 7/18/2014 1:21:55 PM

532 Affective Computing, Emotional Development, and Autism
Buder, E. H., Warlaumont, A. S., Oller, D. K.,& Chorna, L.
B. (2010). Dynamic indicators of mother-infant prosodic
and illocutionary coordination. In the Proceedings of Speech
Prosody 2010.
Calvo, R. A. (2010). Latent and emergent models in aec-
tive computing. Emotion Review, 2(3), 288–289.
doi:10.1177/1754073910368735
Cassel, T. D., Messinger, D. S., Ibanez, L. V., Haltigan, J. D.,
Acosta, S. I.,& Buchaman, A. C. (2007). Early social and
emotional communication in the infant siblings of chil-
dren with autism spectrum disorders:An examination of
the broad phenotype. Journal of Autism and Developmental
Disorders, 37, 122–132.
Chang, C.-C.,& Lin, C.-J., (2001). LIBSVM:Alibrary for sup-
port vector machines, 2001. Software available at http://www.
csie.ntu.edu.tw/~cjlin/libsvm.
Chow, S., Haltigan, J. D.,& Messinger, D. S. (2010). Dynamic
infant-parent aect coupling during the Face-to-Face/
Still-Face. Emotion, 10, 101–114.
Cicchetti, D.,& Sroufe, L. A. (1976). e relationship between
aective and cognitive development in Down’s syndrome
infants. Child Development, 47(4), 920–929.
Cockburn, J., Bartlett, M., Tanaka, J. Movellan, J.,& Schultz,
R. (2008). SmileMaze:Atutoring system in real-time facial
expression perception and production in children with
Autism Spectrum Disorder, Proceedings from the IEEE
International Conference on Automatic Face& Gesture
Recognition (peer-reviewed conference proceeding),
978-986..
Cohn, J., Campbell, S. B.,& Ross, S. (1991). Infant response
in the still-face paradigm at 6months predicts avoidant
and secure attachment at 12months. Development and
Psychopathology, 3(4), 367-376.
Cohn, J.,& Kanade, T. (2007). Automated facial image analysis
for measurement of emotion expression. In J. A.Coan& J.
B. Allen (Eds.), e handbook of emotion elicitation and assess-
ment (pp. 222–238). NewYork:Oxford.
Cole, P. M., Martin, S. E.,& Dennis, T. A. (2004).
Emotion regulation as a scientic construct:method-
ological challenges and directions for child develop-
ment research. Child Development, 75(2), 317-333.
doi:10.1111/j.1467-8624.2004.00673.x
Constantino, DavisTodd, Schindler, Gross, Brophy, S.L., . . . Reich,
W. (2003). Validation of a brief quantitative measure of
autistic traits:Comparison of the Social Responsiveness Scale
with the Autism Diagnostic Interview-Revised. Journal-of-
Autism-and-Developmental-Disorders, 33(4), 427–433.
Constantino, J., Lajonchere, C., Lutz, M., Gray, T., Abbacchi,
A., McKenna, K., . . . Todd, R. (2006). Autistic social impair-
ment in the siblings of children with pervasive developmental
disorders. American Journal of Psychiatry, 163(2), 294–296.
D’Mello, S. K.& Graesser, A. C. (2012). Malleability of
Students’ Perceptions of an Aect-Sensitive Tutor and its
Inuence on Learning. In G. Youngblood& P. McCarthy
(Eds.) Proceedings of 25th Florida Articial Intelligence
Research Society Conference (pp.432-437). Menlo Park,
CA:AAAI Press.
D’Mello, S. K.,& Graesser, A. (2013). AutoTutor and aec-
tive AutoTutor:Learning by talking with cognitively and
emotionally intelligent computers that talk back. ACM
Transactions in Interactive. Intelligence System, 2(4), 1–39.
doi:10.1145/2395123.2395128
Dale, R., Warlaumont, A. S.,& Richardson, D. C. (2011).
Nominal cross recurrence as a generalized lag sequential
analysis for behavioral streams. International Journal of
Bifurcation and Chaos, 21(4), 1153–1161. doi:10.1142/
s0218127411028970
Darwin, C. (1872/1998). e expression of the emotions in man
and animals (3rd edition). NewYork:Oxford University.
Dautenhahn, K.,& Werry, I. (2004). Towards interactive
robots in autism therapy:Background, motivation and chal-
lenges. Pragmatics& Cognition, 12(1), 1–35. doi:10.1075/
pc.12.1.03dau
de Graag, J. A., Cox, R. F.A., Hasselman, F., Jansen, J.,&
de Weerth, C. (2012). Functioning within a relation-
ship:Mother–infant synchrony and infant sleep. Infant
Behavior and Development, 35(2), 252–263. doi:10.1016/j.
infbeh.2011.12.006
Deák, G. O., Fasel, I.,& Movellan, J. (2001). e emergence
of shared attention:Using robots to test developmental
theories. Proceedings 1st International Workshop on Epigenetic
Robotics:Lund University Cognitive Studies, 85, 95–104.
Delgado, C. E.F., Messinger, D. S.,& Yale, M. E. (2002). Infant
responses to direction of parental gaze:Acomparison of two
still-face conditions. Infant Behavior& Development, 25(3),
311-318.
Diehl, J. J., Schmitt, L. M., Villano, M.,& Crowell, C. R.
(2012). e clinical use of robots for individuals with autism
spectrum disorders:Acritical review. Research in Autism
Spectrum Disorders, 6(1), 249–262.
Duchenne, G. B. (1990/1862). e mechanism of human facial
expression (R. A. Cuthbertson, Trans.). NewYork:Cambridge
University Press.
Duquette, A., Michaud, F.,& Mercier, H. (2008). Exploring
the use of a mobile robot as an imitation agent with chil-
dren with low-functioning autism. Autonomic Robots, 24(2),
147–157. doi:10.1007/s10514-007-9056-5
Ekman, P.,& Friesen, W. (1992). Changes in FACS Scoring
(Instruction Manual). San Francisco, CA:Human
Interaction Lab.
Ekman, P., Friesen, W. V.,& Hager, J. C. (2002). Facial Action
Coding System Investigator's Guide. Salt Lake City, UT, A
Human Face.
Ekman, P.,& Friesen, W. V. (1982). Felt, false, and miserable
smiles. Journal of Nonverbal Behavior, 6(4), 238–252.
El-Sheikh, M.,& Erath, S. A. (2011). Family conict, auto-
nomic nervous system functioning, and child adapta-
tion:State of the science and future directions. Development
and Psychopathology, 23(2), 703-721. doi:10.1017/
S0954579411000034
El-Sheikh, M., Keiley, M., Erath, S.,& Dyer, W. J. (2013).
Marital conict and growth in children’s internalizing
symptoms:e role of autonomic nervous system activity.
Developmental Psychology, 49(1), 92–108. doi:10.1037/
a0027703
Erath, S. A., El-Sheikh, M., Hinnant, J. B.,& Cummings, E. M.
(2011). Skin conductance level reactivity moderates the asso-
ciation between harsh parenting and growth in child exter-
nalizing behavior. Developmental Psychology, 47(3), 693–706.
doi:10.1037/a0021909
Fasel, I., Deák, G. O., Triesch, J.,& Movellan, J. (2002).
Combining embodied models and empirical research
for understanding the development of shared attention.
Paper presented at the Development and Learning, 2002.
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 532 7/18/2014 1:21:56 PM

Messinger, Duvivier, Warren, Mahoor, Baker, Warlaumont, Ruvolo 533
Proceedings. e 2nd International Conference on Development
and Learning, 2, 21–27
Feil-Seifer, D.,& Mataric, M. (2011). Automated detection and
classication of positive vs. negative robot interactions with chil-
dren with autism using distance-based features. Paper presented
at the Proceedings of the 6th international conference on
Human-robot interaction, Lausanne, Switzerland.
Feil-Seifer, D.,& Matarić, M. (2009). Toward socially assistive
robotics for augmenting interventions for children with
autism spectrum disorders. In O. Khatib, V. Kumar,& G.
Pappas (Eds.), Experimental robotics (vol. 54, pp. 201–210).
Berlin Heidelberg:Springer.
Feldman, R.,& Greenbaum, C. W. (1997). Aect regulation
and synchrony in mother-infant play as precursors to the
development of symbolic competence. Infant Mental Health
Journal, 18(1), 4–23.
Feldman, R., Greenbaum, C. W.,& Yirmiya, N. (1999).
Mother-infant aect synchrony as an antecedent of the
emergence of self-control. Developmental Psychology, 35(1),
223–231.
Feldman, R., Greenbaum, C. W., Yirmiya, N.,& Mayes, L.
C. (1996). Relations between cyclicity and regulation in
mother-infant interaction at 3 and 9months and cognition
at 2years. Journal of Applied Developmental Psychology, 17(3),
347–365. doi:http://www.sciencedirect.com/science/article/
pii/S0193397396900313
Feldstein, S., Jae, J., Beebe, B., Crown, C. L., Jasnow, L., Fox,
H.,& Gordon, S. (1993). Coordinated interpersonal tim-
ing in adult-infant vocal interactions:Across-site replication.
Infant Behavior& Development, 16, 455–470.
Fowles, D. (2008) e Measurement of Electrodermal Activity
in Children. In Louis A.Schmidt& Sidney J.Segalowitz,
Developmental Psychophysiology:eory, systems, and method,s
(pp.286-316). NewYork:Cambridge University Press.
Frank, M. G., Ekman, P.,& Friesen, W. V. (1993). Behavioral
markers and the recognizability of the smile of enjoyment.
Journal of Personality and Social Psychology, 64(1), 83–93.
Fuller, B. F. (1991). Acoustic discrimination of three types of
infant cries. Nursing Research, 40(3), 156–160.
Gilissen, R., Bakermans-Kranenburg, M. J., Ijzendoorn, M.
H.v.,& Linting, M. (2008). Electrodermal reactivity during
the Trier Social Stress Test for Children:Interaction between
the serotonin transporter polymorphism and children’s
attachment representation. Developmental Psychobiology,
50(6), 615–625. doi:10.1002/dev.20314
Goodrich, M. A., Colton, M. A., Brinton, B.,& Fujiki, M.
(2011). A case for low-dose robotics in autism therapy. Paper
presented at the Proceedings of the 6th international confer-
ence on Human-robot interaction, Lausanne, Switzerland.
Gottman, J.,& Levenson, R. W. (1985). A valid measure for
obtaining self-report of aect. Journal of Consulting and
Clinical Psychology, 53, 151–160.
Graesser, A., Chipman, P., Haynes, B. C.,& Olney, A.
(2005). AutoTutor:An intelligent tutoring system with
mixed-initiative dialogue. IEEE Transactions on Education,
48(4), 612–618. doi:Citeulike-article-id:9781567;
doi:10.1109/TE.2005.856149
Graesser, A.,& D’Mello, S. K. (2011). eoretical perspectives
on aect and deep learning. In R. A. Calvo& S. K. D’Mello
(Eds.), New perspectives on aect and learning technologies
(vol. 3, pp. 11–21). NewYork:Springer.
Gustafson, G. E.,& Green, J. A. (1991). Developmental coordi-
nation of cry sounds with visual regard and gestures. Infant
Behavior& Development, 14(1), 51–57. doi:10.1016/0
163-6383(91)90054-V
Hirstein, W., Iversen, P.,& Ramachandran, V. S. (2001).
Autonomic responses of autistic children to people and
objects. Proceedings of the Royal Society:B Biological Sciences,
268(1479), 1883–1888.
Jae, J., Beebe, B., Feldstein, S., Crown, C. L.,& Jasnow, M. D.
(2001). Rhythms of dialogue in infancy:Coordinated tim-
ing in development. Monographs of the Society for Research in
Child Development, 66(2), vi-131.
Jasso, H., Triesch, J.,& Gedeon, D. (2008, 9–12 Aug. 2008). A
reinforcement learning model of social referencing. Paper pre-
sented at the Development and Learning, 2008. ICDL 2008.
7th IEEE International Conference on.
Kahn, P. H., Gary, H. E.,& Shen, S. (2013). Children’s social
relationships with current and near-future robots. Child
Development Perspectives, 7(1), 32–37. doi:10.1111/
cdep.12011
Kim, E. S., Berkovits, L. D., Bernier, E. P., Leyzberg, D., Shic,
F., Paul, R.,& Scassellati, B. (2012). Social robots as embed-
ded reinforcers of social behavior in children with autism.
Journal of Autism and Developmental Disorders. doi:10.1007/
s10803-012-1645-2
Klin, A., Lin, D. J., Gorrindo, P., Ramsay, G.,& Jones, W.
(2009). Two-year-olds with autism orient to non-social con-
tingencies rather than biological motion. Nature, 459(7244),
257–261. doi:10.1038/nature07868
Kochanska, G. (2002). Mutually responsive orientation
between mothers and their young children:Acontext for
the early development of conscience. Current Directions in
Psychological Science, 11(6), 191–195.
Kochanska, G., Forman, D. R.,& Coy, K. C. (1999).
Implications of the mother-child relationship in infancy
socialization in the second year of life. Infant Behavior&
Development, 22(2), 249–265.
Kochanska, G.,& Murray, K. T. (2000). Mother-child mutually
responsive orientation and conscience development:From
toddler to early school age. Child Development, 71(2),
417–431.
Konvalinka, I., Xygalatas, D., Bulbulia, J., Schjodt, U.,
Jegindo, E. M., Wallot, S., . . . Roepstor, A. (2011).
Synchronized arousal between performers and related spec-
tators in a re-walking ritual. Proceedings of the National
Academy of Sciences, 108(20), 8514–8519. doi:10.1073/
pnas.1016955108
Kort, B., Reilly, R.,& Picard, R. W. (2001). An aective model
of interplay between emotions and learning:Reengineering
educational pedagogy—Building a learning companion. Paper
presented at the International Conference on Advanced
Learning Technologies, Madison, USA.
Lane, H. C., Noren, D., Auerbach, D., Birch, M.,& Swartout,
W. (2011). Intelligent tutoring goes to the museum in the big
city:Apedagogical agent for informal science education. Paper
presented at the Proceedings of the 15th international con-
ference on Articial intelligence in education, Auckland,
New Zealand.
Levenson, R. W.,& Gottman, J. M. (1983). Marital interac-
tion:Physiological linkage and aective exchange. Journal of
Personality& Social Psychology, 45, 587–597.
Lewis, J. M., Gedeon, D.,& Triesch, J. (2010). Building a model
of infant social interaction. Paper presented at the Proceedings
of the 32nd Annual Conference of the Cognitive Science
Society, Austin, TX.
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 533 7/18/2014 1:21:56 PM

534 Affective Computing, Emotional Development, and Autism
Leyzberg, S., Spaulding, M. Toneva, B.,& Scassellati, B. (2012).
e physical presence of a robot tutor increases cognitive learning
gains. Paper presented at the Proceedings of the 34th Annual
Conference of the Cognitive Science Society (cogSci 2012),
1882-1887. Austin, TX|Saporro, Japan:Cognitive Science
Society. Austin, TX|Saporro, Japan, August 1-4
Liu, C., Conn, K., Sarkar, N.,& Stone, W. (2008). Online aect
detection and robot behavior adaptation for intervention of
children with autism. IEEE Transactions on Robotics, 24(4),
883-896. doi:10.1109/tro.2008.2001362
Lucey, S., Ashraf, A. B.,& Cohn, J. (2007). Investigating spon-
taneous facial action recognition through AAM represen-
tations of the face. In K. Kurihara (Ed.), Face recognition.
Mammendorf, Germany:Pro Literatur Verlag.
Mahoor, M. H., Messinger, D. S., Ibanez, L., Kimijima M.,
Wang, Y., Cadavid, S.,& Cohn, J. F. (2008). Studying
facial expressions using manifold learning and support vector
machines. Paper presented at the IEEE 7th International
Conference on Development and Learning, Monterey,
CA.
Mao, X.,& Li, Z. (2010). Agent based aective tutoring sys-
tems:Apilot study. Computers and Education, 55(1), 202–
208. doi:10.1016/j.compedu.2010.01.005
Massaro, D. W.,& Bosseler, A. (2006). Read my lips:e
importance of the face in a computer-animated tutor for
vocabulary learning by children with autism. Autism, 10(5),
495–510. doi:10.1177/1362361306066599
Matias, R.,& Cohn, J. F. (1993). Are MAX-specied infant
facial expression during the face-to-face interaction con-
sistent with dierential emotions theory? Developmental
Psychology, 29(3), 524-531.
Mazefsky, C. A., Pelphrey, K. A.,& Dahl, R. E. (2012). e
need for a broader approach to emotion regulation research
in autism. Child Development Perspectives, 6(1), 92–97. doi:
10.1111/j.1750-8606.2011.00229.x
Mazzei, D., Lazzeri, N., Billeci, L., Igliozzi, R., Mancini, A.,
Ahluwalia, A., . . . De Rossi, D. (2011). Development
and evaluation of a social robot platform for therapy in
autism. Conference Proceedings of IEEE Engineering in
Medicine& Biology Society, 2011, 4515–4518. doi:10.1109/
iembs.2011.6091119
Messinger, D., Cassel, T., Acosta, S., Ambadar, Z.,& Cohn, J.
(2008). Infant smiling dynamics and perceived positive emo-
tion. Journal of Nonverbal Behavior, 32, 133–155.
Messinger, D., Mahoor, M., Chow, S.,& Cohn, J. F.
(2009). Automated measurement of facial expression in
infant-mother interaction:Apilot study. Infancy, 14, 285–
305. NIHMS99269.
Messinger, D., Mahoor, M., Chow, S., Haltigan, J. D.,
Cadavid, S.,& Cohn, J. F.. (2014). Early Emotional
Communication:Novel Approaches to Interaction. Social
emotions in nature and artifact:Emotions in human and
human-computer interaction. J. Gratch and S. Marsella,
Oxford University Press, USA. 14:162-180.
Messinger, D., Ruvolo, P., Ekas, N.,& Fogel, A. (2010). Applying
machine learning to infant interaction:e development is
in the details. Neural Networks, 23(10), 1004–1016.
Messinger, D., Young, G. S., Ozono, S., Dobkins, K.,
Carter, A., Zwaigenbaum, L., . . . Sigman, M. (2013).
Beyond autism:Ababy siblings research consortium study
of high-risk children at three years of age. Journal of the
American Academy of Child and Adolescent Psychiatry, 52(3),
300–308 e301. doi:10.1016/j.jaac.2012.12.011
Messinger, D. S., Mattson, W. I., Mahoor, M. H.,& Cohn, J. F.
(2012). e eyes have it:making positive expressions more
positive and negative expressions more negative. Emotion.
12(3):430-436.
Montirosso, R., Riccardi, B., Molteni, E., Borgatti, R.,&
Reni, G. (2010). Infant’s emotional variability associated
to interactive stressful situation:Anovel analysis approach
with Sample Entropy and Lempel–Ziv Complexity. Infant
Behavior and Development, 33(3), 346–356. doi:http://
dx.doi.org/10.1016/j.infbeh.2010.04.007
Movellan, J. R., Eckhardt, M., Virnes, M.,& Rodriguez, A.
(2009). Sociable robot improves toddler vocabulary skills. In
Proceedings of the 4th ACM/IEEE International Conference on
Human-Robot Interaction (HRI 2009)(pp.307-308, http://
dx.doi.org/10.1145/1514095.1514189). La Jolla, CA,
USA:ACM/IEEE.
Mower, E., Black, M. P., Flores, E., Williams, M.,&
Narayanan, S. (2011). Rachel:Design of an emotionally tar-
geted interactive agent for children with autism. Paper pre-
sented at the Proceedings of the 2011 IEEE International
Conference on Multimedia and Expo (ICME) Barcelona,
Spain.
Mundy, P. C., Henderson, H. A., Inge, A. P.,& Coman, D. C.
(2007). e modier model of autism and social develop-
ment in higher functioning children with autism. Research&
Practice for Persons with Severe Disabilities, 32(2), 1–16.
Murphy, M., Bolton, P. F., Pickles, A., Fombonne, E., Piven,
J.,& Rutter, M. (2000). Personality traits of the rela-
tives of autistic probands. Psychological Medicine, 30(6),
1411–1424.
Newtson, D. (1993). e dynamics of action and interaction.
Adynamic systems approach to development:Applications.
Anonymous. Cambridge, MA, MIT Press:241-264.
NICHD-ECCRN. (2001). Child-care and family predic-
tors of preschool attachment and stability from infancy.
Developmental Psychology, 37(6), 847–862.
Oller, D. K., Niyogi, P., Gray, S., Richards, J. A., Gilkerson, J.,
Xu, D., . . . Warren, S. F. (2010). Automated vocal analysis of
naturalistic recordings from children with autism, language
delay, and typical development. Proceedings of the National
Academy of Sciences of the United States of America, 107(30),
13354–13359. doi:10.1073/pnas.1003882107
Oller, D. K., Yale, M. E.,& Delgado, R. E. (1997). Development
of coordination across modalities of communication:Coding and
analysis tools. Paper presented at the Biennial Meeting of the
Society for Research in Child Development, Washington, D.
C.
Oster, H. (2006). Baby FACS:Facial Action Coding System for
infants and young children. Unpublished monograph and
coding manual. NewYork University.
Ozono, S., Young, G. S., Carter, A., Messinger, D., Yirmiya,
N., Zwaigenbaum, L., . . . Stone, W. L. (2011). Recurrence
risk for autism spectrum disorders:ABaby Siblings Research
Consortium study. Pediatrics, 128(3), 15.
Paiva, A., Dias, J., Sobral, D., Aylett, R., Sobreperez, P., Woods,
S., . . . Hall, L. (2004). Caring for agents and agents that
care:Building empathic relations with synthetic agents. Paper
presented at the Proceedings of the ird International
Joint Conference on Autonomous Agents and Multiagent
Systems—Volume 1, NewYork.
Papaeliou, C., Minadakis, G.,& Cavouras, D. (2002). Acoustic
patterns of infant vocalizations expressing emotions and
communicative functions. Journal of Speech Language and
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 534 7/18/2014 1:21:56 PM

Messinger, Duvivier, Warren, Mahoor, Baker, Warlaumont, Ruvolo 535
Hearing Research, 45(2), 311–317. doi:10.1044/1092-438
8(2002/024)
Paul, R., Fuerst, Y., Ramsay, G., Chawarska, K.,& Klin, A.
(2011). Out of the mouths of babes:Vocal production
in infant siblings of children with ASD. Journal of Child
Psychology and Psychiatry, 52(5), 588–598. doi:10.1111/j.1
469-7610.2010.02332.x
Petroni, M., Malowany, A. S., Johnston, C. C.,& Stevens,
B. J. (1995). Classication of infant cry vocaliza-
tions using articial neural networks (ANNs). 1995
International Conference on Acoustics, Speech, and Signal
Processing (ICASSP-95), 5, 3475–3478. doi:10.1109/
ICASSP.1995.479734
Picard, R. (1997). Aective computing. Cambridge:MIT Press.
Pierno, A. C., Mari, M., Lusher, D.,& Castiello, U. (2008).
Robotic movement elicits visuomotor priming in chil-
dren with autism. Neuropsychologia, 46(2), 448–454.
doi:10.1016/j.neuropsychologia.2007.08.020
Poh, M.-Z., Loddenkemper, T., Reinsberger, C., Swenson,
N. C., Goyal, S., Sabtala, M. C., . . . Picard, R. W. (2012).
Convulsive seizure detection using a wrist-worn electroder-
mal activity and accelerometry biosensor. Epilepsia, 53(5),
e93–e97. doi:10.1111/j.1528-1167.2012.03444.x
Poh, M. Z., Swenson, N. C.,& Picard, R. W. (2010). A wear-
able sensor for unobtrusive, long-term assessment of electro-
dermal activity. IEEE Transactions Biomedical Engineering,
57(5), 1243–1252. doi:10.1109/tbme.2009.2038487
Prabhakar, K., Oh, S., Wang, P. , Abowd, G. D.,& Rehg, J. M.
(2010). Tem p o r a l c a u s a l i t y f o r t h e a n a l y s i s o f v i s u a l e v e n t s . Paper
presented at the IEEE Conference on Computer Vision and
Patt ern Recog nitio n (C VPR), pp. 1967-1974., Barcelona,
Spain
Prabhakar, K.,& Rehg, J. (2012). Categorizing turn-taking
interactions. In A. Fitzgibbon, S. Lazebnik, P. Perona, Y.
Sato,& C. Schmid (Eds.), Computer vision—ECCV 2012
(vol. 7576, pp. 383–396). Berlin/Heidelberg:Springer.
Rehg, J. M. (2011). Behavior imaging:Using computer vision to
study autism. Paper presented at the MVA.IAPR Conference on
Machine Vision Applications, June 13-15, 2011, Nara, Japan.
Rehg, J. M., Abowd, G. D., Rozga, A., Romero, M., Clements, M.
A., Sclaro, S., . . . Ye , Z. ( 2 01 3) . Decoding children’s social behav-
ior. Paper presented at the IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), Portland, OR.
Robins, B., Dautenhahn, K., Boekhorst, T.,& Billard, A. (2005).
Robotic assistants in therapy and education of children with
autism:Can a small humanoid robot help encourage social
interaction skills? Universal Access in the Information Society,
4(2), 105–120. doi:10.1007/s10209-005-0116-3
Rogers, S. J.,& Ozono, S. (2005). Annotation:What do we
know about sensory dysfunction in autism? Acritical review
of the empirical evidence. Journal of Child Psychology and
Psychiatry, 46(12), 1255–1268. doi:10.1111/j.1469-7610.
2005.01431.x
Ruef, A.,& Levenson, R. (2007). Studying the time course of
aective episodes using the aect rating dial. In J. A. Coan&
J. J.B. Allen (Eds.), e handbook of emotion elicitation and
assessment.
Ruvolo, P., Fasel, I.,& Movellan, J. (2008). Auditory mood detec-
tion for social and educational robots. IEEE International
Conference on Robotics and Automation, 2008, 3551–3556.
doi:10.1109/ROBOT.2008.4543754
Ruvolo, P.,& Movellan, J. (2008). Automatic cry detection
in early childhood education settings. 7th International
Conference on Development and Learning (ICDL 2008), 204–
208. doi:10.1109/DEVLRN.2008.4640830
Schoen, S. A., Miller, L., Brett-Green, B.,& Hepburn, S. L.
(2008). Psychophysiology of children with autism spectrum
disorder. Research in Autism Spectrum Disorders, 2(3), 417–
429. doi:10.1016/j.rasd.2007.09.002
Schore, A. N. (1994). Aect regulation& the origin of self:e neu-
robiology of emotional development. Hillsdale, NJ:Erlbaum.
Sheinkopf, S. J., Iverson, J. M., Rinaldi, M. L.,& Lester, B. M.
(2012). Atypical cry acoustics in 6-month-old infants at risk
for autism spectrum disorder. Autism Research, 5(5), 331–
339. doi:10.1002/aur.1244
Sheinkopf, S. J., Mundy, P., Oller, D. K.,& Steens, M.
(2000). Vocal atypicalities of preverbal autistic children.
Journal of Autism and Developmental Disorders, 30(4),
345–354.
Szatmari, P., MacLean, J. E., Jones, M. B., Bryson, S. E.,
Zwaigenbaum, L., Bartolucci, G., . . . Tu, L. (2000). e
familial aggregation of the lesser variant in biological and
nonbiological relatives of PDD probands:Afamily history
study. Journal of Child Psychology and Psychiatry, 41(5),
579–586.
Tanaka, F., Cicourel, A.,& Movellan, J. R. (2007). Socialization
between toddlers and robots at an early childhood educa-
tion center. Proceedings of the National Academy of Sciences,
104(46), 17954–17958. doi:10.1073/pnas.0707769104
Triesch, J., Teuscher, C., Deák, G. O.,& Carlson, E. (2006).
Gaze following:Why (not) learn it? Developmental Science,
9(2), 125–147.
Tronick, E. Z., Als, H., Adamson, L., Wise, S., Brazelton, B.
(1978). e infant’s response to entrapment between con-
tradictory messages in face-to-face interation. American
Academy of Child Psychiatry, 17(1), 1-13.
Tronick, E. Z.,& Cohn, J. F. (1989). Infant-mother face-to-
face interaction:Age and gender dierences in coordination
and the occurrence of miscoordination. Child Development,
60(1), 85–92.
Waldinger, R. J., Schulz, M. S., Hauser, S. T., Allen, J. P.,&
Crowell, J. A. (2004). Reading others’ emotions:e role of
intuitive judgments in predicting marital satisfaction, qual-
ity, and stability. Journal of Family Psychology, 18, 58–71.
Warlaumont, A. S., Oller, D. K., Dale, R., Richards, J. A.,
Gilkerson, J.,& Xu, D. (2010). Vocal interaction dynamics
of children with and without autism. In S. Ohlsson& R.
Catrambone (Eds.), Proceedings of the 32nd Annual Conference
of the Cognitive Science Society. Austin, TX:Cognitive Science
Society, 121-126.
Wassink, T. H., Brzustowicz, L. M., Bartlett, C. W.,& Szatmari,
P. (2004). e search for autism disease genes. Mental
Retardation and Developmental Disabilities Research Reviews,
10(4), 272–283.
Webber, C. L., Jr.,& Zbilut, J. P. (2005). Recurrence quanti-
cation analysis of nonlinear dynamical systems. In M. A.
Riley& G. Van Orden (Eds.), Tutorials in contemporary non-
linear methods for the behavioral sciences (pp. 26–94).
Williams, P. L., Warick, R., Dyson, M.,& Bannister, L. H.
(1989). Gray’s anatomy. Edinburgh:Churchill Livingstone.
Xu, D., Yapanel, U.,& Gray, S. (2009). Reliability of the
LENA (TM) language environment analysis system in young
children’s natural home environment (No. LTF-05-2).
Boulder, CO:LENA Foundation. Retrieved from http://
www. lenafoundation.org/TechReport.aspx/Reliability/
LTR-05-2
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 535 7/18/2014 1:21:56 PM

536 Affective Computing, Emotional Development, and Autism
Yale, M. E., Messinger, D. S., Cobo-Lewis, A. B. (2003). e
temporal coordination of early infant communication.
Developmental Psychology, 39(5), 815-824.
Yirmiya, N., Gamliel, I., Pilowsky, T., Feldman, R., Baron-Cohen,
S.,& Sigman, M. (2006). e development of siblings of
children with autism at 4 and 14months:Social engage-
ment, communication, and cognition. Journal of Child
Psychology and Psychiatry, 47(5), 511–523. doi:10.1111/j.1
469-7610.2005.01528.x
Yuditskaya, S. (2010). Automatic vocal recognition of a child’s
perceived emotional state within the Speechome corpus.
Massachusetts Institute of Technology. Retrieved from
http://dspace.mit.edu/handle/1721.1/62086
OUP UNCORRECTED PROOF – FIRSTPROOFS, Thu Jul 17 2014, NEWGEN
book.indb 536 7/18/2014 1:21:56 PM