ArticlePDF AvailableLiterature Review

Conceptual Alignment: How Brains Achieve Mutual Understanding

Authors:

Abstract and Figures

We share our thoughts with other minds, but we do not understand how. Having a common language certainly helps, but infants' and tourists' communicative success clearly illustrates that sharing thoughts does not require signals with a pre-assigned meaning. In fact, human communicators jointly build a fleeting conceptual space in which signals are a means to seek and provide evidence for mutual understanding. Recent work has started to capture the neural mechanisms supporting those fleeting conceptual alignments. The evidence suggests that communicators and addressees achieve mutual understanding by using the same computational procedures, implemented in the same neuronal substrate, and operating over temporal scales independent from the signals' occurrences. State-of-the-art artificial agents such as the virtual assistants on our phones are powered by associative deep-learning algorithms. Yet those agents often make communicative errors that, if made by real people, would lead us to question their mental capacities.We argue that these communicative errors are a consequence of focusing on the statistics of the signals we use to understand each other during communicative interactions.Recent empirical work aimed at understanding our communicative abilities is showing that human communicators share concepts, not signals.The evidence shows that communicators and addressees achieve mutual understanding by using the same computational procedures, implemented in the same neuronal substrate, and operating over temporal scales independent from the signals' occurrences.
Neural Dynamics of Sharing Conceptual Spaces within a Single Communicative Exchange (A) and Over Multiple Exchanges (B). (A) Compared with non-communicative control interactions, brain regions necessary for processing conceptual knowledge are already upregulated during communicative interactions before a communicative signal is generated or interpreted (epochs 1 and 2 of Figure I in Box 1) [29]. The right temporal lobe (TL) shows tonic upregulation of neural activity during both the generation and interpretation of communicative signals, without a transient response time locked to the sensorimotor events of these epochs. These temporal dynamics indicate that neural activity in the right TL is modulated by communication over a timescale decoupled from signal occurrences. Ongoing neural activity in the ventromedial prefrontal cortex (vmPFC) is also upregulated as a function of the communicative task set, with generation evoking more computational loads than interpretation irrespective of the communicative nature of the task. This pattern fits with the recent observation that vmPFC lesion patients remain able to generate communicatively effective signals but these communicative decisions are not fine-tuned with a conceptual space defined by the ongoing interaction [106]. Differently from the right TL and the vmPFC, the posterior superior temporal sulcus (pSTS) is sensitive to computational demands that occur early during generation and rise during interpretation; that is, with the presentation of new stimulus material. Repetitive transcranial magnetic stimulation (TMS) over the right pSTS perturbs action understanding on the basis of the recent communicative history, suggesting that this region is necessary for integrating sensory material with a conceptual space defined by the ongoing interaction [107]. (B) As people converge on shared conceptual spaces over multiple communicative exchanges, the right superior temporal gyrus (STG) is increasingly involved irrespective of whether a communicative signal is being generated or interpreted (see [72]). The right STG exhibits more cerebral activity during communicative episodes in which interlocutors can rely on previously shared conceptual spaces (tan blocks depicting one or more consecutive exchanges of a type for which joint solutions have been previously established). (C) Spectrotemporal characteristics of conceptual alignment. Blood oxygen level-dependent (BOLD) signal power and coherence spectra of the right STG indicate that sharing a communicative history (real pairs, green) may result in matched, zero phase-lag cerebral dynamics across
… 
Content may be subject to copyright.
Opinion
Conceptual Alignment:
How Brains Achieve Mutual
Understanding
Arjen Stolk,
1,
* Lennart Verhagen,
2
and Ivan Toni
3
We share our thoughts with other minds, but we do not understand how. Having
a common language certainly helps, but infantsand touristscommunicative
success clearly illustrates that sharing thoughts does not require signals with a
pre-assigned meaning. In fact, human communicators jointly build a eeting
conceptual space in which signals are a means to seek and provide evidence for
mutual understanding. Recent work has started to capture the neural mecha-
nisms supporting those eeting conceptual alignments. The evidence suggests
that communicators and addressees achieve mutual understanding by using
the same computational procedures, implemented in the same neuronal sub-
strate, and operating over temporal scales independent from the signals
occurrences.
Why Doesnt My Phone Understand Me, Yet?
Human communication is often framed in terms of signal transmission [1,2]. This framework
presupposes that communicators already share the same set of codingdecoding rules (e.g., a
common language, body emblems). That intuition seems plausible until we try to build articial
cognitive agents that can deal with human communicators beyond the boundaries of precisely
choreographed interactions (http://www.newyorker.com/tech/elements/why-cant-my-
computer-understand-me). Consider human interactions with common articial agents such
as Apple's Siri, Microsoft's Cortana, or Google Now. Dramatic increases in computing power,
data set availability, and sophistication of machine-learning algorithms have made those articial
agents extremely useful within our daily lives. However, we would question the understanding of
an English-speaking interlocutor that, in all seriousness, guides us to the nearest casino after
being told I have a gambling problem, as Siri does. Even an embodied humanoid agent
designed to physically and communicatively interact with us fails to consider that someone might
want to raise their hand for reasons other than asking a question (http://www.bbc.com/news/
technology-23196867 and Movie S1 in the [2_TD$DIFF]supplemental [3_TD$DIFF]information online). Is there a funda-
mental reason for those communicative failures? Robotics has long struggled with perceptual
and sensorimotor restrictions, but modern articial cognitive agents have access to powerful
feature detectors and associative procedures such as hierarchically organized convolutional
neural networks and reinforcement learning algorithms [35]. Those algorithms can reliably
extract and categorize a signal's features, given the contextual background of statistical
regularities present in a set of training exemplars, and link them to adaptive action selection
procedures [4,6]. Here we argue that adding more feature detectors, sensorimotor associations,
or processing speed is unlikely to solve those communicative failures. Indeed, those communi-
cative failures might be engineered, in the sense that those articial agents have been focused on
information transfer (in Shannon's sense [1]) rather than on the computational problem solved in
human communication (in Marr's sense [7]). Building on previous suggestions [814], we argue
Trends
State-of-the-art articial agents such
as the virtual assistants on our phones
are powered by associative deep-
learning algorithms. Yet those agents
often make communicative errors that,
if made by real people, would lead us to
question their mental capacities.
We argue that these communicative
errors are a consequence of focusing
on the statistics of the signals we use to
understand each other during commu-
nicative interactions.
Recent empirical work aimed at under-
standing our communicative abilities is
showing that human communicators
share concepts, not signals.
The evidence shows that communica-
tors and addressees achieve mutual
understanding by using the same com-
putational procedures, implemented in
the same neuronal substrate, and
operating over temporal scales inde-
pendent from the signalsoccurrences.
1
Helen Wills Neuroscience Institute,
University of California, Berkeley, CA
94720, USA
2
Department of Experimental
Psychology, University of Oxford,
Oxford OX1 3UD, UK
3
Donders Institute for Brain, Cognition,
and Behaviour, Radboud University,
6500 HB Nijmegen, The Netherlands
*Correspondence: astolk@berkeley.edu
(A. Stolk).
TICS 1521 No. of Pages 12
Trends in Cognitive Sciences, Month Year, Vol. xx, No. yy http://dx.doi.org/10.1016/j.tics.2015.11.007 1
© 2015 Elsevier Ltd. All rights reserved.
TICS 1521 No. of Pages 12
that creating a shared conceptual space across communicators might be a more fruitful
characterization of the computational problem solved in human communication.
Considering human communication as a signal codingdecoding problem continues to entice
empirically oriented scholarly elds. For instance, many communication studies in cognitive
neuroscience build on the assumption that retrieval and selection of a signal's meaning is bound
to and triggered by the signal occurrence [2,1520]. Priming, neural synchrony, and shared
sensorimotor associations are among the mechanisms that have been suggested to implement
information transfer [2,1520]. Here we discuss a mechanism geared to implement a different
computational function of human communication. This mechanism has emerged because it has
become feasible to empirically interrogate, at the implementation level, longstanding theoretical
perspectives of human communication [814]. We review evidence showing that the main load
of human communication is carried by a neuronal mechanism continuously aligning our con-
ceptual structures with those of another agent. We argue that this dynamic alignment provides a
conceptual frame necessary for interpreting intrinsically ambiguous communicative signals. We
discuss neuronal mechanisms supporting the integration of a communicative signal within those
dynamically adjusted conceptual structures.
Dynamic Conceptual Alignments
Across multiple generations, natural selection can drive organisms toward shared coding
decoding rules [21,22], and articial agents can share novel symbols by trial and error [2325].
However, establishing those symbols without the presence of pre-existing common knowledge
requires articial agents to use thousands of pair-wise interactions [23] or several iterations
across independent populations of communicators [22]. This learning dynamic is not remotely
related to how humans converge on a shared meaning and disambiguate situations lacking
predened codingdecoding schemes, an ability that often operates on the basis of a single trial
[2630]. Rapid resolution of novel or ambiguous symbols is the daily bread and butter of human
communication and is crucial for understanding the problem solved during mutual under-
standing (see Glossary)[31]. Dealing with ambiguities by converging on shared meanings is not
an exceptional situation that requires exceptional learning times. We promptly resolve multiple
communicative ambiguities in everyday conversations, as well as when we learn a language as
an infant [10,32]. Even commonly used words do not contain xed meanings they may provide
us with clues to a communicative meaning [3335] but are coordinated through an interactive
process by which people seek and provide evidence that they understand one another [36,37].
Accordingly, the notion of communicative meaningencompasses both the ability to recognize
the communicator's communicative intentionand the content of the behavior manifesting that
intention (informative intention[9]). These naturalistic observations are supported by recent
empirical observations on how human communicators colonize a semiotically virgin territory
[38,39]. If the computational problem of communicators is to align their conceptual spaces,
signals should differentiate contingently to the local interactional conditions, leading to rapid
semiotic diversity across communicative groups. Conversely, if human communication is about
optimizing signal codingdecoding [1,15,16] equal communicative demands and initial con-
ditions across communicators (e.g., background knowledge) should lead to similar signals
across communicative groups. Empirical and natural observations clearly favor the former
scenario [26,29,3943] (Box 1).
Given the multiple semantic ambiguities present in an everyday utterance, how can we quickly
and reliably identify signals adequate to focus the mind of an addressee on our communicative
intention or infer the communicative intention suggested by our interlocutor? For instance, a
customer might solicit a drink in a bar with a propositionally inconsequential and semantically
ambiguous statement (A cup of Joe;Figure 1A). A bartender might reply with a logically
unrelated and similarly ambiguous statement (Were closing in ve). Yet, an eavesdropper at the
Glossary
Conceptual alignment: condition in
which individualsmental
representations have become
aligned, or sufciently compatible,
despite those individuals
idiosyncratic experiences and
knowledge structures. People can,
for instance, refer to the same coffee
cup in a kitchen although they may
have different (visual) perspectives or
background knowledge of the object.
The alignment is conceptual because
it occurs at a level abstracted away
from actual experience, such that the
same people may refer to the same
object in a different moment or
situation (see mutual understanding).
Mutual understanding: when
different minds mutually infer they
agree on an understanding of an
object, person, place, event, or idea
(see conceptual alignment).
2Trends in Cognitive Sciences, Month Year, Vol. xx, No. yy
TICS 1521 No. of Pages 12
counter would not be surprised when the interlocutors quickly resolve the pragmatic implications
of those semantic ambiguities. The customer would also have no difculty in interpreting the
bartender's reply (Thatll be a ver) in the context of his opening statement, rather than limiting
its search for a relevant context to temporally closer or arithmetically more consistent events
(e.g., Add one...;Figure 1A) in the turn-taking sequence.
This vignette illustrates that, in human communication, an action can be a response to a signal
occurring at any time along the interaction's trajectory, irrespective of linear order or syntactic
regularities. These hierarchically embedded and temporally irregular conceptual dependencies
between utterances cannot be easily resolved by neural and computational processes exclu-
sively focused on statistically predominant features within signal sequences [2,44]; that is, the
regularities captured by convolutional neural networks and reinforcement learning algorithms [3].
Box 1. Capturing Communication in the Laboratory
Understanding how humans communicate requires experimental protocols that capture mutual understanding. Yet,
paradoxically, cognitive neuroscientists have often approached the study of human com munication by minimizing mutual
understanding; for example, by studying an individual agent producing scripted utterances or processing isolated
sentences [8082]. Recently, empirical studies have started to pay attention to the communicative context in which those
signals are embedded. One approach particularly effective for capturing communicatorsshared communicative history
involves people communicating in a novel medium [24,26,29,39,42,95,96]. The novel medium minimizes participants
access to a number of pre-existing conventions exploited in everyday communication (e.g., a common language, body
emblems, facial expressions). Consequently, the generation and comprehension of communicative signals become
strongly conditional on the conceptual space idiosyncratically dened by the ongoing interaction. For instance, when
pairs of players are asked to communicate by moving geometric shapes on a digital board (Figure IA), the same signal can
be used by different pairs to coordinate different meanings. The same signal can even have different meanings in different
trials of the same pair and vice versa (for examples see movies in [29] and [72]).There are no a priori correct solutions to
this communicative task nor a limited set of options from which the players can choose. Several pieces of evidence
indicate that the players jointly and dynamically establish an agreement (also known as a conceptual pact[97]) on the
meaning of a signal. For instance, changes in the communicator's movement characteristics after a misinterpretation of
the addressee are dependent on the nature of the errormade by the addressee [98], suggesting that communicators
take into account how addressees interpret their signals and adjust them accordingly. Thus, as in everyday dialogue,
effective communication in this game arises only when players align their conceptual solutions. Yet, communicative
difculty and communicatorsshared cognitive history can fairly easily be manipulated by varying the complexity of the
spatial goal congurations (for examples see [98]) and having pairs encounter problems for which they previously have
established a joint solution (see [72]), respectively. Accordingly, this computer game offers an expe rimental platform with
the possibility to manipulate and quantify communicatorseeting conceptual alignments over a series of interactions and
isolate the fundamental mechanisms of human social interaction. See Figure IB for a comparison of the communication
game with everyday dialogue.
“A cup of joe”
“A cup of joe”
“We’re closing in five”
“We’re closing in five”
“Add one for the missus then”
“Add one for the missus then”
“That’ll be a fiver”
“That’ll be a fiver”
“I like fishing at the river bank”
“So where can i find a nice bank”
tap to edit
Got it.
The best-rated one i found is
wells fargo bank on solano
ave, which averages 3½
stars.
(A) (B)
15 Results
Nearby
0.8 mi
1.2 mi
Wells fargo bank
Wells fargo bank
1800 solano ave
1095 university ave
14 reviews on yelp
?
Figure 1. Prototypical Human Communicative Interaction with Another Human (A), and with an Articial Agent
(B). Artwork courtesy of the Art Institute of Chicago. Screenshot of Apple Siri interaction.
Trends in Cognitive Sciences, Month Year, Vol. xx, No. yy 3
TICS 1521 No. of Pages 12
The vignette also illustrates how communication requires more than pruning a decision tree of
possible options stored in memory [45,46]. The concepts used in everyday interaction are not
well-dened structures with denite values [47]. Indeed, a communicator often needs to
generate novel candidate conceptualizations of a signal, as when a customer would hear ver
for the rst time.
(B)
(A)
1
2
3
Interpreta!on
of a
communica!ve
signal
Genera!on
of a
communica!ve
signal
2
1
Communicator Addressee
Experimental communica"onNatural dialogue
What’s iden!cal?
Mul!ple communica!on channels
vocaliza!ons, bodily and facial postures/movements,
eye contact
Single communica!on channel
movements on a computer monitor:
experimental control over communica!ve environment
Access to pre-exis!ng conven!ons
a common language, body emblems, facial
expressions
Novel communica!ve signals
lack of pre-exis!ng shared representa!ons:
experimental control over shared cogni!ve history
Spontaneous turn-taking Experimentally-controlled roles
Communica!ve meaning of a signal relies on a shared conceptual space
mutually coordinated, updated according to the flee!ng idiosyncrasies of an ongoing interac!on
What’s dierent?
1
2
3
4
Figure I. Experimentally Controlled Communicative Interaction. (A) The two-player game setup is computer
programmed and presented individually on two separate monitors. The players control their shape movements (horizontal
and vertical translations and 908rotations) using hand-held controllers. In this example the player in blue, labeled as the
Communicator, tries to make clear to the player in orange, labeled as the Addressee, that her triangulartoken should be
positioned in the bottomright square, pointing left. At onset of each interaction a shape is assigned to each player, followed
by presentation of the goal conguration to the Communicator (epoch 1). During this epoch he can plan for as long as
needed, but he has only5 s to execute his movements in the next. Afterpressing a start button, the Communicator's shape
will appear in the center of the grid. He now can execute his actions, visible to the Addressee who needs to infer the
Communicator's intentions from his movements (epoch 2); for instance, by rst going to her target location, ostensibly
pausetoindicate the relevance ofthat location (number 1 action),then wiggleto indicateher shape's orientation (number2
action), and then completinghis own target conguration (number 3 action).Please note that this is only oneamong a series
of possible solutions. For instance, some participants converge on using the number of subsequent wigglesto mark the
number of clockwiserotations that the Addressee needs to make to achievethe target orientation of her shape, while others
do not use the wigglebut leave the triangle location along the direction to which the triangle needs to point. Following the
Communicator'smovements, the Addresseecan plan and execute her actionsto complete the goal conguration(epoch 3).
Finally,feedback on their communicativesuccess is presentedto both players in the form of a green tickor a red cross (epoch
4). Adapted from [29]. (B) Comparison of the communication game with everyday social interaction.
4Trends in Cognitive Sciences, Month Year, Vol. xx, No. yy
TICS 1521 No. of Pages 12
Resting on theoretical accounts that have highlighted the social implicatures intrinsic to a given
communicative signal [8,9,11], we propose that the customer generates a set of possible-world
scenarios based on his shared conceptual space with the bartender and plans how to probe his
inference to the best explanation in his next utterance or move. This generative ability would
require a mechanism for nding consistent relationships between abstract features of commu-
nicative signals, a mechanism that spins a conceptual web across potentially unrelated elements
of the communicative interaction (e.g., a link between Joe, hypotheses about ver, a large
body of background knowledge about bartenders) [14,37,4850]. This mechanism would kick
in even before the rst signal in the vignette is produced; for example, when the customer
assumes that the person behind the bar is familiar with the intended meaning of Joeand is
willing to sell drinks. Several possible-world contexts might need to be prepared to achieve
the exibility characteristic of human interactions (e.g., the bar might be already closed,
the presumed bartender is a cleaner). Thus, humans need to be able to efciently explore
large search spaces and establish connections between different conceptual structures [29].
Although it remains unclear how to make this operation computationally tractable [5,51,52],
similar exploration-exploitation trade-offs might be used during domain-general foraging deci-
sions in unstable environments [45,5355]. As the interaction unfolds, communicators continu-
ously update their conceptual spaces, keeping their mutually inferred thoughts aligned to the
current situation and to each other. This alignment process builds on semantic structures
operating over multiple timescales, from the eeting idiosyncrasies of an ongoing communicative
interaction to long-term semantic memories and regularities acquired throughout development
[56].
The conceptual alignment account can be contrasted with signal-centered frameworks of
communication in which retrieval and selection of a signal's meaning is bound to and triggered
by the signal occurrence [1,2,1520]. For instance, the scripted responses of current articial
agents might be informed by long-term statistical regularities but fail to consider the importance
of the ongoing communicative dynamics for building mutual understanding (Figure 1B). This
fundamental aspect of human communication can be easily overlooked when the focus is on
the signal itself rather than on the conceptual space evolving between communicators. In the
conceptual alignment framework, meaning is not a property of a signal but a property of a
mutually inferred conceptual space in which signals are merely a means to probe and bias that
conceptual space (Figure 2, Key Figure). By embedding signals in a conceptual space dened by
the ongoing interaction, communicators can exibly infer the meanings of those signals and
overcome the curse of dimensionality(i.e., a combinatorial explosion with a number of features)
intrinsic in signal-centered approaches to human communicative situations [2,12,57]. The
conceptual alignment framework provides an account of our ability to disambiguate everyday
symbols and to reliably zoom onto relevant features of multimodal communicative signals [8],
and more generally for the evolutionary anomaly of extreme referential exibility in human signals
[10,58,59].
Neural Evidence for a Shared Conceptual Space Supporting Human
Communication
According to the conceptual alignment framework, the communicative meaning of a signal relies
on a eeting conceptual space dened by the ongoing interaction. This notion leads to four
predictions of the characteristics of neural activity supporting human communication. First,
achieving mutual understanding should evoke neural activity reecting exible conceptual
processes [6062] rather than sensorimotor operations with limited generalization potential
[2,16,18,6365]. Second, there should be shared patterns of neural activity during the genera-
tion and interpretation of communicative signals, given that these processes relate to the same
conversational context [66,67]. Third, the timing of this shared neural pattern should lead, not
follow, the occurrence of a communicative signal, given that the conceptual space that gives
Trends in Cognitive Sciences, Month Year, Vol. xx, No. yy 5
TICS 1521 No. of Pages 12
Key Figure
A Shift in Our Conceptualization of Human Mutual Understanding
Time
Signal-centered frameworks
Meaning is a property of the signal
Conceptual alignment framework
Meaning is a property of a mutually
coordinated conceptual space
Coee Encode Decode
Decode Encode
C
o
n
c
e
p
t
u
a
l
e
m
b
e
d
d
i
n
g
C
o
n
c
e
p
t
u
a
l
e
m
b
e
d
d
i
n
g
“Joe”
...
“fiver”
Coee
? 5 dollars
Payment
5 dollars
Coee
Barman
Drink
2 cups
Communica!ve
signal
Barman
concept space
Customer
concept space
Time
Order
5 dollars
Coee
Customer
Drink
2 cups
...
“fiver”
“Joe”
“A cup of joe”
“A cup of joe”
“We’re closing in five”
“We’re closing in five”
“Add one for the missus then”
“Add one for the missus then”
“That’ll be a fiver”
“That’ll be a fiver”
Figure 2. To date, several accounts of our communicative abilities have focused on the information content of the signals
we use to understand each other during communicative interactions. These accounts implicitly assume that statistical
regularities in these signals could coordinate the production and comprehension of their meanings across communicators.
By contrast, according to the conceptual alignment framework, human communicators mutually coordinate a eeting
conceptual space in which signals are merely a means to seek and provide evidence for mutual understanding. By
embedding communicative signals in this dynamically adjusted space (portrayed by the broken lines), communicators can
exibly resolve the ambiguities inherent in these signals (e.g., Joe), often at their rst occurrence (e.g., ver). Artwork
courtesy of the Art Institute of Chicago.
6Trends in Cognitive Sciences, Month Year, Vol. xx, No. yy
TICS 1521 No. of Pages 12
vmPFC
pSTS
TL
Genera!on
of a
communica!ve
signal
Interpreta!on
of a
communica!ve
signal
pSTS
TL
vmPFC
pSTS pSTS
TL
vmPFC vmPFC
TL
Absolute index of
neural ac!vity
Absolute index of
neural ac!vity
Absolute index of
neural ac!vity
Genera!on ( 10 sec)
(A) Single exchange
BOLD signal
STG
Interpreta!on ( 5 sec)
Communica!ve signal
Communica!ve signal
Time ( 82 sec)
(B) Mul!ple exchanges
STG
BOLD state correla!on
NKNK
*
Real pairs Random pairs
Real pairs
Random pairs
BOLD signal coherence
Frequency (Hz)
Novel episodes
(C)
BOLD signal power
Frequency (Hz)
Signal frequency
Signal frequency
Known episodes
Communica"ve
Non-communica"ve
Communica"ve
Non-communica"ve
Figure 3. Neural Dynamics of Sharing Conceptual Spaces within a Single Communicative Exchange (A) and
Over Multiple Exchanges (B). (A) Compared with non-communicative control interactions, brain regions necessary for
processing conceptual knowledge are already upregulated during communicative interactions before a communicative
signal is generated or interpreted (epochs 1 and 2 of Figure I in Box 1)[29]. The right temporal lobe (TL) shows tonic
upregulation of neural activity during both the generation and interpretation of communicative signals, without a transient
response time locked to the sensorimotor events of these epochs. These temporal dynamics indicate that neural activity in
the right TL is modulated by communication over a timescale decoupled from signal occurrences. Ongoing neural activity in
the ventromedial prefrontal cortex (vmPFC) is also upregulated as a function of the communicative task set, with generation
evoking more computational loads than interpretation irrespective of the communicative nature of the task. This pattern ts
with the recent observation that vmPFC lesion patients remain able to generate communicati vely effective signals but these
communicative decisions are not ne-tuned with a conceptual space dened by the ongoing interaction [106]. Differently
from the right TL and the vmPFC, the posterior superior temporal sulcus (pSTS) is sensitive to computational demands that
occur early during generation and rise during interpretation; that is, with the presentation of new stimulus material. Repetitive
transcranial magnetic stimulation (TMS) over the right pSTS perturbs action understanding on the basis of the recent
communicative history, suggesting that this region is necessary for integrating sensory material with a conceptual space
dened by the ongoing interaction [107]. (B) As people converge on shared conceptual spaces over multiple communicative
exchanges, the right superior temporal gyrus (STG) is increasingly involved irrespective of whether a communicative signal is
being generated or interpreted (see [72]). The right STG exhibits more cerebral activity during communicative episodes in
which interlocutors can rely on previously shared conceptual spaces (tan blocks depicting one or more consecutive
exchanges of a type for which joint solutions have been previously established). (C) Spectrotemporal characteristics of
conceptual alignment. Blood oxygen level-dependent (BOLD) signal power and coherence spectra of the right STG indicate
that sharing a communicative history (real pairs, green) may result in matched, zero phase-lag cerebral dynamics across
(Figure legend continued on the bottom of the next page.)
Trends in Cognitive Sciences, Month Year, Vol. xx, No. yy 7
TICS 1521 No. of Pages 12
meaning to the signal is dened by the ongoing communicative interaction rather than by the
signal itself. Fourth, the temporal dynamics of the shared neural pattern should reect the
communicatorsadjustments of their shared conceptual space.
The rst three predictions were veried in a magnetoencephalography (MEG) study that revealed
changes in neural activity sensitive to the task context rather than to the occurrence of specic
communicative events [29]. Achieving mutual understanding (Box 1) upregulated neural activity
in the right temporal lobe (TL) and in the ventromedial prefrontal cortex (vmPFC), two brain
regions necessary for processing conceptual knowledge [6871]. The same upregulation of
neural activity was found across communicators and addressees irrespective of whether a
communicative signal was being generated or interpreted (Figure 3A). It should be emphasized
that conventional neuroimaging approaches, focused on changes in neural activity relative to a
trial or session baseline, would not be able to detect these tonic upregulations of activity. These
novel effects were isolated using dedicated timefrequency analyses focused on an absolute
index of neural activity (Figure 3A). The nding indicates that these brain regions supported
communication by means of a tonic upregulation of neural activity linked to task features shared
across interlocutors rather than to sensorimotor events differing between interlocutors by
experimental design. The overlapping neural upregulation occurred before any communicative
signal was generated or interpreted and the magnitude of the neural upregulation had measur-
able consequences on communicative performances (see [29]). These empirical observations
conrm the intuition that the meaning of a communicative signal is inferred by embedding that
signal in a conceptual space whose activation precedes in time the processing of the commu-
nicative stimulus material itself (see Box 2 for a discussion of a neuronal implementation).
The fourth prediction was veried using two fMRI scanners to simultaneously record cerebral
activity in pairs of communicators trying to understand each other over a series of communica-
tive interactions (Box 1) in which shared conceptual spaces had to be coordinated de novo
(novel interactions) or retrieved from a preceding training session (known interactions)[72].
This experimental manipulation evoked more cerebral activity in the right TL and the vmPFC
during known than during novel interactions, in both communicators and addressees. Crucially,
an anterior portion of the right TL followed the behavioral dynamics of mutual understanding over
the course of the experiment. This region, the superior temporal gyrus (STG), was not only
sensitive to the known/novel nature of communicative interactions (Figure 3B) but also became
increasingly involved as people converged on shared conceptual spaces over the course of
novel interactions (see [72]). Participants with a common communicative history showed
synchronized intercerebral dynamics in their STG activity. Crucially, this pair-specic intercere-
bral coherence occurred over a timescale decoupled from signal occurrences (Figure 3C) and
only when the communicators needed to mutually adjust their conceptual spaces (far-right plot
in Figure 3C). This observation indicates that converging on conceptual spaces may result in
cerebral coherence between communicators at temporal scales independent from signal
occurrences, providing empirical evidence against accounts of human communication that
emphasize priming, neural synchrony, or shared sensorimotor processes as the basic mecha-
nism for mutual understanding [1,2,1520,73].
communicators over a timescale decoupled from signal occurrences [see the black boxcar traces in (A) and (B)]. Gray
surface indicates frequencies with a statistically signicant difference in coherence between real and random pairs. The bar
plot indicates that this pair-specic cerebral coherence occurs when pairs need to mutually adjust their conceptual spaces
(N, novel episodes), but not when those pairs can fully rely on previously shared conceptual spaces that do not require
equally frequent updating (K, known episodes). BOLD state in the bar plot refers to cerebral activity evoked during a known
or novel episode, controlled for transient task events within each communicative episode. Random pairs (black) are
combinations of participants that were engaged in the same communicative task as real pairs but without a shared
communicative ground (e.g., communicator from pair 1 with addressee from pair 2). Asterisk denotes a statistically
signicant interaction between pair and episode type. Adapted from [29,72].
8Trends in Cognitive Sciences, Month Year, Vol. xx, No. yy
TICS 1521 No. of Pages 12
Box 2. A Neuronal Mechanism for Integrating Communicative Signals into Dynamically Adjusted
Conceptual Structures
What neurophysiological mechanism allows a sustained yet adjustable inuence of conceptual knowledge on transient
signal production and perception? A clue is provided by the spectrotemporal nature of neural activity observed over the
frontal and temporal cortices when people interact. These brain regions are tonically upregulated during communicative
interactions indeed already before the occurrence of observable events and with measurable behavioral conse-
quences on communicative performance [29]. These same regions also show strikingly similar phasic neural dynamics
during the generation and interpretation of non-communicative events (Figure 3A). This observ ation is consistent with an
increasing body of evidence showing that contextual demands can modulate ongoing neural activity yet retain
responsiveness to event-related neural processing (e.g., [99,100]).
Second, the neural upregulation found over the frontal and temporal cortices had an extremely broad spectral prole [29].
Physiologically, broadband shifts of neural activity (a change of amplitude across all frequencies) are functionally distinct
from band-limited neuronal oscillations [101] and are thought to reect changes in the mean ring rates of neuronal
populations [102,103]. These population-level ring rates have been shown to be affected by internal cortical states as
much as by external stimuli [104] and are instrumental for integrating driving afferences with contextual information [74
78]. As portrayed in Figure I, ongoing contextual inputs can temporarily hold selective neurons in an excitable state. The
excitable state increases the probability of these neurons propagating afferences, effectively integrating information
associated with the input streams.
This neuronal mechanism, based on upregulated broadband neural activity, might provide a neural marker of eeting
knowledge spaces [29]. By contrast, phasic neural dynamics may be indicative of event-related computatio ns. Analytical
approaches focused on event-related neural activity may largely miss temporally extended computations that support
conceptual processing [99].
cortex (2 - 3 mm)
Current
Contextual aerences
hold neuron in excitable state
Integrated output
Event-related aerences (sensory inputs)
Cell body
Figure I. Neuronal Implementation of Conceptual Spaces. Unlike axonal action potentials, dendritic currents do
not propagate reliably over long distances. Propagation relies on the currentsability to reach a threshold at the neuron's
cell body and thus, in part, on their proximity to that cell body. This location dependency of a neuron's excitatory input is
instrumental for integrating driving afferences with contextual information [7478]; namely, neurons that are temporarily
upregulated through ongoing inputs are more likely to propagate sensory afference than neurons that are in a less
excitable state. When the summed inputs reach the cell body's threshold, a downstream action potential is generated,
effectively integrating information associated with the two input streams (e.g., the situation depicted on the right).
Electrophysiologically, even when dendritic currents fail to trigger axonal action potentials (e.g., the left situation), their
eld potentials are captured as a spatially weighted average by the electroencephalography (EEG)/magnetoencephalo-
graphy (MEG)/electrocorticography (ECoG) signal [105]. Ongoing contextual inputs that hold neuronal populations near
an excitability threshold may thus induce changes in the brain signal that are not temporally bound to the occurrence of
observable events. The upregulation of broadband neural activity during human communicative interactions (Figure 3A)
might be an instance of this contextual phenomenon [29], the extremely broad spectral prole owing to the noise-like
distribution of input arrival times [102]. Artwork courtesy of Greg Dunn Design.
Trends in Cognitive Sciences, Month Year, Vol. xx, No. yy 9
TICS 1521 No. of Pages 12
Concluding Remarks
This [4_TD$DIFF]opinion article provides theoretical and empirical arguments for a shift in our conceptuali-
zation of human mutual understanding. Qualifying previous suggestions [814], we have shown
how the neural mechanisms used during human communication are shared across commu-
nicators and addressees and how these mechanisms follow the dynamics of mutual under-
standing rather than the occurrences of communicative signals. We have argued that
conceptual alignment (i.e., a continuous dynamic alignment of individual knowledge spaces)
provides a cognitive framework suitable for resolving the ambiguities inherent in human com-
municative signals. Neuronally, conceptual alignments appear to be implemented through
spectrally and temporally extended phenomena, namely upregulation of broadband neural
activity [29] (Box 2), integrating driving afferences with contextual information [7478].
Considering human communication as a meeting of minds rather than as transmitting signals
has important implications for several academic elds (see Outstanding Questions). For
instance, it becomes relevant to consider temporally extended neural integration mechanisms
to understand how conceptual spaces support human semantic and pragmatic abilities, over
and above the stimulus centered approaches currently predominantly used in neurocognitive
studies of language and communication [79]. The study of these mechanisms could benet from
experimental protocols focused on generative communication, namely the generation of com-
municative behaviors from an open-ended set of possibilities [38,72], rather than on the
reproduction of well-rehearsed scripts [8082]. It becomes relevant to study how humans
developmentally acquire mutual understanding within and through social interactions, over and
above individualistic processes [8385]. Cerebral alterations leading to communicative impair-
ments like autism spectrum disorder and frontotemporal dementia [70,86,87] might benet from
being reconceptualized as decits in creating and probing a shared conceptual space with a
communicator. Finally, articial cognitive agents might better satisfy human communicative
expectations by using a cognitive architecture that continuously updates the conceptual space
shared with an interlocutor, over and above rapid extraction of the statistically predominant
features of a signal [35,88,89].
Acknowledgments
This work was supported by Rubicon Grant 446-14-007 and VICI Grant 453-08-002 from The Netherlands Organisation for
Scientic Research (to A.S. and I.T., respectively) and Marie Curie Intra-European Fellowship MC-IEF-623513 from the
European Union (to L.V.). The authors thank Lawrence ODwyer, Ruud Berkers, and three anonymous reviewers for
comments on drafts of the manuscript. They are also grateful for many discussions about this topic with Mark Blokpoel and
Iris van Rooij.
[5_TD$DIFF]Supplemental Information
[5_TD$DIFF]Supplemental information associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.tics.
2015.11.007.
References
1. Shannon, C. (1948) A mathematical theory of communication.
Bell Syst. Tech. J. 27, 379423
2. Hasson, U. et al. (2012) Brain-to-brain coupling: a mechanism
for creating and sharing a social world. Trends Cogn. Sci. 16,
114121
3. LeCun, Y. et al. (2015) Deep learning. Nature 521, 436444
4. Mnih, V. et al. (2015) Human-level control through deep rein-
forcement learning. Nature 518, 529533
5. Gershman, S.J. et al. (2015) Computational rationality: a con-
verging paradigm for intelligence in brains, minds, and machines.
Science 349, 273278
6. Frank, M.J. and Badre, D. (2012) Mechanisms of hierarchical
reinforcement learning in corticostriatal circuits 1: computational
analysis. Cereb. Cortex 22, 509526
7. Marr, D. (1982) Vision: A Computational Investigation into
the Human Representation and Processing of Visual Information,
W.H. Freeman
8. Sperber, D. and Wilson, D. (2001) Relevance: Communication
and Cognition, Blackwell
9. Grice, H.P. (1969) Utterers meaning and intentions. Philos. Rev.
78, 147177
10. Levinson, S.C. (2006) On the human interactional engine. In
Roots of Human Sociality (Eneld, N. and Levinson, S., eds),
pp. 3969, Berg
11. Clark, H.H. (1996) Using Language, Cambridge University Press
12. Misyak, J.B. et al. (2014) Unwritten rules: virtual bargaining
underpins social interaction, culture, and society. Trends Cogn.
Sci. 18, 512519
Outstanding Questions
The conceptual alignment framework
implies that communicators continu-
ously update their conceptual spaces,
building on both background knowl-
edge and the current interactional con-
text. What biologically plausible
algorithms can support the rapid explo-
ration of these large search spaces and
the generation of novel links between
elements of those spaces? What neu-
ronal mechanisms can realistically span
the extremely long integration and align-
ment windows?
The conceptual alignment framework
presupposesthe continuous generation
and exploration of many possible-world
scenarios. What motivational mecha-
nisms determinehow deep and exhaus-
tive that generation/exploration process
should be? Could rapid turn taking, an
apparently universal characteristic of
human communication [90,91], modu-
late the exploratory process while pro-
viding a continuous motivational drive
toward building a shared conceptual
space?
How does mutual understanding take
off when initial convergence on a
shared conceptual space is severely
limited? Could analogies provide a
generative principle for hypothesizing
possible interpretations of signals
[48,49,92]? Could the combination of
an analogical meaning hypothesizer
with a context-dependent selection
mechanism lead to the generation of
communicative signals interpretable by
that addressee on rst exposure [93]?
An inuential suggestion holds that we
develop mutual understanding abilities
through social interaction itself [8385],
as when we increasingly include expe-
rience and beliefs about the world and
others in our communicative interac-
tions with others [94]. How do children
acquire the neurocognitive mecha-
nisms necessary to make themselves
understood by others?
Cancommunicativealterations observed
in numerous neurological and psychiatric
disorders (e.g., schizophrenia, autism
spectrum disorders, frontotemporal
dementia) be causally unied as failures
in building a shared conceptual space
with an interlocutor?
10 Trends in Cognitive Sciences, Month Year, Vol. xx, No. yy
TICS 1521 No. of Pages 12
13. Noveck, I.A. and Reboul, A. (2008) Experimental pragmatics:
a Gricean turn in the study of language. Trends Cogn. Sci. 12,
425431
14. Gärdenfors, P. (2000) Conceptual Spaces: The Geometry of
Thought, MIT Press
15. Friston, K.J. and Frith, C.D. (2015) Active inference, communi-
cation and hermeneutics. Cortex 68, 129143
16. Keysers, C. and Perrett, D.I. (2004) Demystifying social cognition:
a Hebbian perspective. Trends Cogn. Sci. 8, 501507
17. Pulvermuller, F. et al. (2014) Motor cognitionmotor semantics:
action perception theory of cognition and communication. Neu-
ropsychologia 55, 7184
18. Pickering, M.J. and Garrod, S. (2013) An integrated theory of
language production and comprehension. Behav. Brain Sci. 36,
329347
19. Dumas, G. et al. (2014) The human dynamic clamp as a para-
digm for social interaction. Proc. Natl. Acad. Sci. U.S.A. 111,
E3726E3734
20. Nummenmaa, L. et al. (2014) Mental action simulation synchro-
nizes actionobservation circuits across individuals. J. Neurosci.
34, 748757
21. Danchin, E. et al. (2004) Public information: from nosy neighbors
to cultural evolution. Science 305, 487491
22. Kirby, S. et al. (2014) Iterated learning and the evolution of
language. Curr. Opin. Neurobiol. 28, 108114
23. Steels, L. (2003) Evolving grounded communication for robots.
Trends Cogn. Sci. 7, 308312
24. Puglisi, A. et al. (2008) Cultural route to the emergence of lin-
guistic categories. Proc. Natl. Acad. Sci. U.S.A. 105, 79367940
25. Kirby, S. (2002) Simulating the Evolution of Language, pp. 121
147, Springer
26. Galantucci, B. (2005) An experimental study of the emergence of
human communication systems. Cogn. Sci. 29, 737767
27. Fusaroli, R. and Tylen, K. (2012) Carving language for social
coordination: a dynamical approach. Interact. Stud. 13, 103124
28. Scott-Phillips, T.C. et al. (2012) How do communication systems
emerge? Proc. Biol. Sci. 279, 19431949
29. Stolk, A. et al. (2013) Neural mechanisms of communicative
innovation. Proc. Natl. Acad. Sci. U.S.A. 110, 1457414579
30. Tomasello, M. et al. (2007) A new look at infant pointing. Child
Dev. 78, 705722
31. Tenenbaum, J.B. et al. (2011) How to grow a mind: statistics,
structure, and abstraction. Science 331, 12791285
32. Noordzij, M.L. et al. (2010) Neural correlates of intentional com-
munication. Front. Neurosci. 4, 188
33. Edmiston, P. and Lupyan, G. (2015) What makes words special?
Words as unmotivated cues. Cognition 143, 93100
34. Elman, J.L. (2004) An alternative view of the mental lexicon.
Trends Cogn. Sci. 8, 301306
35. Lupyan, G. and Bergen, B. (2015) How language programs the
mind. Top. Cogn. Sci. Published online July 17, 2015. http://dx.
doi.org/10.1111/tops.12155
36. Brennan, S.E. et al. (2010) Two minds, one dialog: coordinating
speaking and understanding. Psychol. Learn. Motiv. 53, 301344
37. Hofstadter, D. and Sander, E. (2013) Surfaces and Essences:
Analogy as the Fuel and Fire of Thinking, Basic Books
38. Galantucci, B. and Garrod, S. (2011) Experimental semiotics: a
review. Front. Hum. Neurosci. 5, 11
39. de Ruiter, J.P. et al. (2010) Exploring the cognitive infrastructure
of communication. Interact. Stud. 11, 5177
40. Fay, N. et al. (2010) The interactive evolution of human commu-
nication systems. Cogn. Sci. 34, 351386
41. Evans, N. and Levinson, S.C. (2009) The myth of language
universals: language diversity and its importance for cognitive
science. Behav. Brain Sci. 32, 429448 discussion 448494
42. Scott-Phillips, T.C. et al. (2009) Signalling signalhood and the
emergence of communication. Cognition 113, 226233
43. Scott-Phillips, T. (2014) Speaking Our Minds: Why Human Com-
munication is Different, and How Language Evolved to Make it
Special, Palgrave Macmillan
44. Reber, A.S. (1993) Implicit Learning and Tacit Knowledge: An
Essay on the Cognitive Unconsicious, Oxford University Press
45. Donoso, M. et al. (2014) Human cognition Foundations of human
reasoning in the prefrontal cortex. Science 344, 14811486
46. Botvinick, M. and Weinstein, A. (2014) Model-based hierarchical
reinforcement learning and human action control. Philos. Trans.
R. Soc. Lond. B Biol. Sci. Published online November 5, 2014.
http://dx.doi.org/10.1098/rstb.2013.0480
47. Grice, H.P. (1975) Logic and conversation. In Syntax and Seman-
tics 3: Speech Acts (Cole, P. and Morgan, J.L., eds), pp. 4158,
Academic Press
48. Gentner, D. (2003) Why were so smart. In Language in Mind:
Advances in the Study of Language and Thought (Gentner, D.
and Goldin-Meadow, S., eds), pp. 195235, MIT Press
49. Goldstone, R.L. and Rogosky, B.J. (2002) Using relations within
conceptual systems to translate across conceptual systems.
Cognition 84, 295320
50. Durso, F.T. et al. (1994) Graph-theoretic conrmation of restruc-
turing during insight. Psychol. Sci. 5, 9496
51. Blokpoel, M. et al. (2011) The computational costs of recipient
design and intention recognition in communication. In Proceed-
ings of the 33rd Annual Conference of the Cognitive Science
Society, Cognitive Science Society
52. van Rooij, I. et al. (2011) Intentional communication: computa-
tionally easy or difcult? Front. Hum. Neurosci. 5, 52
53. Humphries, M.D. et al. (2012) Dopaminergic control of the explo-
rationexploitation trade-off via the basal ganglia. Front. Neuro-
sci. 6, 9
54. Kayser, A.S. et al. (2015) Dopamine, locus of control, and the
explorationexploitation tradeoff. Neuropsychopharmacology
40, 454462
55. Pezzulo, G. et al. (2014) Internally generated sequences in learn-
ing and executing goal-directed behavior. Trends Cogn. Sci. 18,
647657
56. Garrod,S. et al. (2007) Foundationsof representation:where might
graphical symbol systems come from? Cogn. Sci. 31, 961987
57. Gershman, S.J. and Niv, Y. (2010) Learning latent structure:
carving nature at its joints. Curr. Opin. Neurobiol. 20, 251256
58. Tomasello, M. (2008) Origins of Human Communication, MIT
Press
59. Centola, D. and Baronchelli, A. (2015) The spontaneous emer-
gence of conventions: an experimental study of cultural evolution.
Proc. Natl. Acad. Sci. U.S.A. 112, 19891994
60. Kumaran, D. et al. (2009) Tracking the emergence of conceptual
knowledge during human decision making. Neuron 63, 889901
61. Siegal, M. and Varley, R. (2002) Neural systems involved in
theory of mind.Nat. Rev. Neurosci. 3, 463471
62. Hoffman, P. et al. (2014) The anterior temporal lobes are critically
involved in acquiring new conceptual knowledge: evidence for
impaired feature integration in semantic dementia. Cortex 50,
1931
63. Hari, R. et al. (2013) Synchrony of brains and bodies during
implicit interpersonal interaction. Trends Cogn. Sci. 17, 105106
64. Jiang, J. et al. (2012) Neural synchronization during face-to-face
communication. J. Neurosci. 32, 1606416069
65. Rizzolatti, G. and Craighero, L. (2004) The mirror-neuron system.
Annu. Rev. Neurosci. 27, 169192
66. Menenti, L. et al. (2012) Toward a neural basis of interactive
alignment in conversation. Front. Hum. Neurosci. 6, 185
67. Kuhlen, A.K. et al. (2012) Content-specic coordination of lis-
tenersto speakersEEG during communication. Front. Hum.
Neurosci. 6, 266
68. Lambon Ralph, M.A. et al. (2010) Coherent concepts are com-
puted in the anterior temporal lobes. Proc. Natl. Acad. Sci. U.S.A.
107, 27172722
69. Milne, E. and Grafman, J. (2001) Ventromedial prefrontal cortex
lesions in humans eliminate implicit gender stereotyping. J. Neu-
rosci. 21, RC150
70. Sabbagh, M.A. (1999) Communicative intentions and language:
evidence from right-hemisphere damage and autism. Brain Lang.
70, 2969
Trends in Cognitive Sciences, Month Year, Vol. xx, No. yy 11
TICS 1521 No. of Pages 12
71. Beeman, M. (1993) Semantic processing in the right hemisphere
may contribute to drawing inferences from discourse. Brain
Lang. 44, 80120
72. Stolk, A. et al. (2014) Cereb ral cohere nce betwee n communi-
cators mar ks the emergenc e of meaning. Proc. Natl. Acad. Sci.
U.S.A. 111, 1818318188
73. Stolk, A. (2014) In sync: metaphor, mechanism or marker of
mutual understanding? J. Neurosci. 34, 53975398
74. Jarsky, T. et al. (2005) Conditional dendritic spike propagation
following distal synaptic activation of hippocampal CA1 pyrami-
dal neurons. Nat. Neurosci. 8, 16671676
75. Behabadi, B.F. et al. (2012) Location-dependent excitatory syn-
aptic interactions in pyramidal neuron dendrites. PLoS Comput.
Biol. 8, e1002599
76. Larkum, M. (2013) A cellular mechanism for cortical associations:
an organizing principle for the cerebral cortex. Trends Neurosci.
36, 141151
77. Smith, S.L. et al. (2013) Dendritic spikes enhance stimulus
selectivity in cortical neurons in vivo.Nature 503, 115120
78. Bittner, K.C. et al. (2015) Conjunctive input processing drives
feature selectivity in hippocampal CA1 neurons. Nat. Neurosci.
18, 11331142
79. Gaskell, M.G. (ed.) (2007) The Oxford Handbook of Psycholin-
guistics, Oxford University Press
80. Silbert, L.J. et al. (2014) Coupled neural systems underlie the
production and comprehension of naturalistic narrative speech.
Proc. Natl. Acad. Sci. U.S.A. 111, E4687E4696
81. Schilbach, L. et al. (2013) Toward a second-person neurosci-
ence. Behav. Brain Sci. 36, 393414
82. Di Cesare, G. et al. (2015) Expressing our internal states and
understanding those of others. Proc. Natl. Acad. Sci. U.S.A. 112,
1033110335
83. Hrdy, S.B. (2009) Mothers and Others: The Evolutionary Origins
of Mutual Understanding, Harvard University Press
84. Carpendale, J.I. and Lewis, C. (2004) Constructing an under-
standing of mind: the development of children's social under-
standing within social interaction. Behav. Brain Sci. 27, 7996
discussion 96-151
85. de Rosnay, M. and Hughes, C. (2006) Conversation and theory
of mind: do children talk their way to socio-cognitive understand-
ing? Br. J. Dev. Psychol. 24, 737
86. Mates, A.W. (2010) Using social decits in frontotemporal
dementia to develop a neurobiology of person reference. In
Language, Interaction and Frontotemporal Dementia: Reverse
Engineering the Social Mind (Mates, A.W. et al., eds), pp. 139
166, Equinox
87. Healey, M.L. et al. (2015) Getting on the same page: the neural
basis for social coordination decits in behavioral variant fronto-
temporal degeneration. Neuropsychologia 69, 5666
88. Levesque, H.J. (2014) On our best behaviour. Artif. Intell. 212,
2735
89. Fung, P. (2015) Robots with heart. Sci. Am. 313, 6063
90. Bornstein, M.H. et al. (2015) Motherinfant contingent vocaliza-
tions in 11 countries. Psychol. Sci. 26, 12721284
91. Stivers, T. et al. (2009) Universals and cultural variation in
turn-taking in conversation. Proc. Natl. Acad. Sci. U.S.A. 106,
1058710592
92. Blokpoel, M. (2015) Understanding Understanding: A Computa-
tional-Level Perspective. Donders Graduate School for Cognitive
Neuroscience Series 195, Radboud Universiteit
93. Stolk, A. et al. (2015) On the generation of shared symbols. In
Cognitive Neuroscience of Natural Language Use (Willems, R.,
ed.), pp. 201227, Cambridge University Press
94. Stolk, A. et al. (2013) Early social experience predicts referential
communicative adjustments in ve-year-old children. PLoS ONE
8, e72667
95. Feiler, L. and Camerer, C.F. (2010) Code creation in endogenous
merger experiments. Econ. Inq. 48, 337352
96. Selten, R. and Warglien, M. (2007) The emergence of simple
languages in an experimental coordination game. Proc. Natl.
Acad. Sci. U.S.A. 104, 73617366
97. Brennan, S.E. and Clark, H.H. (1996) Conceptual pacts and
lexical choice in conversation. J. Exp. Psychol. Learn. Mem.
Cogn. 22, 14821493
98. Blokpoel, M. et al. (2012) Recipient design in human communi-
cation: simple heuristics or perspective taking? Front. Hum.
Neurosci. 6, 253
99. Abitbol, R. et al. (2015) Neural mechanisms underlying contextual
dependency of subjective values: converging evidence from
monkeys and humans. J. Neurosci. 35, 23082320
100. Luczak, A. et al. (2013) Gating of sensory input by spontaneous
cortical activity. J. Neurosci. 33, 16841695
101. Buzsaki, G. et al. (2012) The origin of extracellular elds and
currents EEG, ECoG LFP and spikes. Nat. Rev. Neurosci. 13,
407420
102. Miller, K.J. (2010) Broadband spectral change: evidence for a
macroscale correlate of population ring rate? J. Neurosci. 30,
64776479
103. Manning, J.R. et al. (2009) Broadband shifts in local eld potential
power spectra are correlated with single-neuron spiking in
humans. J. Neurosci. 29, 1361313620
104. Arieli, A. et al. (1996) Dynamics of ongoing activity: explanation of
the large variability in evoked cortical responses. Science 273,
18681871
105. Okun, M. et al. (2010) The subthreshold relation between cortical
local eld potential and neuronal ring unveiled by intracellular
recordings in awake rats. J. Neurosci. 30, 44404448
106. Stolk, A. et al. (2015) Altered communicative decisions following
ventromedial prefrontal lesions. Curr. Biol. 25, 14691474
107. Stolk, A. et al. (2014) Understanding communicative actions: a
repetitive TMS study. Cortex 51, 2534
12 Trends in Cognitive Sciences, Month Year, Vol. xx, No. yy
... What computational and neuronal mechanisms enable people to converge to a shared meaning of their confidence expressions in interactive decision making despite the extensively documented neural and cognitive diversity? This question drives at the heart of recent efforts to understand the neurobiology of how people adapt their communication to their beliefs about their interaction partner (Stolk et al., 2016). A number of studies have provided compelling empirical evidence of brain-to-brain coupling that could underlie adaptive communication of shared beliefs Honey et al., 2012;Hasson et al., 2004;Dikker et al., 2014;Konvalinka et al., 2010). ...
... For communication science, by examining perceptual decision making under uncertainty in social context, we created a laboratory model in which the goal of communication was to arrive at a shared belief about uncertainty (rather than creating a look-up table for the meaning of actions [Stolk et al., 2016;Silbert et al., 2014;Honey et al., 2012]). In this way, we could employ the extensive theoretical, behavioral, and neurobiological body of knowledge in decision science (Pouget et al., 2016;Adler and Ma, 2018;Aitchison et al., 2015;Hanks and Summerfield, 2017;Gold and Shadlen, 2007;Ruff and Fehr, 2014;Schall, 2019;Kiani et al., 2014;Kelly and O'Connell, 2013;Shadlen and Newsome, 2001;Resulaj et al., 2009;Sanders et al., 2016;Wei and Wang, 2015;Eldar et al., 2013;Urai et al., 2017;Yeung and Summerfield, 2012;Fleming and Daw, 2017;Kiani and Shadlen, 2009) to construct a mechanistic neural hypothesis for interpersonal alignment. ...
Article
Full-text available
Private, subjective beliefs about uncertainty have been found to have idiosyncratic computational and neural substrates yet, humans share such beliefs seamlessly and cooperate successfully. Bringing together decision making under uncertainty and interpersonal alignment in communication, in a discovery plus pre-registered replication design, we examined the neuro-computational basis of the relationship between privately held and socially shared uncertainty. Examining confidence-speed-accuracy trade-off in uncertainty-ridden perceptual decisions under social vs isolated context, we found that shared (i.e. reported confidence) and subjective (inferred from pupillometry) uncertainty dynamically followed social information. An attractor neural network model incorporating social information as top-down additive input captured the observed behavior and demonstrated the emergence of social alignment in virtual dyadic simulations. Electroencephalography showed that social exchange of confidence modulated the neural signature of perceptual evidence accumulation in the central parietal cortex. Our findings offer a neural population model for interpersonal alignment of shared beliefs.
... Finally, the conceptual alignment framework offers a neurocognitive account of how individuals with unique perspectives can come to share a common conceptual understanding during a conversation (Stolk et al., 2016(Stolk et al., , 2023. Through their behaviors, conversational partners engage in a process of probing, aligning, and shaping each other's latent conceptual structures, creating a shared conceptual space that allows them to focus on relevant details and coordinate their next steps in the dialogue. ...
... By minimizing participants' access to linguistic and other social conventions, this work reveals the underlying nonlinguistic adaptations that forge common ground (Hawkins et al., 2023). A key insight is that effective communication requires continually building a shared context, informed by what is presumed to be known and believed by the other (Stolk et al., 2016(Stolk et al., , 2023. This joint construction process is critical for achieving group consensus and its neural corollary of interbrain alignment. ...
Article
For over a century, psychology has focused on uncovering mental processes of a single individual. However, humans rarely navigate the world in isolation. The most important determinants of successful development, mental health, and our individual traits and preferences arise from interacting with other individuals. Social interaction underpins who we are, how we think, and how we behave. Here we discuss the key methodological challenges that have limited progress in establishing a robust science of how minds interact and the new tools that are beginning to overcome these challenges. A deep understanding of the human mind requires studying the context within which it originates and exists: social interaction.
... Representation alignment refers to the extent to which the internal representations of two or more information processing systems are aligned [14]. This idea has been explored under various terminologies across different contexts, such as latent space alignment [19], conceptual alignment [20], systems alignment [21], representational similarity [22,23], model alignment [24], and representational alignment [14]. Christian [4] provides a narrative exploration of the challenges and opportunities in human-AI interaction, emphasizing the importance of designing AI systems that understand and adapt to human emotions and social norms. ...
Preprint
Full-text available
Aligning large language models (LLMs) behaviour with human intent is critical for future AI. An important yet often overlooked aspect of this alignment is the perceptual alignment. Perceptual modalities like touch are more multifaceted and nuanced compared to other sensory modalities such as vision. This work investigates how well LLMs align with human touch experiences using the "textile hand" task. We created a "Guess What Textile" interaction in which participants were given two textile samples -- a target and a reference -- to handle. Without seeing them, participants described the differences between them to the LLM. Using these descriptions, the LLM attempted to identify the target textile by assessing similarity within its high-dimensional embedding space. Our results suggest that a degree of perceptual alignment exists, however varies significantly among different textile samples. For example, LLM predictions are well aligned for silk satin, but not for cotton denim. Moreover, participants didn't perceive their textile experiences closely matched by the LLM predictions. This is only the first exploration into perceptual alignment around touch, exemplified through textile hand. We discuss possible sources of this alignment variance, and how better human-AI perceptual alignment can benefit future everyday tasks.
... A misunderstanding attack is a nuanced form of squatting attack where the goal is not to activate a malicious service, but to induce errors within a legitimate service. These attacks exploit the limitations of VCS in accurately interpreting words outside their programmed understanding [134]. For example, Siri might misinterpret "bank" in the context of "river bank". ...
Preprint
The integration of Voice Control Systems (VCS) into smart devices and their growing presence in daily life accentuate the importance of their security. Current research has uncovered numerous vulnerabilities in VCS, presenting significant risks to user privacy and security. However, a cohesive and systematic examination of these vulnerabilities and the corresponding solutions is still absent. This lack of comprehensive analysis presents a challenge for VCS designers in fully understanding and mitigating the security issues within these systems. Addressing this gap, our study introduces a hierarchical model structure for VCS, providing a novel lens for categorizing and analyzing existing literature in a systematic manner. We classify attacks based on their technical principles and thoroughly evaluate various attributes, such as their methods, targets, vectors, and behaviors. Furthermore, we consolidate and assess the defense mechanisms proposed in current research, offering actionable recommendations for enhancing VCS security. Our work makes a significant contribution by simplifying the complexity inherent in VCS security, aiding designers in effectively identifying and countering potential threats, and setting a foundation for future advancements in VCS security research.
... Wang et al. (2023), for example, demonstrated similar rTPJ-rTPJ p coupling when participants competed for the entire reward (indicating high attentiveness); and rTPJ-rSTG coupling when participants collaborated and shared the reward (97.5% trusting the partner, indicating a high level of trust in the partner and therefore lower attention demand). Similarly, in the Token-Coordination task (Stolk et al., 2014), the rSTG-rSTG coherence was observed among highly collaborative pairs, suggesting that such coupling is the foundation of shared meaning (Stolk et al., 2016). For further comparison, Figure 5 summarizes our rTPJ-rSTG coherence results (red for the WIP, yellow for the WNP), along with reported coherence-ROI sites from two previously published articles (Wang et al., 2023 green/cooperation and blue/competition coherence ROIs, and Stolk et al., 2014, black ROI). ...
Article
Unselfishness is admired, especially when collaborations between groups of various scales are urgently needed. However, its neural mechanisms remain elusive. In a tri‐MRI dyad‐hyperscanning experiment involving 26 groups, each containing 4 participants as two rotating pairs in a coordination game, we sought to achieve reciprocity, or “winning in turn by the two interacting players,” as the precursor to unselfishness. Due to its critical role in social processing, the right temporal–parietal junction (rTPJ) was the seed for both time domain (connectivity) and frequency domain (i.e., coherence) analyses. For the former, negative connectivity between the rTPJ and the mentalizing network areas (e.g., the right inferior parietal lobule, rIPL) was identified, and such connectivity was further negatively correlated with the individual's final gain, supporting our task design that “rewarded” the reciprocal participants. For the latter, cerebral coherences of the rTPJs emerged between the interacting pairs (i.e., within‐group interacting pairs), and the coupling between the rTPJ and the right superior temporal gyrus (rSTG) between the players who were not interacting with each other (i.e., within‐group noninteracting pairs). These coherences reinforce the hypotheses that the rTPJ–rTPJ coupling tracks the collaboration processes and the rTPJ–rSTG coupling for the emergence of decontextualized shared meaning. Our results underpin two social roles (inferring others’ behavior and interpreting social outcomes) subserved by the rTPJ‐related network and highlight its interaction with other‐self/other‐concerning brain areas in reaching co‐benefits among unselfish players.
... Recently, a technique called hyperscanning, which allows the simultaneous measurement of brain activation across multiple participants, has been used to assess the neural mechanisms of social interaction in more natural contexts (for review, see Babiloni and Astolfi, 2014;Liu and Pelowski, 2014). Previous hyperscanning studies have consistently shown that when two brains are synchronized, demonstrating inter-brain neural synchronization (INS) across interacting dyads, information can be shared and exchanged more efficiently (Balconi et al., 2017;Nozawa et al., 2021;Reinero et al., 2021;Schoot et al., 2016;Stolk et al., 2016). ...
Article
Full-text available
Collaboration is a critical skill in everyday life. It has been suggested that collaborative performance may be influenced by social factors such as interpersonal distance, which is defined as the perceived psychological distance between individuals. Previous literature has reported that close interpersonal distance may promote the level of selfother integration between interacting members, and in turn, enhance collaborative performance. These studies mainly focused on interdependent collaboration, which requires high levels of shared representations and self-other integration. However, little is known about the effect of interpersonal distance on independent collaboration (e.g., the joint Simon task), in which individuals perform the task independently while the final outcome is determined by the parties. To address this issue, we simultaneously measured the frontal activations of ninety-four pairs of participants using a functional near-infrared spectroscopy (fNIRS)-based hyperscanning technique while they performed a joint Simon task. Behavioral results showed that the Joint Simon Effect (JSE), defined as the RT difference between incongruent and congruent conditions indicating the level of self-other integration between collaborators, was larger in the friend group than in the stranger group. Consistently, the inter-brain neural synchronization (INS) across the dorsolateral and medial parts of the prefrontal cortex was also stronger in the friend group. In addition, INS in the left dorsolateral prefrontal cortex negatively predicted JSE only in the friend group. These results suggest that close interpersonal distance may enhance the shared mental representation among collaborators, which in turn influences their collaborative performance.
... Specifically, it has been posited that listening to the speaker's retrieval will lead the listener to retrieve the same exemplars (i.e., Rp + exemplars) concurrently and covertly (Cuc et al., 2007), which will further result in similar outcomes of memory between individuals in the subsequent individual recall phase (Stone et al., 2013). Recent dual-brain findings in other cognitive tasks have suggested that shared neural representations of the external stimuli or internal cognitive process may be associated with similar brain activity between individuals across time, showing interpersonal neural synchronization (INS) (Hasson et al., 2012;Jiang et al., 2021;Stolk et al., 2016). In support of this proposition, recent hyperscanning studies on social collaboration have shown that shared cognitive representations of concept (Spiegelhalder et al., 2014;Stolk et al., 2013Stolk et al., , 2014, knowledge (Barreto et al., 2021;Sun et al., 2020;Zheng et al., 2018) or even social affiliations (Mayseless et al., 2019;Zheng et al., 2020) are associated with enhanced INS between individuals. ...
Article
Although collaborative remembering is a ubiquitous feature of human beings, its underlying neurocognitive process is not well understood. Here we hypothesized that interpersonal neural synchronization (INS) might underlie collaborative remembering, while real collaboration as opposed to other modes of offline collaboration should enhance INS and facilitate mnemonic similarity. To test these hypotheses, brain activity was measured simultaneously from two individuals who performed a group-based selective retrieval practice task either in a real collaboration or in a pseudo-collaboration, i.e., an individual performed the task together with a pre-recorded audio. The results showed that the memory of two individuals converged to a greater level than the chance level in real collaboration but not in control condition. Moreover, collaborative remembering was associated with significant INS increase in the prefrontal cortex (PFC) relative to the baseline in the real collaboration only. Additionally, INS increase was significantly greater in the real collaboration than in control condition. Finally, the PFC's INS increase was positively correlated with and could accurately predict the level of mnemonic similarity in real collaboration. These findings support the hypothesis that the enhanced INS underlies the cognitive process of collaborative remembering.
Preprint
Full-text available
Group identification may influence collective behaviors and result in variations in collective performance. However, the evidence for this hypothesis and the neural mechanisms involved remain elusive. To this end, we conducted a study using both single-brain activation and multi-brain synchronization analyses to investigate how group identification influences collective problem-solving in a murder mystery case. Our results showed that groups with high levels of identification performed better individually compared to those with low identification, as supported by single-brain activation in the dorsolateral prefrontal cortex (DLPFC). Furthermore, high-identification groups also showed enhanced collective performance, supported by within-group neural synchronization (GNS) in the orbitofrontal cortex (OFC). The DLPFC-OFC connectivity played a crucial role in linking individual and collective performance. Overall, our study provides a two-in-one neural model to explain how group identification affects collective decision-making processes, offering valuable insights into the dynamics of group interactions.
Preprint
Full-text available
How language began is one of the oldest questions in science. This chapter reviews the evidence for the vocal-first, gesture-first and multimodal theories of language origin. First, it considers the nature of signs, and argues that signs exist on a continuum that ranges from iconic to symbolic. Next, a general theoretical model of language creation is outlined, and the naturalistic and experimental evidence for each theory is reviewed. While there is naturalistic evidence to support each theory, the experimental evidence supports the gesture-first theory of language origin. The experimental work indicates the advantage of gesture lies in its affordance for the production of iconic signs, which are crucial to communication success. The chapter closes by considering a challenge for the gesture-first theory: why speech is the primary communication modality of modern humans.
Article
Cocreating meaning in collaboration is challenging. Success is often determined by people's abilities to coordinate their language to converge upon shared mental representations. Here we explore one set of low‐level linguistic behaviors, linguistic alignment, that both emerges from, and facilitates, outcomes of high‐level convergence. Linguistic alignment captures the ways people reuse, that is, “align to,” the lexical, syntactic, and semantic forms of others' utterances. Our focus is on the temporal change of multi‐level linguistic alignment, as well as how alignment is related to communicative outcomes within a unique collaborative problem‐solving paradigm. The primary task, situated within a virtual educational video game, requires creative thinking between three people where the paths for possible solutions are highly variable. We find that over time interactions are marked by decreasing lexical and syntactic alignment, with a trade‐off of increasing semantic alignment. However, greater semantic alignment does not translate into better team performance. Overall, these findings provide greater clarity on the role of linguistic coordination within complex and dynamic collaborative problem‐solving tasks.
Article
Full-text available
Significance Vitality form is a term that describes the manner with which actions are performed. Despite their crucial importance in interpersonal communication, vitality forms have been almost completely neglected in neuroscience. Here, using a functional MRI technique, we investigated the neural correlates of vitality forms in three tasks: action observation, imagination, and execution. We found that, in all three tasks, there is a common specific activation of the dorsocentral sector of the insula in addition to the parietofrontal network that is typically active during arm movements production and observation. Thus, the dorsocentral part of the insula seems to represent a fundamental and previously unsuspected node that modulates the cortical motor circuits, allowing individuals to express their vitality forms and understand those of others.
Book
Available again, an influential book that offers a framework for understanding visual perception and considers fundamental questions about the brain and its functions. David Marr's posthumously published Vision (1982) influenced a generation of brain and cognitive scientists, inspiring many to enter the field. In Vision, Marr describes a general framework for understanding visual perception and touches on broader questions about how the brain and its functions can be studied and understood. Researchers from a range of brain and cognitive sciences have long valued Marr's creativity, intellectual power, and ability to integrate insights and data from neuroscience, psychology, and computation. This MIT Press edition makes Marr's influential work available to a new generation of students and scientists. In Marr's framework, the process of vision constructs a set of representations, starting from a description of the input image and culminating with a description of three-dimensional objects in the surrounding environment. A central theme, and one that has had far-reaching influence in both neuroscience and cognitive science, is the notion of different levels of analysis—in Marr's framework, the computational level, the algorithmic level, and the hardware implementation level. Now, thirty years later, the main problems that occupied Marr remain fundamental open problems in the study of perception. Vision provides inspiration for the continuing efforts to integrate knowledge from cognition and computation to understand vision and the brain.
Book
This book examines the young science of psycholinguistics, which attempts to uncover the mechanisms and representations underlying human language. This interdisciplinary field has seen massive developments over the past decade, with a broad expansion of the research base, and the incorporation of new experimental techniques such as brain imaging and computational modelling. The result is that real progress is being made in the understanding of the key components of language in the mind. This book brings together the views of seventy-five leading researchers to provide a review of the current state of the art in psycholinguistics. The contributors are eminent in a wide range of fields, including psychology, linguistics, human memory, cognitive neuroscience, bilingualism, genetics, development, and neuropsychology. Their contributions are organised into six themed sections, covering word recognition, the mental lexicon, comprehension and discourse, language production, language development, and perspectives on psycholinguistics.
Article
Carpendale & Lewis (C&L) stress the importance of social interaction for social understanding, but focus on the adult-child relationship. in the present commentary, we discuss the development of social understanding within early peer relationships. We argue that peer interaction stretches the limits of early social understandmg, thereby providing both unique challenges and unique opportunities for constructing an understanding of others' minds.
Article
Article
Before we can share our lives with machines, we must teach them to understand and mimic human emotion
Article
After growing up together, and mostly growing apart in the second half of the 20th century, the fields of artificial intelligence (AI), cognitive science, and neuroscience are reconverging on a shared view of the computational foundations of intelligence that promotes valuable cross-disciplinary exchanges on questions, methods, and results. We chart advances over the past several decades that address challenges of perception and action under uncertainty through the lens of computation. Advances include the development of representations and inferential procedures for large-scale probabilistic inference and machinery for enabling reflection and decisions about tradeoffs in effort, precision, and timeliness of computations. These tools are deployed toward the goal of computational rationality: identifying decisions with highest expected utility, while taking into consideration the costs of computation in complex real-world problems in which most relevant calculations can only be approximated. We highlight key concepts with examples that show the potential for interchange between computer science, cognitive science, and neuroscience.