Development of multisensory integration from the perspective of the individual neuron

Abstract

The ability to use cues from multiple senses in concert is a fundamental aspect of brain function. It maximizes the brain's use of the information available to it at any given moment and enhances the physiological salience of external events. Because each sense conveys a unique perspective of the external world, synthesizing information across senses affords computational benefits that cannot otherwise be achieved. Multisensory integration not only has substantial survival value but can also create unique experiences that emerge when signals from different sensory channels are bound together. However, neurons in a newborn's brain are not capable of multisensory integration, and studies in the midbrain have shown that the development of this process is not predetermined. Rather, its emergence and maturation critically depend on cross-modal experiences that alter the underlying neural circuit in such a way that optimizes multisensory integrative capabilities for the environment in which the animal will function.

Full text

Development of multisensory integration from the perspective of
the individual neuron
Barry E. Stein, Terrence R. Stanford, and Benjamin A. Rowland
Department of Neurobiology and Anatomy, Wake Forest School of Medicine, Winston-Salem,
North Carolina 27157, USA
Abstract
The ability to use cues from multiple senses in concert is a fundamental aspect of brain function. It
maximizes the brain’s use of the information available to it at any given moment and enhances the
physiological salience of external events. Because each sense conveys a unique perspective of the
external world, synthesizing information across senses affords computational benefits that cannot
otherwise be achieved. Multisensory integration not only has substantial survival value but can
also create unique experiences that emerge when signals from different sensory channels are
bound together. However, neurons in a newborn’s brain are not capable of multisensory
integration, and studies in the midbrain have shown that the development of this process is not
predetermined. Rather, its emergence and maturation critically depend on cross-modal experiences
that alter the underlying neural circuit in such a way that optimizes multisensory integrative
capabilities for the environment in which the animal will function.
The environment is rife with events that emit multiple types of energy (for example,
electromagnetic radiation and pressure waves), but all can contain information about food,
shelter, mates and/or danger. Even small enhancements in the ability to detect such signals
and evaluate these biologically significant events can have a major impact on the survival of
a species. Thus, it is endlessly fascinating to speculate about how selective pressures have
spurred the evolution and aggregation of different sensory systems that maximize
information gathering within different ecological niches. Each of an organism’s sensory
systems is tuned to a different form of energy, and they can compensate for one another
when necessary, as when hearing and touch compensate for vision under conditions of
darkness. Given the diversity of possible ecological niches, it is perhaps not surprising that
evolution has produced animals with widely divergent appearances, senses and sensory
capabilities. However, no matter how exotic these variants may seem, they share a common
innovation, one that was likely presaged by our single-celled progenitor: the ability to use
their senses synergistically1.
© 2014 Macmillan Publishers Limited. All rights reserved
Correspondence to B.E.S. bestein@wakehealth.edu.
Competing interests statement
The authors declare no competing interests.
NIH Public Access
Author Manuscript
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
Published in final edited form as:
Nat Rev Neurosci
. 2014 August ; 15(8): 520–535.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

Biologically significant events are often registered by more than one sense. Because each
sense independently derives and transmits a report of the event, more accurate perceptual
evaluations of the event and behavioural decisions can be made through the synthesis of
their different sensory signals2–4. This process, called multisensory integration, increases the
collective impact of biologically significant signals on the brain and enables the organism to
achieve performance capabilities that it could not otherwise realize. Consequently,
multisensory integration has enormous survival value and has undoubtedly played a far
more important part in the evolutionary histories of extant species than is currently
recognized. It is also a ubiquitous process that profoundly affects how we perceive the world
and the decisions we make to best meet its challenges. However, we are nearly always
unaware of this process. More often than not, the judgements we think we are making based
on information from a single sense, such as vision, are strongly influenced by seemingly
irrelevant but informative cues from other senses such as hearing and touch. Sensory
judgements are rarely exclusive to a single sense because multiple sensory channels
converge on and share the use of the neural processes that mediate perception and action. It
is little wonder that interest in the operational features of multisensory integration has
become so widespread.
How a developing nervous system creates this capability to use the senses interactively is
even less well understood than the various functional domains in which it will ultimately be
expressed. Although behavioural studies have shown that neonates can detect certain cross-
modal correspondences very early in life5, physiological studies indicate that the capacity to
integrate information across the senses is not an inherent feature of the newborn’s brain.
Rather, as is discussed in detail here using information derived from interactions between
the visual, auditory and somatosensory systems, this capability develops in an experience-
dependent manner during early postnatal life. During this window of time, the operational
parameters of multisensory integration are customized to the features of the local
environment6; that is, multisensory processing rules are configured to efficiently process the
cross-modal signal relationships that the organism encounters. This adaptation occurs
alongside the maturation of the contributing unisensory systems. However, multisensory
development does not require that each individual sense reaches its maturational end point,
and the functional development of an individual sense does not require its interaction with
another. Rather, multisensory development and unisensory development are interconnected
but parallel processes that, at the level of the circuit, often have different computational
targets and constraints. It may not even be appropriate to think of multisensory
‘development’ as strictly an early-life process. These issues are discussed in this Review.
Defining multisensory integration
Multisensory integration refers to the process by which inputs from two or more senses are
combined to form a product that is distinct from, and thus cannot be easily ‘deconstructed’
to reconstitute the components from which it is created7. Whether considering neural signals
or behavioural performance, this is defined operationally as a statistically significant
difference between the response evoked by a cross-modal combination of stimuli and that
evoked by the most effective of its components individually1. With respect to single-neuron
physiology, this comparison is made between the total number of impulses or firing rates
Stein et al. Page 2
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

evoked by stimuli and their combinations1,7–9. This could result in response enhancement or
response depression. These physiological changes produce alterations in sensation and
perception, as well as the behaviours dependent on them. Multisensory enhancement, which
is the most reliable index of multisensory integration (and will be discussed here most
extensively) may reflect computations that yield response magnitudes that are equal to, less
than or greater than the sum of the responses to the individual component stimuli7. In
behaviour, performance enhancements are often quantified by evaluating differences in the
accuracy and speed of detection, localization and/or identification of stimuli3,4,10–26. In
short, multisensory integration refers to a broad class of computations involving multiple
sensory modalities in which information is integrated to produce an enhanced (or degraded)
response. Other computations involving multisensory processing, such as comparing the
features of a stimulus (for example, its shape) across modalities or detecting certain cross-
modal correspondences in timing or rhythm27, require that the comparators maintain their
identity rather than being fused into a single product. As a consequence, they will probably
have different, albeit currently unknown, underlying mechanisms and developmental time
courses.
The multisensory superior colliculus neuron
It was in the 1970s that systematic efforts were first begun to understand the underlying
neural circuits through which multisensory integration is achieved, the behavioural
manifestations that reflect its operation and how this capability becomes instantiated in
individual neurons during early life. These studies used the multisensory neuron in the cat
superior colliculus as a model. Since that time, multisensory neurons have been identified in
many brain areas and species1,28–39, but most of the information about their development
has been derived from studies of the cat superior colliculus. This is, in part, because it seems
to be an excellent model.
Neurons in the deep layers of the superior colliculus are primary sites of multisensory
convergence (neurons in overlying superficial layers are purely visual), thereby affording
the opportunity to gain insight into the circuit requirements and initial processes that are
involved in integrating cross-modal inputs before the resultant products are shared with
other brain areas. Its utility as a model is also facilitated by the presence of many such
neurons and their involvement in the well-defined behavioural roles of the superior
colliculus: mediating the animal’s detection and localization of external events and its
orientation to them40–49, which are behaviours that gradually mature during postnatal
life50,51. As there is a comparatively high incidence of multisensory superior colliculus
neurons, it is practical to target them in electrophysiological experiments, and as most of
them are also output neurons that project to motor regions of the brainstem and spinal
cord52, it is possible to evaluate their properties in the context of their influence on overt
behaviour1,53–55. The computations that describe these relationships and the factors
affecting them have also been described using signal detection and Bayesian
frameworks56–58 that can inform future empirical studies. Furthermore, as the processing
capabilities of superior colliculus neurons at birth are immature59,60, their development can
be followed during postnatal life, when many factors affecting their ultimate functional
capability can be experimentally manipulated.
Stein et al. Page 3
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

The overarching functional development of the superior colliculus, of which its multisensory
integration capability is one important facet, reflects an architecture that facilitates efficient
sensorimotor transduction. Each of its three sensory representations for sight, hearing and
touch (that is, visual, auditory and somatosensory) develops so that it is laid out in a map-
like manner61–65, with all of the maps in overlapping spatial register with one another. This
is a general feature of the superior colliculus that seems to be independent of species66–74.
The different maps are constituted, in large part, by multisensory neurons that have
modality-specific receptive fields in register with one another. The sensory maps, in turn,
overlap with a common motor map, which also involves many of the same multisensory
neurons and through which orientation of the eyes, ears, head and limbs can be initiated. It is
through this elegantly simple design that a salient environmental event excites a localized
region of the map (or maps) to register an appropriate sensory report and initiate a
coordinated overt response (FIG. 1) regardless of which sense or combination of senses was
activated1.
Convergence does not guarantee integration
It is important to note that although the capability of multisensory neurons to integrate their
inputs will markedly facilitate the sensorimotor role of the superior colliculus, this capability
does not inevitably result from the convergence of two or more sensory inputs onto a
common neuron. Multisensory convergence develops in young animals long before those
target neurons are capable of integrating their inputs. When an event is registered by more
than one sense in such animals, the default operation of their (naïve) multisensory neurons is
to respond as if the cross-modal signals were not complementary: the response to a cross-
modal stimulus is no better than it is to the most effective of its individual component
stimuli and is often a weighted average of those responses, and thus less than the most
robust unisensory response9,75–80. This may seem counterintuitive if one expects that the
default mode of a naïve neuron is to sum its cross-modal inputs. However, the empirical data
reinforce the need to compare a neuron’s multisensory response with the most effective of
its unisensory component responses in order to examine the development and expression of
multisensory integration. There are also several circumstances in which cross-modal
stimulus configurations are not integrated in the mature animal. Once again, in each of these
cases, neurons yield responses no greater than those to one of the component stimuli. Thus,
although most multisensory superior colliculus neurons ultimately develop the ability to
integrate their different sensory inputs and must be multisensory to do so, these two
neuronal characteristics are not inextricably bound. In fact, their maturational asynchrony
enables their independent evaluation.
Guiding principles of multisensory integration
Studies of adult cat superior colliculus neurons have yielded three general operating
principles, or ‘rules of thumb’, for multisensory integration. The first two involve space and
time. Cross-modal (for example, visual–auditory) cues that are in close spatial and temporal
register generally enhance the responses of multisensory neurons, whereas those that are
spatially or temporally disparate often elicit response depression or fail to be integrated81,82.
The third principle, that of ‘inverse effectiveness’ (REF. 83) (BOX 1), describes the
Stein et al. Page 4
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

observation that proportionately greater effects of cross-modal cues are obtained when those
individual cues are weakly effective. Thus, the magnitude of multisensory integration is
inversely related to the efficacy of the stimuli being integrated7.
These principles are consistent with the presumptive ‘benefit’ of multisensory integration in
this context: improving the ability to detect an external event, localizing it in space and
using it as a target for a superior colliculus-mediated orientation response. The spatial and
temporal components relate to the idea that because cross-modal stimuli are proximate in
space and time, they are most likely to be derived from the same event. Enhancing the
physiological impact of the initiating event through multisensory integration, especially
when it provides only very weak cues, increases the probability that it will generate a
superior colliculus-mediated response. By contrast, cross-modal stimuli that are disparate
are more likely to belong to unrelated or competing events and will either fail to interact or
will interact competitively, thereby producing response depression83–85. Building this
organizational framework during early life involves multiple steps. Afferents carrying
signals that refer to common regions of sensory space are first routed onto common target
neurons in topographically appropriate patterns. Then, the circuit configures its internal
computations so that convergent cross-modal signals that are most probably derived from
the same event can interact in complementary ways, whereas others either fail to interact or
compete with one another. As shown below, the elaboration and refinement of this
framework are dependent on several postnatal factors.
Box 1
Enhancement, inverse effectiveness and superadditivity
The primary function of the superior colliculus is to guide orienting behaviour towards
salient external stimuli. Given that an organism can orient to but one stimulus at a time, it
is reasonable to view the sensory environment at any given moment as consisting of a
myriad of sensory-specific competitors, each vying to be the goal of the next orienting
movement. With this in mind, the phenomena of multisensory enhancement and
multisensory depression are readily understood as means towards resolving competition
between mutually exclusive alternatives. In other words, stimuli from different modalities
that are spatially congruent enhance the physiological salience of their commonly held
spatial location, whereas those emanating from disparate locations mutually degrade.
With respect to the activity of single superior colliculus multisensory neurons,
multisensory enhancement is always defined as a response to a cross-modal stimulus that
exceeds the response to either of its modality-specific components; however, it is
important to note that it is not a uniform phenomenon. More specifically, the magnitude
of the resultant enhancement, which is expressed as proportion of the best unisensory
response, is inversely proportional to the efficacies of the modality-specific component
stimuli — a relationship dubbed ‘inverse effectiveness’. When the enhanced
multisensory response of a neuron is instead referenced to the sum of its responses to the
modality-specific stimuli, inverse effectiveness typically manifests as a transition from
superadditivity (more than the sum of the unisensory responses) to subadditivity, as the
modality-specific stimuli themselves become more potent7,169. However, regardless of
Stein et al. Page 5
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

how it is quantified, inverse effectiveness, and particularly superadditivity, suggests that
multisensory integration is of most value for detecting stimuli that are weakly effective
on their own. Along with appealing to intuition, modelling studies suggest that this
particular feature of multisensory integration may be part of an optimal solution to the
problem of detecting stimuli in the face of both sensory and neural noise57,170, something
that is very much in line with the primary function of the superior colliculus. As such, it
is worth noting that inverse effectiveness may not generalize to, nor be particularly
advantageous for, multisensory computations that contribute to functions beyond
stimulus detection13,171.
Development of multisensory integration
The superior colliculus of a newborn cat, the principal source of what we know about the
development of multisensory integration, has no functional multisensory neurons and no
multisensory integration capabilities. In late fetal stages, and for a few days following birth,
the only active sensory-responsive neurons are those that respond to tactile cues (FIG. 1).
Their receptive fields are on or around the mouth, nose and whisker pads59, and help the
kitten to process cues obtained from sweeping its face across its mother’s fur in search of the
nipple. When the perioral region is anaesthetized with topical lidocaine, the kitten continues
to sweep its face through the mother’s fur but does not find the nipple; however, when the
animal’s mouth is placed on the nipple, it begins to suckle. Olfactory cues are of limited
value in this context: they help the kitten to find the mother but are not of primary use in
finding the nipple86.
The animal’s other sensory systems are still poorly developed at birth. Its eyes and ear
canals remain closed, so it is blind and deaf. The ear canals require several postnatal days to
open, and auditory-responsive superior colliculus neurons become evident soon thereafter.
This is followed by the appearance of the first multisensory neurons (somatosensory–
auditory neurons) at about 10 days of age. However, the eyelids do not open until postnatal
days 7–11, and visual multisensory neurons (the most common multisensory neuron in this
visually dominant structure) do not appear until 3 weeks after birth59,77. The response
latencies of immature superior colliculus neurons are exceedingly long59,60, and the delay in
visual responsiveness in the multisensory layers of the superior colliculus is in striking
contrast to the appearance of visually responsive neurons elsewhere in the nervous system
and even in the overlying superficial layers, in which visual responsiveness begins before
the end of the first postnatal week87,88. This distinction in the maturation of unisensory
versus multisensory responsiveness of superior colliculus neurons underscores the protracted
developmental time course of multisensory processes. Even after some multisensory neurons
first appear, their incidence increases towards the full complement only gradually over the
next 2–3 months (FIG. 2). Their information-processing capabilities require an even longer
developmental period.
The most relevant factor in this context is the initial inability of these multisensory neurons
to integrate their different sensory inputs. As noted earlier, they generate no more impulses
to the concordant combination of cross-modal cues than they do to the most effective of
Stein et al. Page 6
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

them alone. The capability of superior colliculus neurons to integrate cross-modal signals
does not appear until weeks after the first multisensory neurons have appeared. Even then,
the incidence of neurons with this capability is quite low and steadily increases as
development progresses, not reaching adult status until the animal is several months old77
(FIG. 2). This protracted time course parallels that of the development of inputs from
regions of the association cortex (see below).
The maturational period required for creating the adult-like complement of superior
colliculus neurons capable of multisensory integration seems to be surprisingly long. It is
not due to a paucity of effective sensory inputs, and it does not seem to relate to the vigour
of these inputs. The disconnection between the maturity of the unisensory inputs and that of
multisensory neurons and the immaturity of their multisensory integration capability is
particularly apparent in the newborn rhesus monkey, a precocial species that, like humans, is
born with its eyes and ears open. Because the rhesus monkey undergoes substantially more
prenatal maturation than the cat, its sensory capacities at birth are more advanced, a feature
that is particularly evident in the responses of its visual neurons in both the cortex and
superior colliculus89–92. It can see and hear very well, and it already has many multisensory
superior colliculus neurons. However, although these neurons vigorously respond to their
different sensory inputs and seem to be much more mature than their counterparts in the cat,
they too fail to integrate their cross-modal inputs. As in the cat superior colliculus, their
responses to coincident cross-modal sensory stimuli are no greater than responses to the
most effective of those stimuli individually92,93.
For a reasonable period of early postnatal development, many of the unisensory and
multisensory properties of cat superior colliculus neurons develop in parallel. Receptive
fields in each modality are initially very large and gradually contract during maturation,
progressively improving the spatial fidelity of their individual representations, the alignment
among the three sensory topographies and the register of the different receptive fields of
individual multisensory neurons. This increasing spatial registration is most obvious in
visual–auditory neurons because when the eyes are centred within the head, coordinates in
visual space (eye-centred) align with those in auditory space (head-centred) and thus the two
receptive fields can be mapped in the same exteroceptive reference frame.
At the point at which a neuron’s receptive fields have contracted in size to approximately
150% of the adult average, its probability of showing multisensory integration capabilities
becomes quite high77. This rule of thumb can be useful but is only correlative. It is not
causative, and one can have large receptive fields in neurons capable of multisensory
integration79. The shrinking of these neurons’ receptive fields seems to be most dependent
on exposure to discrete modality-specific stimuli, whereas the development of their
multisensory integration capabilities seems to require exposure to particular configurations
of cross-modal stimuli — both of which usually follow similar time courses. How cross-
modal experience is integrated by the underlying circuit to develop multisensory integration
capabilities in individual neurons is a critical dynamic that is discussed in detail below.
However, in order to describe this relationship properly, the nature of the circuit underlying
superior colliculus multisensory integration at maturity, especially the importance of
Stein et al. Page 7
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

influences descending to the superior colliculus from the association cortex, is discussed
first.
Developmental role of cortical inputs
The adult superior colliculus receives inputs from a host of subcortical and cortical sources
that represent different levels of processing within their unisensory hierarchies. These
include ‘lower-order’ subcortical areas such as the retina and pretectum (visual), the inferior
colliculus (auditory) and the trigeminal nucleus (somatosensory), as well as ‘higher-order’
association cortices62,94–98. These are only a select few examples of the rich set of
unisensory afferents that innervate the superior colliculus and from which it constructs its
three sensory representations. Each of these representations is formed from multiple
projection sources, giving the circuit the appearance of significant afferent redundancy.
Some of the cortical inputs in this circuit have been shown to develop gradually during
postnatal life and, as shown below, provide a key to the maturation of the integrative
capability of superior colliculus neurons. Of particular concern in this context are the inputs
derived from the association cortex (in cats, these include the anterior ectosylvian sulcus
(AES) and rostrolateral suprasylvian sulcus (rLS)). Selective deactivation of these inputs in
the adult animal (and neonate) (FIG. 2) has revealed that at least some afferents make
unique contributions to the information-processing capabilities of superior colliculus
neurons99–106.
The AES, the region that is most important in constructing the multisensory properties of
superior colliculus neurons, contains modality-specific subregions in which most neurons
are responsive to a given sensory input: namely, the anterior ectosylvian visual area (AEV),
the auditory field of the AES (FAES) and somatosensory area IV (SIV)107–109. Although
there are multisensory neurons scattered within these AES subregions, multisensory neurons
are concentrated at the margins, or transitional areas, between them. Nevertheless, neither
those multisensory neurons within the largely unisensory regions nor those concentrated at
their margins project to multisensory superior colliculus neurons52,110. It is the unisensory
AES neurons that project to them, and they do so in sensory combinations that match the
convergence patterns they derive from other sources99,110. A neuron that receives visual and
auditory inputs from sources other than the AES, for example, will receive inputs from the
AEV and the FAES but not from SIV.
In the mature adult, these descending inputs play a crucial part in superior colliculus
multisensory integration, and their deactivation eliminates this superior colliculus capability.
Although most superior colliculus neurons depend on inputs from the AES for this
capability, some require inputs from both the AES and rLS101. In the absence of this
essential input, the responses of a superior colliculus neuron are either equivalent to those
elicited by the most effective stimulus of the cross-modal pair or approximate a weighted
average111. The striking shift downward from response enhancement underscores the
powerful and selective effect of these cortico-collicular inputs on superior colliculus
multisensory integration (FIG. 3) and renders the multisensory responses of these neurons
equivalent to those exhibited during neonatal stages. In both cases, these neurons continue to
Stein et al. Page 8
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

provide common access to the motor machinery of the superior colliculus, but their activity
is not enhanced by the coincident action of multiple sensory inputs.
To better understand the impact of the converging inputs from AES subregions to superior
colliculus neurons in the mature circuit, and to determine whether their functional
maturation is relevant to the development of multisensory integration capabilities, the visual
(AEV) and auditory (FAES) subregions were deactivated individually as well as collectively
while recording from their target visual–auditory superior colliculus neurons99 (FIG. 3). It is
interesting to note that focal deactivation of either area alone had the same deleterious effect
of eliminating superior colliculus multisensory integration as did deactivating them together.
Apparently, synergy is a key feature of descending influences from these subregions. What
is not yet clear is where this functional synergy is exerted or how it is created. Nevertheless,
it seems likely that the key is in the synaptic configurations they form on their superior
colliculus target neurons either directly or via interneurons112.
Unfortunately, although the projection patterns have begun to be evaluated113,114, little is
known about the morphology of the cortico-collicular synaptic configurations on
multisensory neurons or how and when they develop. Cortico-collicular inputs from at least
some regions of the association cortex are already present within days of birth115 and
presumably during late embryonic stages, but their synaptic inputs are likely to be unformed
or non-functional. The gradual postnatal maturation of this input is one of the likely reasons
for the protracted developmental time course of superior colliculus multisensory
integration78. As shown below, it is the maturation of this pathway that is crucial for both
the expression of multisensory integration in adult superior colliculus neurons and for its
acquisition during development. Removal of the AES and rLS early in life precludes the
acquisition of multisensory integration capability76 and the performance benefits they
normally provide in superior colliculus-mediated orientation tasks116. Similar multisensory
deficits are also observed when these areas are deactivated in adult animals101. However,
there is a clear difference in the compensatory capacity of the neonate and adult in this
context. In adults, loss of the influences from only one of these regions of the association
cortex produces striking multisensory deficits, but this is not the case after removing only
one of these areas in early life. Neonates can compensate for the loss of either sub-area,
presumably by an enhancement of the influences (for example, the projections) from the
remaining area76. These two regions are unique in this early compensatory capability for
multisensory integration, as no other brain region is capable of substituting them in this
functional role117.
The necessity of cross-modal experience
The prolonged postnatal period of superior colliculus multisensory development not only
allows the crucial cortical inputs from the association cortex to develop but also allows the
multisensory circuit to obtain considerable experience with sensory cues. Presumably, this
period is sufficiently long to sculpt a circuit that distinguishes those cross-modal
relationships that signal the same biologically significant event from those that do not. This
is not a simple task, and extracting the statistical regularities that are required to reveal such
relationships in any environment is likely to be compromised by the profusion of unrelated
Stein et al. Page 9
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

stimuli. In order to examine this assumption, several experimental strategies have been used
to test the effect of restricting and manipulating an animal’s early experience with cross-
modal stimuli. These included rearing animals in various conditions in which some cross-
modal experiences are eliminated or in which these experiences are systematically altered.
The first strategy tested the importance of these experiences, and the second examined
whether their specifics were incorporated into the principles that would later govern
multisensory integration. There are several ways to eliminate specific cross-modal
experiences, the simplest of which is to rear animals in the dark, thereby precluding visually
contingent cross-modal experience.
Rearing in darkness
The high incidence of visually responsive multisensory superior colliculus neurons
(especially visual–auditory), the ease of restricting visual experiences by dark rearing and
the information available from the extensive use of visual deprivation in early studies of
visual system development118,119 made this an obvious first choice of manipulations. Litters
were placed in the dark with their mothers within several days of birth, and superior
colliculus neurons were studied when they reached adulthood (that is, when they reached >6
months of age). The normal categories of unisensory and multisensory neurons were found,
and visual–auditory neurons were very well represented. However, these multisensory
neurons were immature. Their receptive fields were extremely large, resembling those of
much younger animals, but, more importantly in the current context, they were unable to
engage in multisensory integration. Although superior colliculus neurons responded quite
well to visual and to auditory stimuli, their responses to spatiotemporally concordant visual–
auditory stimuli were no greater than those to the most effective of these component stimuli
alone120 (FIG. 4). Presumably, this failure to integrate resulted from the lack of visual–
auditory experience needed to form links between these senses via associative learning
principles121. Alternatively, however, this failure may have been due to the lack of visual
experience itself rather than the lack of cross-modal experience. Certainly, the absence of
the spatial references that visual inputs provide, and the general increase in activity that they
produce, has widespread consequences on the brain71,122–130. In a visually dominant
structure such as the superior colliculus, the loss of visual input might reduce afferent
activity to such an extent that the architecture needed to support multisensory integration
cannot be created.
Rearing with masking noise
If visual input has only a permissive role in this developmental process, restricting cross-
modal experience without limiting vision itself should not compromise multisensory
development. This was the thinking behind an experiment in which animals were reared in
an illuminated room with multiple speakers arranged around the home cage to provide omni-
directional broadband masking noise. Within this ‘noise room’, the blanket of sound
effectively masked all but the loudest transient auditory stimuli131,132. Some superior
colliculus auditory receptive fields did not contract normally during development, but many
others appeared to do so. Nevertheless, even these contracted receptive fields did not align
well with their visual counterparts (which had contracted to normal size), and the neurons
themselves lacked the capacity to engage in multisensory integration (FIG. 4). Instead, they
Stein et al. Page 10
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

responded to cross-modal cues in the same way as do neurons in dark-reared and neonatal
animals133. This result rules out the idea that a given sensory modality (for example, the
visual modality) is uniquely critical and suggests instead that there is something specific
about the nature of cross-modal exposure that determines whether multisensory integration
capabilities mature. Both dark-rearing and noise-rearing studies were based on sensory
exclusion, albeit very different forms of it. However, both rearing conditions precluded, or
seriously degraded, experience with patterned stimulation in one of the relevant senses. The
possibility that the development of superior colliculus multisensory integration might simply
require patterned experience in each modality, rather than explicit exposure to their cross-
modal combination, still cannot be eliminated.
Rearing with random sensory cues
The possibility that the multisensory deficit would not be obtained in animals with patterned
experience in both senses, even if they had no relevant cross-modal experience, was
examined directly. Animals reared in the dark were periodically exposed to visual and
auditory cues, the timing and location of which were randomized. In this way, each sensory
input activated multisensory superior colliculus neurons but did so independently of one
another. Once again, the receptive fields of multisensory superior colliculus neurons
remained larger than normal, although the neurons developed robust visual and auditory
responses. However, these neurons did not develop the capacity to engage in multisensory
integration79 (FIG. 4). A separate cohort of animals reared in the same room but periodically
exposed to the same stimuli in spatiotemporal concordance did develop this integrative
capability and, in this regard, their multisensory superior colliculus neurons functioned very
much like their counterparts in normally reared animals. However, the sizes of their
receptive fields were also very much like those of immature and dark-reared animals.
Together, the results underscore the idea that multisensory experience (that is, experience
with cross-modal stimuli) triggers changes in the underlying circuitry that lead to the ability
of superior colliculus neurons to integrate these different sensory inputs and that this process
is not disrupted by some immature unisensory properties (for example, incomplete
contraction of receptive fields).
The results also seem to suggest that this developmental requirement for multisensory
integration favours a system that adapts to those cross-modal relationships that are
experienced in the rearing environment. Presumably that is also the environment in which
the animals will later live, so animals will now be armed with a system best suited to those
particular events. The alternative is that the underlying circuit achieves the same
computational end point regardless of the specific features of the cross-modal stimuli
experienced. This would create a generalized system that is broadly useful across
environments. As space and time are broadly invariant across environments, such a
developmental plan could also be successful. Attempts to examine these alternatives
involved exposing animals to ‘anomalous’ cross-modal stimuli. If the specifics of the
stimulus experience were encoded, they should be reflected in the products of multisensory
integration.
Stein et al. Page 11
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

Rearing with anomalous cross-modal experience
Animals reared in the dark room were periodically exposed to synchronous visual–auditory
stimuli that were always spatially disparate. This type of cross-modal event was considered
‘anomalous’ only because it is unlikely that two such cues would co-occur with great
regularity in non-laboratory conditions and would be inconsistent with a single target for a
superior colliculus-mediated orientation response. When the animals had matured, it was
evident that although most of their superior colliculus neurons had not developed
multisensory integration capabilities, and responded no differently than did those in
neonatal, noise-reared or dark-reared animals, a significant minority of them exhibited an
‘abnormality’ that reflected their rearing condition. Their visual and auditory receptive fields
had contracted (albeit not necessarily to normal size) but were displaced laterally to one
another just like the visual and auditory stimuli in the rearing condition. Some neurons
showed no receptive field overlap, making it impossible to present a coincident visual–
auditory stimulus that would fall within both of their receptive fields. However, when the
visual stimulus was placed in the visual receptive field and the auditory stimulus in the
auditory receptive field, the neurons showed clear multisensory integration. The spatial
principle had been altered: enhanced responses were now obtained only with spatially
disparate stimuli134, a finding consistent with the idea that the system adapts to the specific
cross-modal relationships it experiences61,135–138.
However, the fact that only a minority of neurons (29%) developed this capability rather
than the majority (75–85%), as is the case in animals experiencing concordant visual–
auditory stimuli, suggests that the inherent flexibility of the system is limited. The
maturational outcome seems not to be solely determined by the cross-modal relationships
encountered in the environment but also by a native bias, or selective filter, that alters the
access of different cross-modal configurations to the learning process. This bias is probably
derived from the overlapping sensory topographies of the superior colliculus, which reflect
an unequal density of afferents tuned to spatiotemporally concordant cross-modal cues. As a
result of this preferential selectivity, the statistical relationships that become encoded in the
system are not veridical but biased towards concordance. The benefit of such a
predisposition would be to prioritize stimulus configurations that are likely to refer to
singular events, which can be targets for superior colliculus-mediated gaze shifts. Whether
this bias extends beyond the spatial and temporal relationships among cross-modal stimuli to
the particular features of the stimuli is not yet clear. However, studies in the cortices of
human subjects139–142 (see also REF. 143) and non-human primates34,144 suggest that the
semantic concordance of stimuli is a relevant factor that determines their integrative product.
Whether this derives from the brain’s early experiences with such concordant stimuli or
another inherent bias, and whether this is applicable to a structure such as the superior
colliculus, which is primarily concerned with detecting and locating events, is not yet
known.
Experience and cortical inputs
The two major factors needed for superior colliculus multisensory integration — cross-
modal experience and influences from the association cortex — are unlikely to be
Stein et al. Page 12
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

independent developmental events. If, as expected, the cross-modal experience alters
influences from the association cortex in ways that facilitate the development and
manifestation of this superior colliculus property, this would help to explain why association
cortex lesions preclude the maturation of superior colliculus multisensory integration, why
association cortex inputs to a superior colliculus neuron are functional at or before the time
it develops multisensory integration capability and why this capability is lost in neonatal
(FIG. 2) and adult animals when the association cortex is deactivated99,101,102,106.
That cross-modal experience is actually capable of altering the functional nature of cortico-
collicular inputs to facilitate the maturation of multisensory integration has been inferred
from experiments in which visual–auditory experience was precluded by dark rearing, as
discussed above. Instead of developing selective enhancement of multisensory superior
colliculus responses that are crucial for this capacity, the association cortex has been found
to exert a non-selective facilitation of responses to each modality-specific stimulus as well
as their cross-modal combination. Thus, in the absence of cross-modal experience, the
cortico-collicular inputs do develop but fail to provide the specific influences that are
necessary for the development of multisensory integration145.
Further supporting evidence for the interdependent effects of association cortex influences
on superior colliculus neurons and cross-modal experience comes from studies using
pharmacological deactivation of the cortex during early development. Implanting a polymer
infused with an inhibitory agent (that is, muscimol) over the association cortex silenced its
neurons during the period (postnatal weeks 4–12) in which cross-modal sensory experience
is being encoded and superior colliculus multisensory integration capability is first being
expressed146. Even a year or more after the cortical inputs had been reactivated, these
animals were unable to use visual and auditory cues synergistically to enhance their
performance in a standard superior colliculus-mediated detection and localization task (FIG.
5). Furthermore, their superior colliculus neurons failed to integrate these cross-modal
stimuli. Apparently, when these cortical neurons are not privy to cross-modal events,
superior colliculus multisensory integration capability does not develop even though the
superior colliculus itself is not deprived of access to this information.
All of the current observations point to the association cortex as the portal through which
cross-modal experience affects the circuit underlying superior colliculus multisensory
integration and as the site that is crucial for its expression throughout life. How experience
with cross-modal events produces the refinements in this projection that render it capable of
facilitating superior colliculus multisensory integration remains to be determined, and there
are multiple models that may prove to be helpful in this regard. For example, the process
that induces superior colliculus neurons to integrate cross-modal inputs seems to be highly
consistent with the operation of an associative learning rule such as spike-timing-dependent
plasticity (STDP)121,147,148. This algorithm provides a method for selectively potentiating
cortico-collicular connections that have impulse times that match the statistics of cross-
modal experience. The encoding of these statistics is believed to be the computational
foundation on which later multisensory integration is built, as hypothesized by several
neural network models149,150 (BOX 2).
Stein et al. Page 13
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

Ongoing plasticity
The functional plasticity of neonatal superior colliculus neurons is likely to be characteristic
of the midbrain multisensory circuits of many, if not all, species. Furthermore, it is unlikely
to be eliminated when the underlying circuit achieves its mature operational principles.
Indeed, the plasticity of this circuit seems to continue well into adulthood6,137, suggesting
that it may be appropriate to think of its development as an ongoing process.
That such plasticity exists in multisensory superior colliculus neurons in the adult cat
became evident when observing their reactions to a train of sequential stimuli. For example,
a visual–auditory neuron exposed to a sequence of visual and auditory stimuli with a
temporal offset just beyond its window for multisensory integration (that is, each stimulus
elicited a well-defined unisensory response) began to show multisensory interactions after
only a few trials151. The first ‘unisensory’ response in the pair increased in magnitude and
duration, and the latency of the second decreased, as if the two unisensory responses were
becoming fused into a single multisensory response despite the stimulus conditions
remaining unchanged (FIG. 6a). This sort of change is consistent with the same STDP
learning rules that are likely to have led to the acquisition of multisensory integration
capability in the first place152. Because presynaptic activity promotes potentiation in
synaptic weights when it precedes postsynaptic activity, the activity initiated by the first
response produced substantially more potentiation than would be expected if the second
stimulus were not presented.
A change in multisensory responses has also been documented when the same
spatiotemporally concordant cross-modal stimulus is repeatedly presented, as would likely
occur when an organism continues to interact with its source. This was particularly evident
in neurons that overtly responded to only one of their modality-specific inputs (for example,
auditory) but not to the other (for example, visual). The second input was ‘covert’, but its
influence was revealed through its multisensory interaction with the first. After only a few
repetitions of the cross-modal stimulus, not only were the multisensory responses more
robust but the covert unisensory input became overt (FIG. 6b). The ‘exposure’ of the
previously covert channel lasted for a considerable period, but without further exposure to
the cross-modal stimulus it gradually degraded to its previous state145. There are likely to be
many examples of such plasticity and, although few other examples have been examined in
detail thus far, one — the ability to use adult experience to develop what is normally
instantiated during early life — deserves special mention.
Box 2
Modelling the development of multisensory integration
The empirical results suggest that the development of superior colliculus multisensory
integration capabilities is synchronized to the maturation of cortico-collicular afferents
from unisensory regions of the association cortex. If these cortical inputs are removed
early in development or deactivated in the mature adult, superior colliculus neurons lose
the ability to integrate signals across the senses: they resemble the neonatal state. Cuppini
et al.
149 proposed a neural network model that accounts for these observations. In the
Stein et al. Page 14
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

model, naive neonatal superior colliculus neurons are primarily controlled by their
subcortical and primary cortical inputs. Excitatory and inhibitory influences produced by
these inputs implement a competitive dynamic, so that a neuron’s response to a
simultaneous pair of cross-modal stimuli reverts to the most robust response evoked by
an individual stimulus in the pair. According to the model, afferents from the association
cortex mature during the developmental period and establish their own excitatory and
inhibitory dynamics. Unlike the interactions between the non-association cortex inputs,
these dynamics facilitate synergistic interactions between concordant cross-modal stimuli
and restrict competitive interactions to discordant stimuli. The inputs from the association
cortex also suppress the input from subcortical and primary cortical sources. Thus,
although the original connections from subcortical and primary cortical sources are never
lost, their impact on the overt superior colliculus responses becomes minimized, because
the inputs from the association cortex have effectively subsumed their role in directing
superior colliculus responses. This results in a stable system that can appropriately
integrate signals in a manner that is consistent with the animal’s cross-modal experience
acquired in the postnatal period.
When early experience is not sufficient for the acquisition of multisensory integration, as is
the case when it has been restricted, this capability can still be acquired later in life (FIG.
6c). How much of this adult plasticity is due to an extension of the sensitive period because
of early sensory restriction and how much is due to the normal inherent plasticity of the
adult multisensory circuit is not yet known. However, the system is so sensitive to
experience that superior colliculus neurons in dark-reared animals were able to acquire this
capability after comparatively few sessions in which a single visual–auditory stimulus was
repeatedly presented. This occurred even when no overt responses to the stimulus were
required and in the absence of any of the reinforcement contingencies that are normally
associated with learning. It even occurred when cross-modal experience was provided only
when the animal was anaesthetized79,153,154. Furthermore, nearly the same proportion of
superior colliculus neurons acquired multisensory integration capability in these conditions,
and did so with nearly the same enhancement magnitudes, as in normal rearing conditions.
The capability was also retained in the absence of continued experience with the relevant
cues and generalized to other cross-modal stimulus combinations. It seems that what was
acquired in these conditions was the general principle that concordant visual and auditory
cues should be bound together. Presumably, greater specificity could have been learned if
some stimulus features were paired and others were not, as happens under natural
circumstances, but this remains to be explicitly demonstrated. In short, as long as the animal
experiences a consistent relationship between the cross-modal stimulus components, the
capability to integrate those cues develops rapidly even in adulthood.
Why then did this capability for multisensory integration not develop rapidly in animals
whose cortices were deactivated briefly during early development, as reported by Rowland
et al.
146? Superior colliculus neurons in these animals were unable to integrate cross-modal
cues even after more than a year of experience in a normal environment. Although this
capacity was ultimately acquired, it took an additional 1.5 to 4 years — far longer than
expected based on the results of cross-modal training experiments (FIG. 5). It is possible
Stein et al. Page 15
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

that the cortex was unable to respond properly to these stimuli until that time, but this seems
unlikely given the rapid functional return of cortical activity following the deactivation
process155. Although it seems counterintuitive that the richness of the normal environment
would be a poorer condition for acquiring multisensory integration capability than the
impoverished training environment, the ambiguity in the former circumstance may prove to
be the problem. In contrast to the training environment, in which there is only a single cross-
modal stimulus configuration and few competing stimuli, the normal environment contains a
great deal of complexity. There are not only a host of visual and auditory events present that
are unrelated to the events providing cross-modal stimuli, but there is extensive variability in
those that are. Cross-modal cues derived from the same events will vary in their relative
intensities, will be experienced in very different background conditions and can occur at
different distances or angles to the animal, thereby varying in their relative timing and
spatial alignment. Presumably such variability increases the time needed to learn cross-
modal associations but also increases tolerance for their variability. Deriving the cross-
modal relationships in these challenging circumstances does not seem to be a significant
problem for the neonate but may be more problematic for the adult, whose brain is no longer
equivalent to that of the neonate. Its functional sensory systems would have colonized areas
that are normally devoted to the missing or compromised sense, and all of its sensory areas
would have continued developing based on intrinsic factors and the sensory inputs to which
they were responsive. Perhaps this is why many human patients with early visual or auditory
deficits that are later corrected fail to fully recover their visual–auditory integration deficits
despite years of cross-modal experience in a normal environment156–158 (but see REF. 159).
That superior colliculus neurons in young animals master the specifics of the cross-modal
stimulus components in the training environment is evident from the observations that only
when a neuron has both of its receptive fields encroaching on the exposure site does it
develop multisensory integration capabilities153 (FIG. 7), and that these neurons develop
‘anomalies’ reflecting the relationship of the component stimuli79. Superior colliculus
neurons trained with spatially and temporally concordant visual–auditory stimuli exhibit
preferences for a cross-modal stimulus with that configuration. Their responses are
progressively degraded as the component stimuli are separated from one another in space
and/or time, a systematic bias that is not evident in normal animals81,82. Multisensory
response magnitude in normal animals has no systematic relationship with the location of
the visual and auditory stimuli within their overlapping receptive fields, and these neurons
prefer that the visual stimulus precede the auditory by 50–100 ms.
Summary and concluding remarks
It seems that the brain develops the capacity to integrate information from different senses
only after it obtains considerable experience with their cross-modal combinations. For cat
superior colliculus neurons, this acquisition period lasts for several postnatal months. It is
during this time that these neurons master the cross-modal statistics that typify common
detectable events and, through learning mechanisms, presumably craft the principles that
will determine how these cues are integrated. This ensures that the system adapts to the
environment in which it will be used, in most cases resulting in enhanced responses to cross-
Stein et al. Page 16
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

modal stimuli that occur in close spatiotemporal register (that is, those derived from the
same event) and a corresponding facilitation of superior colliculus-mediated orientation.
The association cortex seems to be critical for this process, not only because it functions as a
portal for experience to access the relevant neural circuit but because its descending
influences are required for superior colliculus multisensory integration to occur. Although
the appropriate circuit dynamics for this process are normally achieved over the first few
months of postnatal life when cross-modal experiences can effectively influence the system,
multisensory plasticity is retained into adulthood. Thus, appropriate experiences in
adulthood can compensate for their absence earlier in life, albeit with lowered efficiency.
Nevertheless, regardless of age, the likely substrates for these experience-based changes are
the synapses that convey these cortical influences through their direct and indirect
projections to the superior colliculus. Unfortunately, at present, little information is available
about the morphological development of these synapses or the microcircuits in which they
are embedded. Such information would be of considerable help in understanding how this
process is instantiated at any stage of life.
At present, the primary source of information about multisensory integration at the level of
the single neuron comes from the cat superior colliculus, and many of the lessons that we
have learned from this model are likely to apply to other species and even to other brain
circuits. Indeed, similar experiential requirements and developmental chronologies, albeit
even more delayed, have been noted in multisensory neurons in the cat association
cortex160. These observations are also consistent with the gradual maturation of many
higher-order multisensory perceptual capabilities in humans156,161–165, disruptions of which
may underlie the deficits in multisensory integration that have been noted in several human
developmental disorders166–168.
However, some caution must be exercised here in generalizing from the observations
discussed above. Multisensory integration takes place in many brain areas of many species
with different evolutionary histories, different sensory capabilities, different experiences and
facing different ecological challenges. It is unlikely that a single template is used in all such
circumstances or that all integrated multisensory responses are manifested as a simple
increase in the number of stimulus-evoked impulses. In addition, developing the capacity to
integrate cues for taste and smell or infrared and visual cues in species that depend most on
such processes may involve very different time courses than those for the visual, auditory
and somatosensory inputs to the cat superior colliculus. Whether, and how, species-specific
and region-specific adaptations of multisensory integration facilitate the functional role of
different neuronal populations represents one of the forefronts of this field, one that could
benefit from the combined expertise of neuroscientists, psychologists and ethologists.
Acknowledgments
Portions of the work described here have been supported by US National Institutes of Health grants EY016716 and
NS036916 and a grant from the Wallace Foundation.
Stein et al. Page 17
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

Glossary
Multisensory A process (behaviour) or entity (neuron or circuit) that
incorporates information derived from more than one sensory
modality.
Multisensory
enhancement
The response to a cross-modal stimulus is significantly greater
than its responses to either of the component stimuli.
Bayesian frameworks Statistical frameworks used to model perception in which a
feature of the world is inferred based on acquired sensory
evidence.
Receptive fields Regions of external space or location on the body in which
stimuli will reliably elicit responses from a given neuron.
Cross-modal stimulus A stimulus that activates two or more senses.
Spatiotemporal
concordance
Closely aligned in space and time.
Spike-timing-
dependent plasticity
(STDP)
A principle by which synaptic efficacy is strengthened when the
presynaptic neuron reliably generates an action potential before
the postsynaptic neuron generates an action potential, but is
weakened when the reverse relationship occurs or when the
activity patterns are decorrelated.
References
1. Stein, BE.; Meredith, MA. The Merging of the Senses. MIT Press; 1993.
2. Alais D, Burr D. The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol.
2004; 14:257–262. [PubMed: 14761661]
3. Ernst MO, Banks MS. Humans integrate visual and haptic information in a statistically optimal
fashion. Nature. 2002; 415:429–433. [PubMed: 11807554]
4. Shams L, Ma WJ, Beierholm U. Sound-induced flash illusion as an optimal percept. Neuroreport.
2005; 16:1923–1927. [PubMed: 16272880]
5. Lewkowicz, DJ.; Lickliter, R., editors. The Development of Intersensory Perception: Comparative
Perspectives. Lawrence Erlbaum Associates; 1994.
6. Stein, B., editor. The New Handbook of Multisensory Processing. MIT Press; 2012.
7. Stein BE, Stanford TR, Ramachandran R, Perrault TJ Jr, Rowland BA. Challenges in quantifying
multisensory integration: alternative criteria, models, and inverse effectiveness. Exp. Brain Res.
2009; 198:113–126. [PubMed: 19551377]
8. Binns KE, Salt TE. Importance of NMDA receptors for multimodal integration in the deep layers of
the cat superior colliculus. J. Neurophysiol. 1996; 75:920–930. [PubMed: 8714664]
9. Wallace MT, Meredith MA, Stein BE. Multisensory integration in the superior colliculus of the alert
cat. J. Neurophysiol. 1998; 80:1006–1010. [PubMed: 9705489]
10. Bolognini N, Leo F, Passamonti C, Stein BE, Làdavas E. Multisensory-mediated auditory
localization. Perception. 2007; 36:1477–1485. [PubMed: 18265830]
11. Corneil BD, Munoz DP. The influence of auditory and visual distractors on human orienting gaze
shifts. J. Neurosci. 1996; 16:8193–8207. [PubMed: 8987844]
12. Diederich A, Colonius H. Bimodal and trimodal multisensory enhancement: effects of stimulus
onset and intensity on reaction time. Percept. Psychophys. 2004; 66:1388–1404. [PubMed:
15813202]
Stein et al. Page 18
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

13. Fetsch CR, Pouget A, DeAngelis GC, Angelaki DE. Neural correlates of reliability-based cue
weighting during multisensory integration. Nature Neurosci. 2012; 15:146–154. [PubMed:
22101645]
14. This is a comprehensive review of the utility of Bayesian frameworks for studying multisensory
integration.
14. Foxe JJ, Schroeder CE. The case for feedforward multisensory convergence during early cortical
processing. Neuroreport. 2005; 16:419–423. [PubMed: 15770144]
15. Frens MA, Van Opstal AJ. Visual-auditory interactions modulate saccade-related activity in
monkey superior colliculus. Brain Res. Bull. 1998; 46:211–224. [PubMed: 9667814]
16. Hughes HC, Reuter-Lorenz PA, Nozawa G, Fendrich R. Visual-auditory interactions in
sensorimotor processing: saccades versus manual responses. J. Exp. Psychol. Hum. Percept.
Perform. 1994; 20:131–153. [PubMed: 8133219]
18. This early article helped to establish the relationship between multisensory integration and reaction
speed.
17. Naumer, MJ.; Kaiser, J., editors. Multisensory Object Perception in the Primate Brain. Springer;
2010.
18. Sánchez-García C, Alsius A, Enns JT, Soto-Faraco S. Cross-modal prediction in speech
perception. PLoS ONE. 2011; 6:e25198. [PubMed: 21998642]
19. Senkowski D, Talsma D, Grigutsch M, Herrmann CS, Woldorff MG. Good times for multisensory
integration: effects of the precision of temporal synchrony as revealed by gamma-band
oscillations. Neuropsychologia. 2007; 45:561–571. [PubMed: 16542688]
20. Stein BE, Meredith MA, Huneycutt WS, McDade L. Behavioral indices of multisensory
integration: orientation to visual cues is affected by auditory stimuli. J. Cogn. Neurosci. 1989;
1:12–24. [PubMed: 23968407]
23. This paper showed that the physiological principles of multisensory integration that are evident in
single superior colliculus neurons applies to overt orientation behaviour.
21. Talsma D, Doty TJ, Woldorff MG. Selective attention and audiovisual integration: is attending to
both modalities a prerequisite for early integration? Cereb. Cortex. 2007; 17:679–690. [PubMed:
16707740]
22. Woods TM, Recanzone GH. Visually induced plasticity of auditory spatial perception in
macaques. Curr. Biol. 2004; 14:1559–1564. [PubMed: 15341742]
23. Zhang M, Weisser VD, Stilla R, Prather SC, Sathian K. Multisensory cortical processing of object
shape and its relation to mental imagery. Cogn. Affect. Behav. Neurosci. 2004; 4:251–259.
[PubMed: 15460931]
24. Goldring JE, Dorris MC, Corneil BD, Ballantyne PA, Munoz DP. Combined eye-head gaze shifts
to visual and auditory targets in humans. Exp. Brain Res. 1996; 111:68–78. [PubMed: 8891638]
25. Sperdin HF, Cappe C, Foxe JJ, Murray MM. Early, low-level auditory-somatosensory
multisensory interactions impact reaction time speed. Front. Integr. Neurosci. 2009; 3:2. [PubMed:
19404410]
26. Von Saldern S, Noppeney U. Sensory and striatal areas integrate auditory and visual signals into
behavioral benefits during motion discrimination. J. Neurosci. 2013; 33:8841–8849. [PubMed:
23678126]
27. Bremner, AJ.; Lewkowicz, DJ.; Spence, C., editors. Multisensory Development. Oxford Univ.
Press; 2012.
28. Bizley JK, King AJ. Visual-auditory spatial processing in auditory cortical neurons. Brain Res.
2008; 1242:24–36. [PubMed: 18407249]
29. Brett-Green B, Fifková E, Larue DT, Winer JA, Barth DS. A multisensory zone in rat
parietotemporal cortex: intra- and extracellular physiology and thalamocortical connections. J.
Comp. Neurol. 2003; 460:223–237. [PubMed: 12687687]
30. Driver J, Noesselt T. Multisensory interplay reveals crossmodal influences on ‘sensory-specific’
brain regions, neural responses, and judgments. Neuron. 2008; 57:11–23. [PubMed: 18184561]
31. Ghazanfar AA, Schroeder CE. Is neocortex essentially multisensory? Trends Cogn. Sci. 2006;
10:278–285. [PubMed: 16713325]
Stein et al. Page 19
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

32. Graziano, M.; Gross, C.; Taylor, C.; Moore, T. Crossmodal Space and Crossmodal Attention.
Spence, C.; Driver, J., editors. Oxford Univ. Press; 2004. p. 51-67.
33. King AJ, Palmer AR. Integration of visual and auditory information in bimodal neurones in the
guinea-pig superior colliculus. Exp. Brain Res. 1985; 60:492–500. [PubMed: 4076371]
34. Romanski LM. Representation and integration of auditory and visual stimuli in the primate ventral
lateral prefrontal cortex. Cereb. Cortex. 2007; 17(Suppl. 1):i61–i69. [PubMed: 17634387]
35. Schroeder CE, Foxe J. Multisensory contributions to low-level, ‘unisensory’ processing. Curr.
Opin. Neurobiol. 2005; 15:454–458. [PubMed: 16019202]
39. An examination of the impact of multisensory integration in cortical regions believed to be
unisensory. The paper examines the evidence for multisensory information coding even within
classically defined, modality-specific areas of that cortex.
36. Wallace M, Ramachandran R, Stein B. A revised view of sensory cortical parcellation. Proc. Natl
Acad. Sci. USA. 2004; 101:2167–2172. [PubMed: 14766982]
37. Winkowski DE, Knudsen EI. Top-down control of multimodal sensitivity in the barn owl optic
tectum. J. Neurosci. 2007; 27:13279–13291. [PubMed: 18045922]
38. Macaluso, E.; Driver, J. The Handbook of Multisensory Processes. Calvert, GA.; Spence, C.; Stein,
BE., editors. MIT Press; 2004. p. 529-548.
39. Cohen, YE.; Andersen, RA. The Handbook of Multisensory Processes. Calvert, GA.; Spence, C.;
Stein, BE., editors. MIT Press; 2004. p. 463-479.
40. Burnett LR, Stein BE, Chaponis D, Wallace MT. Superior colliculus lesions preferentially disrupt
multisensory orientation. Neuroscience. 2004; 124:535–547. [PubMed: 14980725]
41. Burnett LR, Stein BE, Perrault TJ Jr, Wallace MT. Excitotoxic lesions of the superior colliculus
preferentially impact multisensory neurons and multisensory integration. Exp. Brain Res. 2007;
179:325–338. [PubMed: 17146648]
42. Groh JM, Sparks DL. Saccades to somatosensory targets. II. Motor convergence in primate
superior colliculus. J. Neurophysiol. 1996; 75:428–438. [PubMed: 8822568]
43. Groh JM, Sparks DL. Saccades to somatosensory targets. III. Eye-position-dependent
somatosensory activity in primate superior colliculus. J. Neurophysiol. 1996; 75:439–453.
[PubMed: 8822569]
44. Harrington LK, Peck CK. Spatial disparity affects visual-auditory interactions in human
sensorimotor processing. Exp. Brain Res. 1998; 122:247–252. [PubMed: 9776523]
45. Jay MF, Sparks DL. Sensorimotor integration in the primate superior colliculus. I. Motor
convergence. J. Neurophysiol. 1987; 57:22–34. [PubMed: 3559673]
46. Jay MF, Sparks DL. Sensorimotor integration in the primate superior colliculus. II. Coordinates of
auditory signals. J. Neurophysiol. 1987; 57:35–55. [PubMed: 3559680]
47. Munoz DP, Wurtz RH. Saccade-related activity in monkey superior colliculus. I. Characteristics of
burst and buildup cells. J. Neurophysiol. 1995; 73:2313–2333. [PubMed: 7666141]
48. Munoz DP, Wurtz RH. Saccade-related activity in monkey superior colliculus. II. Spread of
activity during saccades. J. Neurophysiol. 1995; 73:2334–2348. [PubMed: 7666142]
49. Lomber SG, Payne BR, Cornwell P. Role of the superior colliculus in analyses of space: superficial
and intermediate layer contributions to visual orienting, auditory orienting, and visuospatial
discriminations during unilateral and bilateral deactivations. J. Comp. Neurol. 2001; 441:44–57.
[PubMed: 11745634]
50. Levine MS, Hull CD, Buchwald NA. Development of motor activity in kittens. Dev. Psychobiol.
1980; 13:357–371. [PubMed: 7390018]
51. Norton TT. Receptive-field properties of superior colliculus cells and development of visual
behavior in kittens. J. Neurophysiol. 1974; 37:674–690. [PubMed: 4837772]
52. Meredith MA, Wallace MT, Stein BE. Visual, auditory and somatosensory convergence in output
neurons of the cat superior colliculus: multisensory properties of the tecto-reticulo-spinal
projection. Exp. Brain Res. 1992; 88:181–186. [PubMed: 1541354]
53. Sparks DL. Translation of sensory signals into commands for control of saccadic eye movements:
role of primate superior colliculus. Physiol. Rev. 1986; 66:118–171. [PubMed: 3511480]
Stein et al. Page 20
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

54. Wurtz RH, Goldberg ME. The role of the superior colliculus in visually-evoked eye movements.
Bibl. Ophthalmol. 1972; 82:149–158.
55. Moschovakis AK, et al. An anatomical substrate for the spatiotemporal transformation. J.
Neurosci. 1998; 18:10219–10229. [PubMed: 9822775]
56. Knill DC, Pouget A. The Bayesian brain: the role of uncertainty in neural coding and computation.
Trends Neurosci. 2004; 27:712–719. [PubMed: 15541511]
57. Patton PE, Anastasio TJ. Modeling cross-modal enhancement and modality-specific suppression in
multisensory neurons. Neural Comput. 2003; 15:783–810. [PubMed: 12689387]
58. Rowland B, Stanford T, Stein B. A Bayesian model unifies multisensory spatial localization with
the physiological properties of the superior colliculus. Exp. Brain Res. 2007; 180:153–161.
[PubMed: 17546470]
59. Stein BE, Labos E, Kruger L. Sequence of changes in properties of neurons of superior colliculus
of the kitten during maturation. J. Neurophysiol. 1973; 36:667–679. [PubMed: 4713313]
64. This was the first demonstration of the developmental chronology of sensory representations in the
multisensory superior colliculus.
60. Stein BE, Labos E, Kruger L. Determinants of response latency in neurons of superior colliculus in
kittens. J. Neurophysiol. 1973; 36:680–689. [PubMed: 4713314]
61. Benedetti F. Differential formation of topographic maps on the cerebral cortex and superior
colliculus of the mouse by temporally correlated tactile-tactile and tactile-visual inputs. Eur. J.
Neurosci. 1995; 7:1942–1951. [PubMed: 8528470]
62. Meredith MA, Clemo HR, Stein BE. Somatotopic component of the multisensory map in the deep
laminae of the cat superior colliculus. J. Comp. Neurol. 1991; 312:353–370. [PubMed: 1748738]
63. Meredith MA, Stein BE. The visuotopic component of the multisensory map in the deep laminae
of the cat superior colliculus. J. Neurosci. 1990; 10:3727–3742. [PubMed: 2230957]
64. Middlebrooks JC, Knudsen EI. A neural code for auditory space in the cat’s superior colliculus. J.
Neurosci. 1984; 4:2621–2634. [PubMed: 6491727]
65. Stein BE, Magalhaes-Castro B, Kruger L. Superior colliculus: visuotopic-somatotopic overlap.
Science. 1975; 189:224–226. [PubMed: 1094540]
66. Chalupa LM, Rhoades RW. Responses of visual, somatosensory, and auditory neurones in the
golden hamster’s superior colliculus. J. Physiol. 1977; 270:595–626. [PubMed: 903907]
67. Dräger UC, Hubel DH. Physiology of visual cells in mouse superior colliculus and correlation with
somatosensory and auditory input. Nature. 1975; 253:203–204. [PubMed: 1110771]
68. Dräger UC, Hubel DH. Topography of visual and somatosensory projections to mouse superior
colliculus. J. Neurophysiol. 1976; 39:91–101. [PubMed: 1249606]
69. Gordon B. Receptive fields in deep layers of cat superior colliculus. J. Neurophysiol. 1973;
36:157–178. [PubMed: 4574712]
70. Graham J, Pearson HE, Berman N, Murphy EH. Laminar organization of superior colliculus in the
rabbit: a study of receptive-field properties of single units. J. Neurophysiol. 1981; 45:915–932.
[PubMed: 7241177]
71. King AJ, Schnupp JW, Thompson ID. Signals from the superficial layers of the superior colliculus
enable the development of the auditory space map in the deeper layers. J. Neurosci. 1998;
18:9394–9408. [PubMed: 9801378]
72. Knudsen EI. Auditory and visual maps of space in the optic tectum of the owl. J. Neurosci. 1982;
2:1177–1194. [PubMed: 7119872]
73. Stein BE, Dixon JP. Superior colliculus cells respond to noxious stimuli. Brain Res. 1978; 158:65–
73. [PubMed: 21348352]
74. Stein BE, Magalhães-Castro B, Kruger L. Relationship between visual and tactile representations
in cat superior colliculus. J. Neurophysiol. 1976; 39:401–419. [PubMed: 1255230]
75. Alvarado JC, Vaughan JW, Stanford TR, Stein BE. Multisensory versus unisensory integration:
contrasting modes in the superior colliculus. J. Neurophysiol. 2007; 97:3193–3205. [PubMed:
17329632]
76. Jiang W, Jiang H, Stein BE. Neonatal cortical ablation disrupts multisensory development in
superior colliculus. J. Neurophysiol. 2006; 95:1380–1396. [PubMed: 16267111]
Stein et al. Page 21
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

77. Wallace MT, Stein BE. Development of multisensory neurons and multisensory integration in cat
superior colliculus. J. Neurosci. 1997; 17:2429–2444. [PubMed: 9065504]
83. This was the first study of the development of multisensory integration in superior colliculus
neurons.
78. Wallace MT, Stein BE. Onset of cross-modal synthesis in the neonatal superior colliculus is gated
by the development of cortical influences. J. Neurophysiol. 2000; 83:3578–3582. [PubMed:
10848574]
79. Xu J, Yu L, Rowland BA, Stanford TR, Stein BE. Incorporating cross-modal statistics in the
development and maintenance of multisensory integration. J. Neurosci. 2012; 32:2287–2298.
[PubMed: 22396404]
86. This study demonstrates some of the experiential prerequisites for instantiating multisensory
integration capabilities in superior colliculus neurons.
80. Meredith MA, Stein BE. Interactions among converging sensory inputs in the superior colliculus.
Science. 1983; 221:389–391. [PubMed: 6867718]
88. This was the first paper to demonstrate the multisensory integration capability of single superior
colliculus neurons.
81. Kadunce DC, Vaughan JW, Wallace MT, Stein BE. The influence of visual and auditory receptive
field organization on multisensory integration in the superior colliculus. Exp. Brain Res. 2001;
139:303–310. [PubMed: 11545469]
82. Meredith MA, Nemitz JW, Stein BE. Determinants of multisensory integration in superior
colliculus neurons. I. Temporal factors. J. Neurosci. 1987; 7:3215–3229. [PubMed: 3668625]
83. Meredith MA, Stein BE. Spatial factors determine the activity of multisensory neurons in cat
superior colliculus. Brain Res. 1986; 365:350–354. [PubMed: 3947999]
84. Kadunce DC, Vaughan JW, Wallace MT, Benedek G, Stein BE. Mechanisms of within- and cross-
modality suppression in the superior colliculus. J. Neurophysiol. 1997; 78:2834–2847. [PubMed:
9405504]
85. Pluta SR, Rowland BA, Stanford TR, Stein BE. Alterations to multisensory and unisensory
integration by stimulus competition. J. Neurophysiol. 2011; 106:3091–3101. [PubMed: 21957224]
86. Larson MA, Stein BE. The use of tactile and olfactory cues in neonatal orientation and localization
of the nipple. Dev. Psychobiol. 1984; 17:423–436. [PubMed: 6745502]
87. Kao CQ, McHaffie JG, Meredith MA, Stein BE. Functional development of a central visual map in
cat. J. Neurophysiol. 1994; 72:266–272. [PubMed: 7965010]
88. Hubel DH, Wiesel TN. Receptive fields of cells in striate cortex of very young, visually
inexperienced kittens. J. Neurophysiol. 1963; 26:994–1002. [PubMed: 14084171]
89. Horton JC, Hocking DR. An adult-like pattern of ocular dominance columns in striate cortex of
newborn monkeys prior to visual experience. J. Neurosci. 1996; 16:1791–1807. [PubMed:
8774447]
90. Hubel DH, Wiesel TN. Receptive fields and functional architecture of monkey striate cortex. J.
Physiol. 1968; 195:215–243. [PubMed: 4966457]
91. Rakic P. Prenatal development of the visual system in rhesus monkey. Philos. Trans. R. Soc. Lond.
B. 1977; 278:245–260. [PubMed: 19781]
92. Wallace MT, McHaffie JG, Stein BE. Visual response properties and visuotopic representation in
the newborn monkey superior colliculus. J. Neurophysiol. 1997; 78:2732–2741. [PubMed:
9356422]
93. Wallace MT, Stein BE. Sensory organization of the superior colliculus in cat and monkey. Prog.
Brain Res. 1996; 112:301–311. [PubMed: 8979837]
94. Edwards SB, Ginsburgh CL, Henkel CK, Stein BE. Sources of subcortical projections to the
superior colliculus in the cat. J. Comp. Neurol. 1979; 184:309–329. [PubMed: 762286]
95. Huerta MF, Harting JK. The projection from the nucleus of the posterior commissure to the
superior colliculus of the cat: patch-like endings within the intermediate and deep grey layers.
Brain Res. 1982; 238:426–432. [PubMed: 7093663]
Stein et al. Page 22
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

96. Meredith MA, Clemo HR. Auditory cortical projection from the anterior ectosylvian sulcus (Field
AES) to the superior colliculus in the cat: an anatomical and electrophysiological study. J. Comp.
Neurol. 1989; 289:687–707. [PubMed: 2592605]
97. Stein BE, Spencer RF, Edwards SB. Corticotectal and corticothalamic efferent projections of SIV
somatosensory cortex in cat. J. Neurophysiol. 1983; 50:896–909. [PubMed: 6313872]
98. Scannell JW, et al. Visual motion processing in the anterior ectosylvian sulcus of the cat. J.
Neurophysiol. 1996; 76:895–907. [PubMed: 8871207]
99. Alvarado JC, Stanford TR, Rowland BA, Vaughan JW, Stein BE. Multisensory integration in the
superior colliculus requires synergy among corticocollicular inputs. J. Neurosci. 2009; 29:6580–
6592. [PubMed: 19458228]
100. Alvarado JC, Stanford TR, Vaughan JW, Stein BE. Cortex mediates multisensory but not
unisensory integration in superior colliculus. J. Neurosci. 2007; 27:12775–12786. [PubMed:
18032649]
101. Jiang W, Wallace MT, Jiang H, Vaughan JW, Stein BE. Two cortical areas mediate multisensory
integration in superior colliculus neurons. J. Neurophysiol. 2001; 85:506–522. [PubMed:
11160489]
110. Two areas of association cortical areas were shown to have crucial roles in superior colliculus
multisensory integration.
102. Jiang W, Jiang H, Stein BE. Two corticotectal areas facilitate multisensory orientation behavior.
J. Cogn. Neurosci. 2002; 14:1240–1255. [PubMed: 12495529]
103. Jiang W, Stein BE. Cortex controls multisensory depression in superior colliculus. J.
Neurophysiol. 2003; 90:2123–2135. [PubMed: 14534263]
104. Stein BE, Wallace MW, Stanford TR, Jiang W. Cortex governs multisensory integration in the
midbrain. Neuroscientist. 2002; 8:306–314. [PubMed: 12194499]
105. Stein BE. The development of a dialogue between cortex and midbrain to integrate multisensory
information. Exp. Brain Res. 2005; 166:305–315. [PubMed: 15988596]
106. Wallace MT, Stein BE. Cross-modal synthesis in the midbrain depends on input from cortex. J.
Neurophysiol. 1994; 71:429–432. [PubMed: 8158240]
107. Clemo HR, Stein BE. Topographic organization of somatosensory corticotectal influences in cat.
J. Neurophysiol. 1984; 51:843–858. [PubMed: 6726314]
108. Mucke L, Norita M, Benedek G, Creutzfeldt O. Physiologic and anatomic investigation of a
visual cortical area situated in the ventral bank of the anterior ectosylvian sulcus of the cat. Exp.
Brain Res. 1982; 46:1–11. [PubMed: 7067781]
109. Olson CR, Graybiel AM. Ectosylvian visual area of the cat: location, retinotopic organization,
and connections. J. Comp. Neurol. 1987; 261:277–294. [PubMed: 3624544]
110. Wallace MT, Meredith MA, Stein BE. Converging influences from visual, auditory, and
somatosensory cortices onto output neurons of the superior colliculus. J. Neurophysiol. 1993;
69:1797–1809. [PubMed: 8350124]
111. Alvarado JC, Rowland BA, Stanford TR, Stein BE. A neural network model of multisensory
integration also accounts for unisensory integration in superior colliculus. Brain Res. 2008;
1242:13–23. [PubMed: 18486113]
112. McHaffie, JG., et al. The New Handbook of Multisensory Processing. Stein, B., editor. MIT
Press; 2012. p. 31-47.
113. Fuentes-Santamaria V, Alvarado JC, Stein BE, McHaffie JG. Cortex contacts both output neurons
and nitrergic interneurons in the superior colliculus: direct and indirect routes for multisensory
integration. Cereb. Cortex. 2008; 18:1640–1652. [PubMed: 18003596]
114. Fuentes-Santamaria V, McHaffie JG, Stein BE. Maturation of multisensory integration in the
superior colliculus: expression of nitric oxide synthase and neurofilament SMI-32. Brain Res.
2008; 1242:45–53. [PubMed: 18486108]
115. McHaffie JG, Kruger L, Clemo HR, Stein BE. Corticothalamic and corticotectal somatosensory
projections from the anterior ectosylvian sulcus (SIV cortex) in neonatal cats: an anatomical
demonstration with HRP and 3H-leucine. J. Comp. Neurol. 1988; 274:115–126. [PubMed:
2458394]
Stein et al. Page 23
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

116. Jiang W, Jiang H, Rowland BA, Stein BE. Multisensory orientation behavior is disrupted by
neonatal cortical ablation. J. Neurophysiol. 2007; 97:557–562. [PubMed: 16971678]
117. Wilkinson LK, Meredith MA, Stein BE. The role of anterior ectosylvian cortex in cross-modality
orientation and approach behavior. Exp. Brain Res. 1996; 112:1–10. [PubMed: 8951401]
118. Rauschecker JP. Developmental plasticity and memory. Behav. Brain Res. 1995; 66:7–12.
[PubMed: 7755902]
119. Wiesel TN, Hubel DH. Single-cell responses in striate cortex of kittens deprived of vision in one
eye. J. Neurophysiol. 1963; 26:1003–1017. [PubMed: 14084161]
120. Wallace MT, Perrault TJ Jr, Hairston WD, Stein BE. Visual experience is necessary for the
development of multisensory integration. J. Neurosci. 2004; 24:9580–9584. [PubMed:
15509745]
121. Feldman DE. The spike-timing dependence of plasticity. Neuron. 2012; 75:556–571. [PubMed:
22920249]
122. Crair MC. Neuronal activity during development: permissive or instructive? Curr. Opin.
Neurobiol. 1999; 9:88–93. [PubMed: 10072369]
123. Hua JY, Smith SJ. Neural activity and the dynamics of central nervous system development.
Nature Neurosci. 2004; 7:327–332. [PubMed: 15048120]
124. Kirkwood A, Lee HK, Bear MF. Co-regulation of long-term potentiation and experience-
dependent synaptic plasticity in visual cortex by age and experience. Nature. 1995; 375:328–331.
[PubMed: 7753198]
125. Knudsen EI, Brainard MS. Visual instruction of the neural map of auditory space in the
developing optic tectum. Science. 1991; 253:85–87. [PubMed: 2063209]
126. Withington DJ. The effect of binocular lid suture on auditory responses in the guinea-pig superior
colliculus. Neurosci. Lett. 1992; 136:153–156. [PubMed: 1641183]
127. Rauschecker JP, Harris LR. Auditory compensation of the effects of visual deprivation in the
cat’s superior colliculus. Exp. Brain Res. 1983; 50:69–83. [PubMed: 6641852]
137. This article provides an early demonstration of the ability of superior colliculus neurons to engage
in cross-modal compensation by expanding one sensory representation as a result of restricted
experience in another.
128. Champoux F, Bacon BA, Lepore F, Guillemot J-P. Effects of early binocular enucleation on
auditory and somatosensory coding in the superior colliculus of the rat. Brain Res. 2008;
1191:84–95. [PubMed: 18155185]
129. Mundiñano IC, Martínez-Millán L. Somatosensory cross-modal plasticity in the superior
colliculus of visually deafferented rats. Neuroscience. 2010; 165:1457–1470. [PubMed:
19932888]
130. Merabet LB, Pascual-Leone A. Neural reorganization following sensory loss: the opportunity of
change. Nature Rev. Neurosci. 2010; 11:44–52. [PubMed: 19935836]
131. Chang EF, Merzenich MM. Environmental noise retards auditory cortical development. Science.
2003; 300:498–502. [PubMed: 12702879]
132. Efrati A, Gutfreund Y. Early life exposure to noise alters the representation of auditory
localization cues in the auditory space map of the barn owl. J. Neurophysiol. 2011; 105:2522–
2535. [PubMed: 21368005]
133. Xu J, Yu L, Rowland BA, Stanford TR, Stein BE. Noise-rearing disrupts the maturation of
multisensory integration. Eur. J. Neurosci. 2014; 39:602–613. [PubMed: 24251451]
144. This was the first study to demonstrate that superior colliculus neurons require experience with
co-varying cross-modal signals to develop the circuitry needed to integrate the information they
provide.
134. Wallace MT, Stein BE. Early experience determines how the senses will interact. J.
Neurophysiol. 2007; 97:921–926. [PubMed: 16914616]
135. Benedetti F, Ferro I. The effects of early postnatal modification of body shape on the
somatosensory-visual organization in mouse superior colliculus. Eur. J. Neurosci. 1995; 7:412–
418. [PubMed: 7773438]
Stein et al. Page 24
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

136. Benedetti F. Orienting behaviour and superior colliculus sensory representations in mice with the
vibrissae bent into the contralateral hemispace. Eur. J. Neurosci. 1995; 7:1512–1519. [PubMed:
7551177]
137. Brainard MS, Knudsen EI. Sensitive periods for visual calibration of the auditory space map in
the barn owl optic tectum. J. Neurosci. 1998; 18:3929–3942. [PubMed: 9570820]
138. King AJ. Neural plasticity: how the eye tells the brain about sound location. Curr. Biol. 2002;
12:R393–R395. [PubMed: 12062073]
139. Beauchamp MS, Lee KE, Argall BD, Martin A. Integration of auditory and visual information
about objects in superior temporal sulcus. Neuron. 2004; 41:809–823. [PubMed: 15003179]
140. De Gelder B, Bertelson P. Multisensory integration, perception and ecological validity. Trends
Cogn. Sci. 2003; 7:460–467. [PubMed: 14550494]
141. King AJ, Calvert GA. Multisensory integration: perceptual grouping by eye and ear. Curr. Biol.
2001; 11:R322–R325. [PubMed: 11369224]
142. Baier B, Kleinschmidt A, Müller NG. Cross-modal processing in early visual and auditory
cortices depends on expected statistical relationship of multisensory information. J. Neurosci.
2006; 26:12260–12265. [PubMed: 17122051]
143. Fiebelkorn IC, Foxe JJ, Molholm S. Dual mechanisms for the cross-sensory spread of attention:
how much do learned associations matter? Cereb. Cortex. 2010; 20:109–120. [PubMed:
19395527]
144. Barraclough NE, Xiao D, Baker CI, Oram MW, Perrett DI. Integration of visual and auditory
information by superior temporal sulcus neurons responsive to the sight of actions. J. Cogn.
Neurosci. 2005; 17:377–391. [PubMed: 15813999]
145. Yu L, Rowland BA, Xu J, Stein BE. Multisensory plasticity in adulthood: cross-modal experience
enhances neuronal excitability and exposes silent inputs. J. Neurophysiol. 2013; 109:464–474.
[PubMed: 23114212]
146. Rowland B, Jiang W, Stein B. Brief cortical deactivation early in life has long-lasting effects on
multisensory behavior. J. Neurosci. 2014; 34:7198–7202. [PubMed: 24849354]
158. There is a sensitive period during early life when cross-modal experience, through the association
cortex, normally alters the multisensory circuit of the superior colliculus so that it can engage in
multisensory integration.
147. Bi G, Poo M. Synaptic modification by correlated activity: Hebb’s postulate revisited. Annu. Rev.
Neurosci. 2001; 24:139–166. [PubMed: 11283308]
148. Gerstner W, Ritz R, van Hemmen JL. Why spikes? Hebbian learning and retrieval of time-
resolved excitation patterns. Biol. Cybern. 1993; 69:503–515. [PubMed: 7903867]
149. Cuppini C, Magosso E, Rowland B, Stein B, Ursino M. Hebbian mechanisms help explain
development of multisensory integration in the superior colliculus: a neural network model. Biol.
Cybern. 2012; 106:691–713. [PubMed: 23011260]
150. Grossberg S, Roberts K, Aguilar M, Bullock D. A neural model of multimodal adaptive saccadic
eye movement control by superior colliculus. J. Neurosci. 1997; 17:9706–9725. [PubMed:
9391024]
151. Yu L, Stein BE, Rowland BA. Adult plasticity in multisensory neurons: short-term experience-
dependent changes in the superior colliculus. J. Neurosci. 2009; 29:15910–15922. [PubMed:
20016107]
152. Mégevand P, Molholm S, Nayak A, Foxe JJ. Recalibration of the multisensory temporal window
of integration results from changing task demands. PLoS ONE. 2013; 8:e71608. [PubMed:
23951203]
153. Yu L, Rowland BA, Stein BE. Initiating the development of multisensory integration by
manipulating sensory experience. J. Neurosci. 2010; 30:4904–4913. [PubMed: 20371810]
154. Royal DW, Krueger J, Fister MC, Wallace MT. Adult plasticity of spatiotemporal receptive fields
of multisensory superior colliculus neurons following early visual deprivation. Restor. Neurol.
Neurosci. 2010; 28:259–270. [PubMed: 20404413]
155. Smith AL, et al. An investigation of the role of auditory cortex in sound localization using
muscimol-releasing Elvax. Eur. J. Neurosci. 2004; 19:3059–3072. [PubMed: 15182314]
Stein et al. Page 25
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

156. Putzar L, Gondan M, Röder B. Basic multisensory functions can be acquired after congenital
visual pattern deprivation in humans. Dev. Neuropsychol. 2012; 37:697–711. [PubMed:
23145567]
169. A demonstration of the inherent plasticity of multisensory integration in the human brain.
157. Schorr EA, Fox NA, van Wassenhove V, Knudsen EI. Auditory-visual fusion in speech
perception in children with cochlear implants. Proc. Natl Acad. Sci. USA. 2005; 102:18748–
18750. [PubMed: 16339316]
158. Putzar L, Hötting K, Röder B. Early visual deprivation affects the development of face
recognition and of audio-visual speech perception. Restor. Neurol. Neurosci. 2010; 28:251–257.
[PubMed: 20404412]
159. Rouger J, et al. Evidence that cochlear-implanted deaf patients are better multisensory integrators.
Proc. Natl Acad. Sci. USA. 2007; 104:7295–7300. [PubMed: 17404220]
160. Carriere BN, et al. Visual deprivation alters the development of cortical multisensory integration.
J. Neurophysiol. 2007; 98:2858–2867. [PubMed: 17728386]
161. Gori M, Giuliana L, Sandini G, Burr D. Visual size perception and haptic calibration during
development. Dev. Sci. 2012; 15:854–862. [PubMed: 23106739]
162. Gori M, Sandini G, Burr D. Development of visuo-auditory integration in space and time. Front.
Integr. Neurosci. 2012; 6:77. [PubMed: 23060759]
176. The protracted postnatal development of aspects of visual–auditory integration in humans is
examined in this study.
163. Lewkowicz DJ. Perception of dynamic and static audiovisual sequences in 3- and 4-month-old
infants. Child Dev. 2008; 79:1538–1554. [PubMed: 18826541]
164. Neil PA, Chee-Ruiter C, Scheier C, Lewkowicz DJ, Shimojo S. Development of multisensory
spatial integration and perception in humans. Dev. Sci. 2006; 9:454–464. [PubMed: 16911447]
179. A description of the postnatal development of multisensory-processing capabilities in human
subjects.
165. Barutchu A, et al. Audiovisual integration in noise by children and adults. J. Exp. Child Psychol.
2010; 105:38–50. [PubMed: 19822327]
166. Foxe, JJ., et al. Severe multisensory speech integration deficits in high-functioning school-aged
children with autism spectrum disorder (ASD) and their resolution during early adolescence.
Cereb. Cortex. 2013. http://dx.doi.org/10.1093/cercor/bht213
182. This study identifies the early multisensory-processing disorder associated with autism spectrum
disorder and its resolution.
167. Hairston WD, Burdette JH, Flowers DL, Wood FB, Wallace MT. Altered temporal profile of
visual-auditory multisensory interactions in dyslexia. Exp. Brain Res. 2005; 166:474–480.
[PubMed: 16028030]
168. Stevenson RA, et al. Multisensory temporal integration in autism spectrum disorders. J. Neurosci.
2014; 34:691–697. [PubMed: 24431427]
169. Stanford TR, Quessy S, Stein BE. Evaluating the operations underlying multisensory integration
in the cat superior colliculus. J. Neurosci. 2005; 25:6499–6508. [PubMed: 16014711]
170. Cuppini C, Ursino M, Magosso E, Rowland BA, Stein BE. An emergent model of multisensory
integration in superior colliculus neurons. Front. Integr. Neurosci. 2010; 4:6. [PubMed:
20431725]
171. Fetsch CR, Deangelis GC, Angelaki DE. Bridging the gap between theories of sensory cue
integration and the physiology of multisensory neurons. Nature Rev. Neurosci. 2013; 14:429–
442. [PubMed: 23686172]
Stein et al. Page 26
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

Figure 1. The organization and development of the multisensory superior colliculus
a | The cut-away diagram shows the location of the superior colliculus (SC) in the midbrain
of the cat and the association cortex (anterior ectosylvian sulcus (AES) and rostrolateral
suprasylvian sulcus (rLS)), from which the SC receives crucial cortico-collicular inputs. b |
The three sensory representations (visual, auditory and somatosensory; shown at the top) in
the SC are organized into an overlapping multisensory topographic map, as shown below
(grey map). In each individual map, the horizontal meridian runs roughly rostral–caudal and
the vertical meridian runs medial– lateral. Thus, forward or central space is represented
rostrally, rearward or peripheral space is represented caudally, superior space is represented
medially and inferior space is represented laterally. The multisensory map shows the
topographic correspondence among the three maps, with the purple regions encompassing
the variations in the two meridians that exist among the three maps. External events, such as
the presence of the rodent, are often registered by multiple senses (in this case, vision and
audition) and relayed via converging cross-modal afferents onto common multisensory
target neurons in the map, which are exemplified by crosses in the maps. In adult animals,
this leads to enhancements in neuronal activity (that is, physiological salience) and,
behaviourally, to a higher probability of detecting the event, localizing in space and
orienting to it. c | The basic developmental chronology of sensory responsiveness within the
deep layers of the cat SC is shown. Some neurons are already responsive to touch
(somatosensation) prenatally. Hearing (audition) becomes effective in activating some SC
neurons before the end of the first week of age and sight (vision) at approximately 3 weeks.
Despite the convergence of inputs that produces multisensory neurons early in life, these
neonatal multisensory neurons cannot yet integrate their cross-modal inputs. This capability
for multisensory integration does not appear until approximately 4 weeks of age and
gradually matures until the adult-like condition is achieved after several months. Part a is
adapted with permission from REF. 101, The American Physiological Society. Part b is
adapted with permission from Stein, Barry E., and M. Alex Meredith., The Merging of the
Stein et al. Page 27
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

Senses, Figure 8.1, © 1993 Massachusetts Institute of Technology, by permission of The
MIT Press.
Stein et al. Page 28
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

Figure 2. Developmental profile of multisensory integration in the cat superior colliculus
Multisensory neurons (shown in red) first appear in the second postnatal week and steadily
increase in number thereafter, nearing adult levels by postnatal week 20. However, the first
neurons with multisensory integration capabilities (shown in green) are not seen until the
fourth week of life. Thereafter, their incidence increases roughly in parallel with the total
incidence of multisensory neurons. The multisensory responses of neurons with
multisensory integration capabilities are nearly always depressed by deactivating the
association cortex, from which descending cortico-collicular afferents originate (shown in
blue). Although the timing of these three developmental trajectories is parallel, the delay in
the development of multisensory integration is consistent with the idea that the ability to
respond to multiple modalities and the ability to integrate the information they provide are
different phenomena, which are mediated by related but not identical developmental
processes. Inset bar graphs provide sample responses from individual multisensory neurons
at three different ages, one before the development of multisensory integration capabilities
(left), one after this development (middle) and one from an adult animal. The responses
from the adult neuron are also displayed as ‘impulse rasters’ in which each dot represents a
single impulse and each row (ordered from bottom to top) represents the response to a single
stimulus presentation. The grey bars show multisensory enhancement (ME). A, auditory; S,
somatosensory; V, visual; VA, visual–auditory; VS, visual–somatosensory. Republished
with permission of Society for Neuroscience, from Development of multisensory neurons
and multisensory integration in cat superior colliculus. Wallace, M. T. & Stein, B. E. 17,
1997; permission conveyed through Copyright Clearance Center, Inc.
Stein et al. Page 29
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

Figure 3. A synergy between unisensory subregions of the association cortex drives multisensory
integration capabilities in the mature superior colliculus
Depicted is the degree of multisensory enhancement (ME) above the largest unisensory
response in a typical visual–auditory superior colliculus neuron when the auditory subregion
(the auditory field of the anterior ectosylvian sulcus (FAES)) and/or visual subregion
(anterior ectosylvian visual area (AEV)) of the AES was reversibly deactivated. In the
control condition, the average multisensory response exceeded the largest average
unisensory component response (shown as 100% response on the graph). However,
deactivation of either cortical subregion alone eliminated this multisensory enhancement,
rendering the multisensory response statistically indistinguishable (denoted by NS) from the
largest unisensory response. Subsequent reactivation after each deactivation series restored
the neuron’s functional capabilities. Asterisks indicate statistical significance.
Stein et al. Page 30
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

Figure 4. The development of multisensory integration depends on concordant experience with
cross-modal cues
Depicted are the responses of exemplar neurons that illustrate the most common outcomes
of four different rearing conditions. Shown for each exemplar are summary histograms
describing the visual (V), auditory (A) and multisensory (VA) responses as the average
number of impulses elicited by each stimulus. Also indicated (dashed line) is the sum of the
average V and A responses in each condition, as well as the percentage increase elicited by
their combined presentation (horizontal lines above each bar indicate the SEM, NS indicates
not significantly different from the greatest unisensory response in that condition and
asterisks indicate a statistically significant difference). Rearing with visual experience but
with degraded auditory experience (that is, noise rearing, left), with auditory experience but
without visual experience (that is, dark rearing, right) or with random independent visual
and auditory experience (bottom) yields multisensory superior colliculus (SC) neurons
lacking multisensory integration capabilities, as shown by the lack of significant difference
between the bars representing the multisensory (VA) response and the strongest unisensory
(V or A) response. However, rearing with concordant VA experience (top middle) allows
SC neurons to develop their multisensory integration capabilities, as shown by the
significant increase in the VA response, which in many cases exceeded even the sum of a
neuron’s unisensory responses. Republished with permission of Society for Neuroscience,
from Incorporating cross-modal statistics in the development and maintenance of
multisensory integration. Xu, J., Yu, L., Rowland, B. A., Stanford, T. R. & Stein, B. E. 32,
2012; permission conveyed through Copyright Clearance Center, Inc.
Stein et al. Page 31
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

Figure 5. Deactivating the association cortex in early life delays the development of multisensory
integration
Unilateral muscimol-infused implants were used to deactivate neurons in the association
cortex during the period in early life when superior colliculus (SC) multisensory integration
capabilities are first being instantiated. During this time period (not shown), these cortical
neurons were unable to process (contralateral) sensory information and were unable to
influence their ipsilateral SC target neurons (which also respond to contralateral stimuli). a |
When the animals had matured to 1.5 years of age, they were tested on their ability to locate
events in space. Their performance was significantly impaired. They were unable to show
the normal enhanced localization ability to events in contralateral space that had both visual
and auditory components. However, these performance benefits were normal when the
events were in ipsilateral space. This behavioural deficit was paralleled by a physiological
deficit in ipsilateral SC neurons (left inset). Most failed to produce a better response to the
visual–auditory combination than to the most effective of these stimuli individually
(multisensory enhancement (ME)). b | However, when animals were re-tested on the same
task at 4 years of age, behavioural performance on both sides of space was equivalent, and
the SC physiological deficits seemed to have been resolved as well (inset). Apparently, the
circuit was still able to acquire the experience needed to develop its multisensory integration
capability during adulthood (albeit much more slowly).
Stein et al. Page 32
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

Figure 6. Adult plasticity in multisensory integration
Illustrated are three conditions in which multisensory (visual–auditory (VA)) plasticity has
been shown. a | A raster display (each dot represents one impulse and each row represents
the response to a single stimulus presentation, ordered bottom to top) shows the effect of
repeated presentations of sequentially arranged spatially concordant VA stimuli on a
multisensory superior colliculus (SC) neuron. Square waves atop the display represent the
stimuli. Their repetition increased the magnitude and duration of this characteristic neuron’s
response to the first stimulus in the sequence and decreased the latency of the response to
the second, leading to the minimization of the temporal gap between the response trains. The
changes can be seen by comparing the first set of trials in the grey zone at the bottom of the
raster, with the last set of trials, also in grey, at the top. b | The visual and auditory receptive
fields of an exemplar SC neuron are shown on the left on a polar plot of VA space (each
concentric circle is 10 degrees). Repeated presentation of spatiotemporally concordant VA
stimuli increased both the multisensory and unisensory responses of this characteristic
neuron. The results of preliminary tests illustrated on the pre-exposure graph show that there
was no auditory response (A) and the average multisensory response (VA) was 76% greater
than the largest unisensory (V) response (and thus greater than their sum (dashed line)).
After repeated exposure to the VA stimulus, magnitudes of all responses (post-exposure
graph) were enhanced, and the previously subthreshold (auditory) input was ‘exposed’. c |
Multisensory neurons in adult dark-reared animals initially do not integrate cross-modal
stimuli but can be rapidly trained to do so by repeated exposure to spatiotemporally
concordant VA stimuli. The visual and auditory receptive fields of three exemplar neurons
are shown on the left on a polar plot of VA space (each concentric circle is 10 degrees). At
the bottom are the numbers of cross-modal exposures provided to each (3,600–50,400). In
the middle are the raster displays in response to V, A and AV stimuli, and to the right are
graphs of the average impulse counts in each condition. The receptive fields retain immature
Stein et al. Page 33
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

(that is, large) sizes (compared with that shown in part b for example) with this
impoverished sensory experience despite developing their integrative capability, and the
magnitude of the integrated multisensory response is larger in neurons with more cross-
modal experience (exposure). The horizontal line above each bar represents the SEM.
Asterisks indicate statistical significance. Part a is republished with permission of Society
for Neuroscience, from Adult plasticity in multisensory neurons: short-term experience-
dependent changes in the superior colliculus. Yu, L., Stein, B. E. & Rowland, B. A. 29,
2009; permission conveyed through Copyright Clearance Center, Inc. Part b is reprinted
with permission from REF. 145, The American Physiological Society. Part c is republished
with permission of Society for Neuroscience, from Initiating the development of
multisensory integration by manipulating sensory experience. Yu, L., Rowland, B. A. &
Stein, B. E. 30, 2010; permission conveyed through Copyright Clearance Center, Inc.
Stein et al. Page 34
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript

Figure 7. Learning to integrate requires a neuron to experience both cues simultaneously
Shown on the left are schematics of the receptive fields (RFs) of three visual–auditory (VA)
neurons. The visual (icon of a light bulb) and auditory (icon of a speaker) stimuli fell into
one or both RFs of a given neuron and were repeatedly presented in close temporal
proximity. As shown in the summary histograms to the right, only the neuron in which both
stimuli were in their respective RFs ultimately developed the capability to integrate those
inputs and showed a significantly enhanced multisensory response (indicated by the
asterisks). The horizontal line above each bar represents the SEM. NS, not statistically
significant. Republished with permission of Society for Neuroscience, from Initiating the
development of multisensory integration by manipulating sensory experience. Yu, L.,
Rowland, B. A. & Stein, B. E. 30, 2010; permission conveyed through Copyright Clearance
Center, Inc.
Stein et al. Page 35
Nat Rev Neurosci
. Author manuscript; available in PMC 2014 October 31.
NIH-PA Author Manuscript NIH-PA Author Manuscript NIH-PA Author Manuscript