![A jumble of splotches or a face? [If you have difficulty 'seeing' the face, try looking through half-closed eyes-Editor.]](https://www.researchgate.net/profile/William-Hirstein/publication/233556531/figure/fig3/AS:669981175975949@1536747368632/A-jumble-of-splotches-or-a-face-If-you-have-difficulty-seeing-the-face-try-looking_Q320.jpg)
Content uploaded by William Hirstein
Author content
All content in this area was uploaded by William Hirstein on Jan 01, 2014
Content may be subject to copyright.
Content uploaded by William Hirstein
Author content
All content in this area was uploaded by William Hirstein on Jan 01, 2014
Content may be subject to copyright.
Readings for
Codes, Cues, Clues & Affordances
Presentation by Michael Lissack
At “Emergences of Designs”
March 25, 2006
Washington Academy of Sciences
National Science Foundation
Contents:
High-Level Perception, Representation, and Analogy:
A Critique of Artificial Intelligence Methodology
David Chalmers, Robert French, Douglas Hofstadter 3
The Science of Art: A Neurological Theory of Aesthetic Experience
V.S. Ramachandran and William Hirstein 39
Cracking the code of art’s allure
Anthony Freeman 60
Art and the Brain: Editorial Introduction
Joseph A. Goguen 70
Three Laws of Qualia: What Neurology Tells Us about the Biological
Functions of Consciousness, Qualia and the Self
V. S. Ramachandran and William Hirstein 85
Some Speculative Hypotheses about the Nature and Perception of
Dance and Choreography
Ivar Hagendoorn 115
Visualization as Interpretive Practice: The Case of Detective Fiction
Andrea K. Laue 147
Code and Myth: An Introduction
Benjamin Bratton 155
High-Level Perception, Representation, and Analogy:
A Critique of Artificial Intelligence Methodology
David J. Chalmers, Robert M. French, Douglas R. Hofstadter
Center for Research on Concepts and Cognition
Indiana University
Bloomington, Indiana 47408
CRCC Technical Report 49 — March 1991
E-mail addresses:
dave@cogsci.indiana.edu
french@cogsci.indiana.edu
dughof@cogsci.indiana.edu
To appear in Journal of Experimental and Theoretical Artificial Intelligence.
3
3
High-Level Perception, Representation, and Analogy:
A Critique of Artificial Intelligence Methodology
Abstract
High-level perception—the process of making sense of complex data at an abstract, conceptual
level—is fundamental to human cognition. Through high-level perception, chaotic environmen-
tal stimuli are organized into the mental representations that are used throughout cognitive pro-
cessing. Much work in traditional artificial intelligence has ignored the process of high-level
perception, by starting with hand-coded representations. In this paper, we argue that this dis-
missal of perceptual processes leads to distorted models of human cognition. We examine some
existing artificial-intelligence models—notably BACON, a model of scientific discovery, and the
Structure-Mapping Engine, a model of analogical thought—and argue that these are flawed pre-
cisely because they downplay the role of high-level perception. Further, we argue that perceptu-
al processes cannot be separated from other cognitive processes even in principle, and therefore
that traditional artificial-intelligence models cannot be defended by supposing the existence of a
“representation module” that supplies representations ready-made. Finally, we describe a model
of high-level perception and analogical thought in which perceptual processing is integrated with
analogical mapping, leading to the flexible build-up of representations appropriate to a given
context.
4
4
1 The Problem of Perception
One of the deepest problems in cognitive science is that of understanding how people make
sense of the vast amount of raw data constantly bombarding them from their environment. The
essence of human perception lies in the ability of the mind to hew order from this chaos, whether
this means simply detecting movement in the visual field, recognizing sadness in a tone of voice,
perceiving a threat on a chessboard, or coming to understand the Iran–Contra affair in terms of
Watergate.
It has long been recognized that perception goes on at many levels. Immanuel Kant divided
the perceptual work of the mind into two parts: the faculty of Sensibility, whose job it is to pick
up raw sensory information, and the faculty of Understanding, which is devoted to organizing
these data into a coherent, meaningful experience of the world. Kant found the faculty of
Sensibility rather uninteresting, but he devoted much effort to the faculty of Understanding. He
went so far as to propose a detailed model of the higher-level perceptual processes involved,
dividing the faculty into twelve Categories of Understanding.
Today Kant’s model seems somewhat baroque, but his fundamental insight remains valid.
Perceptual processes form a spectrum, which for convenience we can divide into two
components. Corresponding roughly to Kant’s faculty of Sensibility, we have low-level
perception, which involves the early processing of information from the various sensory
modalities. High-level perception, on the other hand, involves taking a more global view of this
information, extracting meaning from the raw material by accessing concepts, and making sense
of situations at a conceptual level. This ranges from the recognition of objects to the grasping of
abstract relations, and on to understanding entire situations as coherent wholes.
Low-level perception is far from uninteresting, but it is high-level perception that is most
relevant to the central problems of cognition. The study of high-level perception leads us directly
to the problem of mental representation. Representations are the fruits of perception. In order for
raw data to be shaped into a coherent whole, they must go through a process of filtering and
organization, yielding a structured representation that can be used by the mind for any number of
purposes. A primary question about representations, currently the subject of much debate,
concerns their precise structure. Of equal importance is the question of how these representations
might be formed in the first place, via a process of perception, starting from raw data. The
process of representation-formation raises many important questions: How are representations
influenced by context? How can our perceptions of a situation radically reshape themselves when
necessary? Where in the process of perception are concepts accessed? Where does meaning
enter, and where and how does understanding emerge?
The main thesis of this paper is that high-level perception is deeply interwoven with other
1
5
5
cognitive processes, and that researchers in artificial intelligence must therefore integrate
perceptual processing into their modeling of cognition. Much work in artificial intelligence has
attempted to model conceptual processes independently of perceptual processes, but we will
argue that this approach cannot lead to a satisfactory understanding of the human mind. We will
examine some existing models of scientific discovery and analogical thought in support of this
claim, and will argue that the exclusion of perceptual processes from these models leads to
serious limitations. The intimate link between analogical thought and high-level perception will
be investigated in detail, and we will describe a computational model in which the two processes
are integrated.
Low-level and high-level perception
The lowest level of perception occurs with the reception of raw sensory information by
various sense organs. Light impinges on the retina, sound waves cause the eardrum to vibrate,
and so on. Other processes further along the information-processing chain may also be usefully
designated as low-level. In the case of vision, for instance, after information has passed up the
optic nerve, much basic processing occurs in the lateral geniculate nuclei and the primary visual
cortex, as well as the superior colliculus. Included here is the processing of brightness contrasts,
of light boundaries, and of edges and corners in the visual field, and perhaps also location
processing.
Low-level perception is given short shrift in this paper, as it is quite removed from the more
cognitive questions of representation and meaning. Nonetheless, it is an important subject of
study, and a complete theory of perception will necessarily include low-level perception as a
fundamental component.
The transition from low-level to high-level perception is of course quite blurry, but we may
delineate it roughly as follows. High-level perception begins at that level of processing where
concepts begin to play an important role. Processes of high-level perception may be subdivided
again into a spectrum from the concrete to the abstract. At the most concrete end of the spectrum,
we have object recognition, exemplified by the ability to recognize an apple on a table, or to pick
out a farmer in a wheat field. Then there is the ability to grasp relations. This allows us to
determine the relationship between a blimp and the ground (“above”), or a swimmer and a
swimming pool (“in”). As one moves further up the spectrum towards more abstract relations
(“George Bush is in the Republican Party”), the issues become distant from particular sensory
modalities. The most abstract kind of perception is the processing of entire complex situations,
such as a love affair or a war.
One of the most important properties of high-level perception is that it is extremely flexible.
2
6
6
A given set of input data may be perceived in a number of different ways, depending on the
context and the state of the perceiver. Due to this flexibility, it is a mistake to regard perception
as a process that associates a fixed representation with a particular situation. Both contextual
factors and top-down cognitive influences make the process far less rigid than this. Some of the
sources of this flexibility in perception are as follows.
Perception may be influenced by belief. Numerous experiments by the “New Look” theorists
in psychology in the 1950’s (e.g., Bruner 1957) showed that our expectations play an important
role in determining what we perceive even at quite a low level. At a higher level, that of
complete situations, such influence is ubiquitous. Take for instance the situation in which a hus-
band walks in to find his wife sitting on the couch with a male stranger. If he has a prior belief
that his wife has been unfaithful, he is likely to perceive the situation one way; if he believes that
an insurance salesman was due to visit that day, he will probably perceive the situation quite dif-
ferently.
Perception may be influenced by goals. If we are trying to hike on a trail, we are likely to
perceive a fallen log as an obstacle to be avoided. If we are trying to build a fire, we may
perceive the same log as useful fuel for the fire. Another example: Reading a given text may
yield very different perceptions, depending on whether we are reading it for content or proof-
reading it.
Perception may be influenced by external context. Even in relatively low-level perception, it
is well known that the surrounding context can significantly affect our perception of visual
images. For example, an ambiguous figure halfway between an “A” and an “H” is perceived one
way in the context of “C—T”, and another in the context of “T—E”. At a higher level, if we
encounter somebody dressed in tuxedo and bow-tie, our perception of them may differ depending
on whether we encounter them at a formal ball or at the beach.
Perceptions of a situation can be radically reshaped where necessary. In Maier’s well-
known two-string experiment (Maier 1931), subjects are provided with a chair and a pair of
pliers, and are told to tie together two strings hanging from the ceiling. The two strings are too
far apart to be grasped simultaneously. Subjects have great difficulty initially, but after a number
of minutes some of them hit upon the solution of tying the pliers to one of the strings, and
swinging the string like a pendulum. Initially, the subjects perceive the pliers first and foremost
as a special tool; if the weight of the pliers is perceived at all, it is very much in the background.
To solve this problem, subjects have to radically alter the emphasis of their perception of the pair
of pliers. Its function as a tool is set aside, and its weightiness is brought into the foreground as
the key feature in this situation.
The distinguishing mark of high-level perception is that it is semantic: it involves drawing
meaning out of situations. The more semantic the processing involved, the greater the role played
3
7
7
by concepts in this processing, and thus the greater the scope for top-down influences. The most
abstract of all types of perception, the understanding of complete situations, is also the most
flexible.
Recently both Pylyshyn (1980) and Fodor (1983) have argued against the existence of top-
down influences in perception, claiming that perceptual processes are “cognitively impenetrable”
or “informationally encapsulated”. These arguments are highly controversial, but in any case
they apply mostly to relatively low-level sensory perception. Few would dispute that at the
higher, conceptual level of perception, top-down and contextual influences play a large role.
2Artificial Intelligence and the Problem of Representation
The end product of the process of perception, when a set of raw data has been organized into
a coherent and structured whole, is a representation. Representations have been the object of
much study and debate within the field of artificial intelligence, and much is made of the
“representation problem”. This problem has traditionally been phrased as “What is the correct
structure for mental representations?”, and many possibilities have been suggested, ranging from
predicate calculus through frames and scripts to semantic networks and more. We may divide
representations into two kinds: long-term knowledge representations that are stored passively
somewhere in the system, and short-term representations that are active at a given moment in a
particular mental or computational process. (This distinction corresponds to the distinction
between long-term memory and working memory.) In this discussion, we will mostly be
concerned with short-term, active representations, as it is these that are the direct product of
perception.
The question of the structure of representations is certainly an important one, but there is
another, related problem that has not received nearly as much attention. This is that of
understanding how such a representation could be arrived at, starting from environmental data.
Even if it were possible to discover an optimal type of representational structure, this would leave
unresolved two important problems, namely:
The problem of relevance: How is it decided which subsets of the vast amounts of
data from the environment get used in various parts of the representational structure?
Naturally, much of the information content at the lowest level will be quite irrelevant
at the highest representational level. To determine which parts of the data are relevant
to a given representation, a complex filtering process is required.
The problem of organization: How are these data put into the correct form for the
4
8
8
representation? Even if we have determined precisely which data are relevant, and
we have determined the desired framework for the representation—a frame-based
representation, for instance—we still face the problem of organizing the data into the
representational form in a useful way. The data do not come prepackaged as slots and
fillers, and organizing them into a coherent structure is likely to be a highly non-trivi-
al task.
These questions, taken together, amount in essence to the problem of high-level perception,
translated into the framework of artificial intelligence.
The traditional approach in artificial intelligence has been to start by selecting not only a
preferred type of high-level representational structure, but also the data assumed to be relevant to
the problem at hand. These data are organized by a human programmer who appropriately fits
them into the chosen representational structure. Usually, researchers use their prior knowledge
of the nature of the problem to hand-code a representation of the data into a near-optimal form.
Only after all this hand-coding is completed is the representation allowed to be manipulated by
the machine. The problem of representation-formation, and thus the problem of high-level
perception, is ignored. (These comments do not, of course, apply to work in machine vision,
speech processing, and other perceptual endeavors. However, work in these fields usually stops
short of modeling processes at the conceptual level and is thus not directly relevant to our
critique of high-level cognitive modeling.)
The formation of appropriate representations lies at the heart of human high-level cognitive
abilities. It might even be said that the problem of high-level perception forms the central task
facing the artificial-intelligence community: the task of understanding how to draw meaning out
of the world. It might not be stretching the point to say that there is a “meaning barrier”, which
has rarely been crossed by work in AI. On one side of the barrier, some models in low-level
perception have been capable of building primitive representations of the environment, but these
are not yet sufficiently complex to be called “meaningful”. On the other side of the barrier, much
research in high-level cognitive modeling has started with representations at the conceptual level,
such as propositions in predicate logic or nodes in a semantic network, where any meaning that is
present is already built in. There has been very little work that bridges the gap between the two.
Objectivism and traditional AI
Once AI takes the problem of representation-formation seriously, the next stage will be to
deal with the evident flexibility of human high-level perceptual processes. As we have seen,
objects and situations can be comprehended in many different ways, depending on context and
5
9
9
top-down influences. We must find a way of ensuring that AI representations have a
corresponding degree of flexibility. William James, in the late nineteenth century, recognized
this aspect of cognitive representations:
“There is no property ABSOLUTELY essential to one thing. The same property which figures as
the essence of a thing on one occasion becomes a very inessential feature upon another. Now that
I am writing, it is essential that I conceive my paper as a surface for inscription. . . . But if I wished
to light a fire, and no other materials were by, the essential way of conceiving the paper would be
as a combustible material. . . . The essence of a thing is that one of its properties which is so
important for my interests that in comparison with it I may neglect the rest. . . . The properties
which are important vary from man to man and from hour to hour. . . . many objects of daily
use—as paper, ink, butter, overcoat—have properties of such constant unwavering importance,
and have such stereotyped names, that we end by believing that to conceive them in those ways is
to conceive them in the only true way. Those are no truer ways of conceiving them than any
others; there are only more frequently serviceable ways to us.” (James 1890, pp. 222–224)
James is saying, effectively, that we have different representations of an object or situation at dif-
ferent times. The representational process adapts to fit the pressures of a given context.
Despite the work of philosopher-psychologists such as James, the early days of artificial
intelligence were characterized by an objectivist view of perception, and of the representation of
objects, situations, and categories. As the linguist George Lakoff has characterized it, “On the
objectivist view, reality comes complete with a unique correct, complete structure in terms of
entities, properties and relations. This structure exists, independent of any human understanding.”
(Lakoff 1987, p. 159) While this objectivist position has been unfashionable for decades in
philosophical circles (especially after Wittgenstein’s work demonstrating the inappropriateness of
a rigid correspondence between language and reality), most early work in AI implicitly accepted
this set of assumptions.
The Physical Symbol System Hypothesis (Newell & Simon 1976), upon which most of the
traditional AI enterprise has been built, posits that thinking occurs through the manipulation of
symbolic representations, which are composed of atomic symbolic primitives. Such symbolic
representations are by their nature somewhat rigid, black-and-white entities, and it is difficult for
their representational content to shift subtly in response to changes in context. The result, in prac-
tice—irrespective of whether this was intended by the original proponents of this framework—is
a structuring of reality that tends to be as fixed and absolute as that of the objectivist position
outlined above.
By the mid-seventies, a small number of AI researchers began to argue that in order to
6
10
10
progress, the field would have to part ways with its commitment to such a rigid representational
framework. One of the strongest early proponents of this view was David Marr, who noted that
“the perception of an event or object must include the simultaneous computation of several
different descriptions of it, that capture diverse aspects of the use, purpose or circumstances of the
event or object.” (Marr 1977, p. 44)
Recently, significant steps have been taken toward representational flexibility with the advent
of sophisticated connectionist models whose distributed representations are highly context-
dependent (Rumelhart & McClelland 1986). In these models, there are no representational primi-
tives in internal processing. Instead, each representation is a vector in a multi-dimensional space,
whose position is not anchored but can adjust flexibly to changes in environmental stimuli.
Consequently, members of a category are not all represented by identical symbolic structures;
rather, individual objects will be represented in subtly different ways depending upon the context
in which they are presented. In networks with recurrent connections (Elman 1990),
representations are even sensitive to the current internal state of the model. Other recent work
taking a flexible approach to representation includes the classifier-system models of Holland
(1986) and his colleagues, where genetically-inspired methods are used to create a set of
“classifiers” that can respond to diverse aspects of various situations.
In these models, a flexible perceptual process has been integrated with an equally flexible de-
pendence of action upon representational content, yielding models that respond to diverse
situations with a robustness that is difficult to match with traditional methods. Nonetheless, the
models are still somewhat primitive, and the representations they develop are not nearly as
complex as the hand-coded, hierarchically-structured representations found in traditional models;
still, it seems to be a step in the right direction. It remains to be seen whether work in more
traditional AI paradigms will respond to this challenge by moving toward more flexible and ro-
bust representational forms.
On the possibility of a representation module
It might be granted that given the difficulty of the problem of high-level perception, AI
researchers could be forgiven for starting with their representations in a made-to-order form.
They might plausibly claim that the difficult problem of representation-formation is better left
until later. But it must be realized that behind this approach lies a tacit assumption: that it is
possible to model high-level cognitive processes independently of perceptual processes. Under
this assumption, the representations that are currently, for the most part, tailored by human hands,
7
11
11
would eventually be built up by a separate lower-level facility—a “representation module” whose
job it would be to funnel data into representations. Such a module would act as a “front end” to
the models of the cognitive processes currently being studied, supplying them with the
appropriately-tailored representations.
We are deeply skeptical, however, about the feasibility of such a separation of perception
from the rest of cognition. A representation module that, given any situation, produced the single
“correct” representation for it, would have great difficulty emulating the flexibility that
characterizes human perception. For such flexibility to arise, the representational processes
would have to be sensitive to the needs of all the various cognitive processes in which they might
be used. It seems most unlikely that a single representation would suffice for all purposes. As we
have seen, for the accurate modeling of cognition it is necessary that the representation of a given
situation can vary with various contextual and top-down influences. This, however, is directly
contrary to the “representation module” philosophy, wherein representations are produced quite
separately from later cognitive processes, and then supplied to a “task-processing” module.
To separate representation-building from higher-level cognitive tasks is, we believe, impossi-
ble. In order to provide the kind of flexibility that is apparent in cognition, any fully cognitive
model will probably require a continual interaction between the process of representation-
building and the manipulation of those representations. If this proves to be the case, then the
current approach of using hand-coded representations not only is postponing an important issue
but will, in the long run, lead up a dead-end street.
We will consider this issue in greater depth later, when we discuss current research in the
modeling of analogical thought. For now, we will discuss in some detail one well-known AI
program for which great claims have been made. We argue that these claims represent a lack of
appreciation of the importance of high-level perception.
BACON: A case study
A particularly clear case of a program in which the problem of representation is bypassed is
BACON, a well-known program that has been advertised as an accurate model of scientific
discovery (Langley et al 1987). The authors of BACON claim that their system is “capable of
representing information at multiple levels of description, which enables it to discover complex
laws involving many terms”. BACON was able to “discover”, among other things, Boyle’s law
of ideal gases, Kepler’s third law of planetary motion, Galileo’s law of uniform acceleration, and
Ohm’s law.
Such claims clearly demand close scrutiny. We will look in particular at the program’s
“discovery” of Kepler’s third law of planetary motion. Upon examination, it seems that the
8
12
12
success of the program relies almost entirely on its being given data that have already been
represented in near-optimal form, using after-the-fact knowledge available to the programmers.
When BACON performed its derivation of Kepler’s third law, the program was given only
data about the planets’ average distances from the sun and their periods. These are precisely the
data required to derive the law. The program is certainly not “starting with essentially the same
initial conditions as the human discoverers”, as one of the authors of BACON has claimed (Simon
1989, p. 375). The authors’ claim that BACON used “original data” certainly does not mean that
it used all of the data available to Kepler at the time of his discovery, the vast majority of which
were irrelevant, misleading, distracting, or even wrong.
This pre-selection of data may at first seem quite reasonable: after all, what could be more
important to an astronomer-mathematician than planetary distances and periods? But here our
after-the-fact knowledge is misleading us. Consider for a moment the times in which Kepler
lived. It was the turn of the seventeenth century, and Copernicus’ De Revolutionibus Orbium
Cœlestium was still new and far from universally accepted. Further, at that time there was no
notion of the forces that produced planetary motion; the sun, in particular, was known to produce
light but was not thought to influence the motion of the planets. In that prescientific world, even
the notion of using mathematical equations to express regularities in nature was rare. And Kepler
believed—in fact, his early fame rested on the discovery of this surprising coincidence—that the
planets’ distances from the sun were dictated by the fact that the five regular polyhedra could be
fit between the five “spheres” of planetary motion around the sun, a fact that constituted seductive
but ultimately misleading data.
Within this context, it is hardly surprising that it took Kepler thirteen years to realize that
conic sections and not Platonic solids, that algebra and not geometry, that ellipses and not
Aristotelian “perfect” circles, that the planets’ distances from the sun and not the polyhedra in
which they fit, were the relevant factors in unlocking the regularities of planetary motion. In
making his discoveries, Kepler had to reject a host of conceptual frameworks that might, for all he
knew, have applied to planetary motion, such as religious symbolism, superstition, Christian
cosmology, and teleology. In order to discover his laws, he had to make all of these creative
leaps. BACON, of course, had to do nothing of the sort. The program was given precisely the set
of variables it needed from the outset (even if the values of some of these variables were
sometimes less than ideal), and was moreover supplied with precisely the right biases to induce
the algebraic form of the laws, it being taken completely for granted that mathematical laws of a
type now recognized by physicists as standard were the desired outcome.
It is difficult to believe that Kepler would have taken thirteen years to make his discovery if
his working data had consisted entirely of a list where each entry said “Planet X: Mean Distance
from Sun Y, Period Z”. If he had further been told “Find a polynomial equation relating these
9
13
13
entities”, then it might have taken him a few hours. Addressing the question of why Kepler took
thirteen years to do what BACON managed within minutes, Langley et al (1987) point to
“sleeping time, and time for ordinary daily chores”, and other factors such as the time taken in
setting up experiments, and the slow hardware of the human nervous system (!). In an interesting
juxtaposition to this, researchers in a recent study (Qin & Simon 1990) found that starting with
the data that BACON was given, university students could make essentially the same
“discoveries” within an hour-long experiment. Somewhat strangely, the authors (including one of
the authors of BACON) take this finding to support the plausibility of BACON as an accurate
model of scientific discovery. It seems more reasonable to regard it as a demonstration of the vast
difference in difficulty between the task faced by BACON and that faced by Kepler, and thus as a
reductio ad absurdum of the BACON methodology.
So many varieties of data were available to Kepler, and the available data had so many
different ways of being interpreted, that it is difficult not to conclude that in presenting their
program with data in such a neat form, the authors of BACON are inadvertently guilty of 20–20
hindsight. BACON, in short, works only in a world of hand-picked, prestructured data, a world
completely devoid of the problems faced by Kepler or Galileo or Ohm when they made their
original discoveries. Similar comments could be made about STAHL, GLAUBER, and other
models of scientific discovery by the authors of BACON. In all of these models, the crucial role
played by high-level perception in scientific discovery, through the filtering and organization of
environmental stimuli, is ignored.
It is interesting to note that the notion of a “paradigm shift”, which is central to much
scientific discovery (Kuhn 1970), is often regarded as the process of viewing the world in a
radically different way. That is, scientists’ frameworks for representing available world
knowledge are broken down, and their high-level perceptual abilities are used to organize the
available data quite differently, building a novel representation of the data. Such a new
representation can be used to draw different and important conclusions in a way that was difficult
or impossible with the old representation. In this model of scientific discovery, unlike the model
presented in BACON, the process of high-level perception is central.
The case of BACON is by no means isolated—it is typical of much work in AI, which often
fails to appreciate the importance of the representation-building stage. We will see this in more
depth in the next section, in which we take a look at the modeling of analogy.
3Models of Analogical Thought
Analogical thought is dependent on high-level perception in a very direct way. When people
make analogies, they are perceiving some aspects of the structures of two situations—the essenc-
10
14
14
es of those situations, in some sense—as identical. These structures, of course, are a product of
the process of high-level perception.
The quality of an analogy between two situations depends almost entirely on one’s perception
of the situations. If Ronald Reagan were to evaluate the validity of an analogy between the U.S.
role in Nicaragua and the Soviet Union’s role in Afghanistan, he would undoubtedly see it as a
poor one. Others might consider the analogy excellent. The difference would come from
different perceptions, and thus representations, of the situations themselves. Reagan’s internal
representation of the Nicaraguan situation is certainly quite different from Daniel Ortega’s.
Analogical thought further provides one of the clearest illustrations of the flexible nature of
our perceptual abilities. Making an analogy requires highlighting various different aspects of a
situation, and the aspects that are highlighted are often not the most obvious features. The
perception of a situation can change radically, depending on the analogy we are making.
Let us consider two analogies involving DNA. The first is an analogy between DNA and a
zipper. When we are presented with this analogy, the image of DNA that comes to mind is that of
two strands of paired nucleotides (which can come apart like a zipper for the purposes of
replication). The second analogy involves comparing DNA to the source code (i.e., non-execut-
able high-level code) of a computer program. What comes to mind now is the fact that
information in the DNA gets “compiled” (via processes of transcription and translation) into
enzymes, which correspond to machine code (i.e., executable code). In the latter analogy, the
perception of DNA is radically different—it is represented essentially as an information-bearing
entity, whose physical aspects, so important to the first analogy, are of virtually no consequence.
In cases such as these, it seems that no single, rigid representation can capture what is going
on in our heads. It is true that we probably have a single rich representation of DNA sitting pas-
sively in long-term memory. However, in the contexts of different analogical mappings, very dif-
ferent facets of this large representational structure are selected out as being relevant, by the
pressures of the particular context. Irrespective of the passive content of the long-term represen-
tation of DNA, the active content that is processed at a given time is determined by a flexible
representational process.
Furthermore, not only is analogy-making dependent on high-level perception, but the reverse
holds true as well: perception is often dependent on analogy-making itself. The high-level
perception of one situation in terms of another is ubiquitous in human thought. If we perceive
Nicaragua as “another Vietnam”, for example, the making of the analogy is fleshing out our
representation of Nicaragua. Analogical thought provides a powerful mechanism for the
enrichment of a representation of a given situation. This is well understood by good educators
and writers, who know that there is nothing like an analogy to provide a better mental picture of a
given situation. Analogies affect our perception all the time: in a love affair, for instance, it is
11
15
15
difficult to stop parallels with past romances from modulating one’s perception of the current
situation. In the large or the small, such analogical perception—the grasping of one situation in
terms of another—is so common that we tend to forget that what is going on is, in fact, analogy.
Analogy and perception are tightly bound together.
It is useful to divide analogical thought into two basic components. First, there is the process
of situation-perception, which involves taking the data involved with a given situation, and
filtering and organizing them in various ways to provide an appropriate representation for a given
context. Second, there is the process of mapping. This involves taking the representations of two
situations and finding appropriate correspondences between components of one representation
with components of the other to produce the match-up that we call an analogy. It is by no means
apparent that these processes are cleanly separable; they seem to interact in a deep way. Given
the fact that perception underlies analogy, one might be tempted to divide the process of analogy-
making sequentially: first situation perception, then mapping. But we have seen that analogy also
plays a large role in perception; thus mapping may be deeply involved in the situation-perception
stage, and such a clean division of the processes involved could be misleading. Later, we will
consider just how deeply intertwined these two processes are.
Both the situation-perception and mapping processes are essential to analogy-making, but of
the two the former is more fundamental, for the simple reason that the mapping process requires
representations to work on, and representations are the product of high-level perception. The per-
ceptual processes that produce these representations may in turn deeply involve analogical map-
ping; but each mapping process requires a perceptual process to precede it, whereas it is not the
case that each perceptual process necessarily depends upon mapping. Therefore the perceptual
process is conceptually prior, although perception and mapping processes are often temporally in-
terwoven. If the appropriate representations are already formed, the mapping process can often
be quite straightforward. In our view, the most central and challenging part of analogy-making is
the perceptual process: the shaping of situations into representations appropriate to a given
context.
The mapping process, in contrast, is an important object of study especially because of the
immediate and natural use it provides for the products of perception. Perception produces a
particular structure for the representation of a situation, and the mapping process emphasizes
certain aspects of this structure. Through the study of analogy-making, we obtain a direct
window onto high-level perceptual processes. The study of which situations people view as
analogous can tell us much about how people represent those situations. Along the same lines,
the computational modeling of analogy provides an ideal testing-ground for theories of high-level
perception. Considering all this, one can see that the investigation of analogical thought has a
huge role to play in the understanding of high-level perception.
12
16
16
Current models of analogical thought
In light of these considerations, it is somewhat disheartening to note that almost all current
work in the computational modeling of analogy bypasses the process of perception altogether.
The dominant approach involves starting with fixed, preordained representations, and launching a
mapping process to find appropriate correspondences between representations. The mapping
process not only takes center stage; it is the only actor. Perceptual processes are simply ignored;
the problem of representation-building is not even an issue. The tacit assumption of such research
is that correct representations have (somehow) already been built.
Perhaps the best-known computational model of analogy-making is the Structure-Mapping
Engine (SME) (Falkenhainer, Forbus, and Gentner 1990), based upon the structure-mapping
theory of Dedre Gentner (1983). We will examine this model within the context of our earlier
remarks. Other models of analogy-making, such as those of Burstein (1986), Carbonell (1986),
Holyoak & Thagard (1989), Kedar-Cabelli (1988), and Winston (1982), while differing in many
respects from the above work, all share the property that the problem of representation-building is
bypassed.
Let us consider one of the standard examples from this research, in which the SME program is
said to discover an analogy between an atom and the solar system. Here, the program is given
representations of the two situations, as shown in Figure 1. Starting with these representations,
SME examines many possible correspondences between elements of the first representation and
elements of the second. These correspondences are evaluated according to how well they
preserve the high-level structure apparent in the representations. The correspondence with the
highest score is selected as the best analogical mapping between the two situations.
A brief examination of Figure 1 shows that the discovery of the similar structure in these
representations is not a difficult task. The representations have been set up in such a way that the
common structure is immediately apparent. Even for a computer program, the extraction of such
common structure is relatively straightforward.
We are in broad sympathy with Gentner’s notion that the mappings in an analogy should
preserve high-level structure (although there is room to debate over the details of the mapping
process). But when the program’s discovery of the correspondences between the two situations is
a direct result of its being explicitly given the appropriate structures to work with, its victory in
finding the analogy becomes somewhat hollow. Since the representations are tailored (perhaps
unconsciously) to the problem at hand, it is hardly surprising that the correct structural
correspondences are not difficult to find. A few pieces of irrelevant information are sometimes
thrown in as decoys, but this makes the task of the mapping process only slightly more
complicated. The point is that if appropriate representations come presupplied, the hard part of
13
17
17
Figure 1. The representations used by SME in finding an analogy between the solar
system and the atom. (From Falkenhainer et al, 1990.)
the analogy-making task has already been accomplished.
Imagine what it would take to devise a representation of the solar system or an atom
independent of any context provided by a particular problem. There are so many data available:
one might, for instance, include information about the moons revolving around the planets, about
the opposite electric charges on the proton and the electron, about relative velocities, about
proximities to other bodies, about the number of moons, about the composition of the sun or the
composition of the nucleus, about the fact that the planets lie in one plane and that each planet
rotates on its axis, and so on. It comes as no surprise, in view of the analogy sought, that the only
relations present in the representations that SME uses for these situations are the following:
“attracts”, “revolves around”, “gravity”, “opposite-sign” and “greater” (as well as the
fundamental relation “cause”). These, for the most part, are precisely the relations that are
relevant factors in this analogy. The criticisms of BACON discussed earlier apply here also: the
representations used by both programs seem to have been designed with 20–20 hindsight.
A related problem arises when we consider the distinction that Gentner makes between
objects, attributes, and relations. This distinction is fundamental to the operation of SME, which
14
CAUSE
GRAVITY
MASS(sun)
REVOLVE(planet,sun)AND
CAUSE
ATTRACTS(sun, planet)
MASS(planet)
CAUSE
SOLAR-SYSTEM
CAUSE
OPPOSITE–SIGN
CHARGE(nucleus)
REVOLVE(electron, nucleus)
ATTRACTS(nucleus,electron)
CHARGE(electron)
CAUSE
RUTHERFORD-ATOM
GREATER
TEMPERATURE(sun) TEMPERATURE(planet)
GREATER
MASS(nucleus) MASS(electron)
GREATER
MASS(sun) MASS(planet)
18
18
works by mapping objects exclusively to objects and relations to relations, while paying little
attention to attributes. In the atom/solar-system analogy such things as the nucleus, the sun, and
the electrons are labeled as “objects”, while mass and charge, for instance, are considered to be
“attributes”. However, it seems most unclear that this representational division is so clean in
human thought. Many concepts, psychologically, seem to float back and forth between being
objects and attributes, for example. Consider a model of economics: should we regard “wealth”
as an object that flows from one agent, or as an attribute of the agents that changes with each
transaction? There does not appear to be any obvious a priori way to make the decision. A
similar problem arises with the SME treatment of relations, which are treated as n-place
predicates. A 3-place predicate can be mapped only to a 3-place predicate, and never to a 4-place
predicate, no matter how semantically close the predicates might be. So it is vitally important that
every relation be represented by precisely the right kind of predicate structure in every
representation. It seems unlikely that the human mind makes a rigid demarcation between 3-
place and 4-place predicates—rather, this kind of thing is probably very blurry.
Thus, when one is designing a representation for SME, a large number of somewhat arbitrary
choices have to be made. The performance of the program is highly sensitive to each of these
choices. In each of the published examples of analogies made by SME, these representations
were designed in just the right way for the analogy to be made. It is difficult to avoid the
conclusion that at least to a certain extent, the representations given to SME were constructed
with those specific analogies in mind. This is again reminiscent of BACON.
In defense of SME, it must be said that there is much of interest about the mapping process it-
self; and unlike the creators of BACON, the creators of SME have made no great claims for their
program’s “insight”. It seems a shame, however, that they have paid so little attention to the
question of just how the SME’s representations could have been formed. Much of what is
interesting in analogy-making involves extracting structural commonalities from two situations,
finding some “essence” that both share. In SME, this problem of high-level perception is swept
under the rug, by starting with preformed representations of the situations. The essence of the sit-
uations has been drawn out in advance in the formation of these representations, leaving only the
relatively easy task of discovering the correct mapping. It is not that the work done by SME is
necessarily wrong: it is simply not tackling what are, in our opinion, the really difficult issues in
analogy-making.
Such criticisms apply equally to most other work in the modeling of analogy. It is interesting
to note that one of the earliest computational models of analogy, Evans’ ANALOGY (Evans
1968), attempted to build its own representations, even if it did so in a fairly rigid manner.
Curiously, however, almost all major analogy-making programs since then have ignored the
problem of representation-building. The work of Kedar-Cabelli (1988) takes a limited step in this
15
19
19
direction by employing a notion of “purpose” to direct the selection of relevant information, but
still starts with all representations pre-built. Other researchers, such as Burstein (1986), Carbonell
(1986), and Winston (1982), all have models that differ in significant respects from the work
outlined above, but none of these addresses the question of perception.
The ACME program of Holyoak and Thagard (1989) uses a kind of connectionist network to
satisfy a set of “soft constraints” in the mapping process, thus determining the best analogical
correspondences. Nevertheless, their approach seems to have remained immune to the
connectionist notion of context-dependent, flexible representations. The representations used by
ACME are preordained, frozen structures of predicate logic; the problem of high-level perception
is bypassed. Despite the flexibility provided by a connectionist network, the program has no
ability to change its representations under pressure. This constitutes a serious impediment to the
attempts of Holyoak and Thagard to capture the flexibility of human analogical thought.
The necessity of integrating high-level perception with more abstract cognitive processing
The fact that most current work on analogical thought has ignored the problem of
representation-formation is not necessarily a damning charge: researchers in the field might well
defend themselves by saying that this process is far too difficult to study at the moment. In the
meantime, they might argue, it is reasonable to assume that the work of high-level perception
could be done by a separate “representation module”, which takes raw situations and converts
them into structured representations. Just how this module might work, they could say, is not
their concern. Their research is restricted to the mapping process, which takes these
representations as input. The problem of representation, they might claim, is a completely
separate issue. (In fact, Forbus, one of the authors of SME, has also worked on modules that
build representations in “qualitative physics”. Some preliminary work has been done on using
these representations as input to SME.)
This approach would be less ambitious than trying to model the entire perception-mapping
cycle, but lack of ambition is certainly no reason to condemn a project a priori. In cognitive sci-
ence and elsewhere, scientists usually study what seems within their grasp, and leave problems
that seem too difficult for later. If this were all there was to the story, our previous remarks might
be read as pointing out the limited scope of the present approaches to analogy, but at the same
time applauding their success in making progress on a small part of the problem. There is,
however, more to the story than this.
By ignoring the problem of perception in this fashion, artificial-intelligence researchers are
making a deep implicit assumption—namely, that the processes of perception and of mapping are
temporally separable. As we have already said, we believe that this assumption will not hold up.
16
20
20
We see two compelling arguments against such a separation of perception from mapping. The
first argument is simpler, but the second has a broader scope.
The first argument stems from the observation, made earlier, that much perception is
dependent on processes of analogy. People are constantly interpreting new situations in terms of
old ones. Whenever they do this, they are using the analogical process to build up richer
representations of various situations. When the controversial book The Satanic Verses was
attacked by Iranian Moslems and its author threatened with death, most Americans were quick to
condemn the actions of the Iranians. Interestingly, some senior figures in American Christian
churches had a somewhat different reaction. Seeing an analogy between this book and the
controversial film The Last Temptation of Christ, which had been attacked in Christian circles as
blasphemous, these figures were hesitant about condemning the Iranian action. Their perception
of the situation was significantly altered by such a salient analogy.
Similarly, seeing Nicaragua as analogous to Vietnam might throw a particular perspective on
the situation there, while seeing the Nicaraguan rebels as “the moral equivalent of the founding
fathers” is likely to give quite a different picture of the situation. Or consider rival analogies that
might be used to explain the role of Saddam Hussein, the Iraqi leader who invaded Kuwait, to
someone who knows little about the situation. If one were unsympathetic, one might describe
him as analogous to Hitler, producing in the listener a perception of an evil, aggressive figure. On
the other hand, if one were sympathetic, one might describe him as being like Robin Hood. This
could produce in the listener a perception of a relatively generous figure, redistributing the wealth
of the Kuwaitis to the rest of the Arab population.
Not only, then, is perception an integral part of analogy-making, but analogy-making is also
an integral part of perception. From this, we conclude that it is impossible to split analogy-
making into “first perception, then mapping”. The mapping process will often be needed as an
important part of the process of perception. The only solution is to give up on any clean temporal
division between the two processes, and instead to recognize that they interact deeply.
The modular approach to the modeling of analogy stems, we believe, from a perception of an-
alogical thought as something quite separate from the rest of cognition. One gets the impression
from the work of most researchers that analogy-making is conceived of as a special tool in rea-
soning or problem-solving, a heavy weapon wheeled out occasionally to deal with difficult
problems. Our view, by contrast, is that analogy-making is going on constantly in the background
of the mind, helping to shape our perceptions of everyday situations. In our view, analogy is not
separate from perception: analogy-making itself is a perceptual process.
For now, however, let us accept this view of mapping as a “task” in which representations, the
products of the perceptual process, are used. Even in this view, the temporal separation of per-
ception from mapping is, we believe, a misguided effort, as the following argument will
17
21
21
demonstrate. This second argument, unlike the previous one, has a scope much broader than just
the field of analogy-making. Such an argument could be brought to bear on almost any area
within artificial intelligence, demonstrating the necessity for “task-oriented” processes to be
tightly integrated with high-level perception.
Consider the implications of the separation of perception from the mapping process, by the
use of a separate representation module. Such a module would have to supply a single “correct”
representation for any given situation, independent of the context or the task for which it is being
used. Our earlier discussion of the flexibility of human representations should already suggest
that this notion should be treated with great suspicion. The great adaptability of high-level
perception suggests that no module that produced a single context-independent representation
could ever model the complexity of the process.
To justify this claim, let us return to the DNA example. To understand the analogy between
DNA and a zipper, the representation module would have to produce a representation of DNA
that highlights its physical, base-paired structure. On the other hand, to understand the analogy
between DNA and source code, a representation highlighting DNA’s information-carrying
properties would have to be constructed. Such representations would clearly be quite different
from each other.
The only solution would be for the representation module to always provide a representation
all-encompassing enough to take in every possible aspect of a situation. For DNA, for example,
we might postulate a single representation incorporating information about its physical, double-
helical structure, about the way in which its information is used to build up cells, about its
properties of replication and mutation, and much more. Such a representation, were it possible to
build it, would no doubt be very large. But its very size would make it far too large for immediate
use in processing by the higher-level task-oriented processes for which it was intended—in this
case, the mapping module. The mapping processes used in most current computer models of
analogy-making, such as SME, all use very small representations that have the relevant
information selected and ready for immediate use. For these programs to take as input large
representations that include all available information would require a radical change in their
design.
The problem is simply that a vast oversupply of information would be available in such a
representation. To determine precisely which pieces of that information were relevant would
require a complex process of filtering and organizing the available data from the representation.
This process would in fact be tantamount to high-level perception all over again. This, it would
seem, would defeat the purpose of separating the perceptual processes into a specialized module.
Let us consider what might be going on in a human mind when it makes an analogy.
Presumably people have somewhere in long-term memory a representation of all their knowledge
18
22
22
about, say, DNA. But when a person makes a particular analogy involving DNA, only certain
information about DNA is used. This information is brought from long-term memory and
probably used to form a temporary active representation in working memory. This second
representation will be much less complex, and consequently much easier for the mapping process
to manipulate. It seems likely that this smaller representation is what corresponds to the
specialized representations we saw used by SME above. It is in a sense a projection of the larger
representation from long-term memory—with only the relevant aspects being projected. It seems
psychologically implausible that when a person makes an analogy, their working memory is
holding all the information from an all-encompassing representation of a situation. Instead, it
seems that people hold in working memory only a certain amount of relevant information with the
rest remaining latent in long-term storage.
But the process of forming the appropriate representation in working memory is undoubtedly
not simple. Organizing a representation in working memory would be another specific example
of the action of the high-level perceptual processes—filtering and organization—responsible for
the formation of representations in general. And most importantly, this process would necessarily
interact with the details of the task at hand. For an all-encompassing representation (in long-term
memory) to be transformed into a usable representation in working memory, the nature of the task
at hand—in the case of analogy, a particular attempted mapping—must play a pivotal causal role.
The lesson to be learned from all this is that separating perception from the “higher” tasks for
which it is to be used is almost certainly a misguided approach. The fact that representations have
to be adapted to particular contexts and particular tasks means that an interplay between the task
and the perceptual process is unavoidable, and therefore that any “modular” approach to analogy-
making will ultimately fail. It is therefore essential to investigate how the perceptual and
mapping processes can be integrated.
One might thus envisage a system in which representations can gradually be built up as the
various pressures evoked by a given context manifest themselves. We will describe such a
system in the next section. In this system, not only is the mapping determined by perceptual pro-
cesses: the perceptual processes are in turn influenced by the mapping process. Representations
are built up gradually by means of this continual interaction between perception and mapping. If
a particular representation seems appropriate for a given mapping, then that representation
continues to be developed, while the mapping continues to be fleshed out. If the representation
seems less promising, then alternative directions are explored by the perceptual process. It is of
the essence that the processes of perception and mapping are interleaved at all stages. Gradually,
an appropriate analogy emerges, based on structured representations that dovetail with the final
mapping. We will examine this system in greater detail shortly.
Such a system is very different from the traditional approach, which assumes the representa-
19
23
23
tion-building process to have been completed, and which concentrates on the mapping process in
isolation. But in order to be able to deal with the great flexibility of human perception and
representation, analogy researchers must integrate high-level perceptual processes into their work.
We believe that the use of hand-coded, rigid representations will in the long run prove to be a
dead end, and that flexible, context-dependent, easily adaptable representations will be recognized
as an essential part of any accurate model of cognition.
Finally, we should note that the problems we have outlined here are by no means unique to
the modeling of analogical thought. The hand-coding of representations is endemic in traditional
AI. Any program that uses pre-built representations for a particular task could be subject to such
a “representation module” argument similar to that given above. For most purposes in cognitive
science, an integration of task-oriented processes with those of perception and representation will
be necessary.
4A Model that Integrates High-Level Perception with Analogy-Making
A model of high-level perception is clearly desirable, but a major obstacle lies in the way.
For any model of high-level perception to get off the ground, it must be firmly founded on a base
of low-level perception. But the sheer amount of information available in the real world makes
the problem of low-level perception an exceedingly complex one, and success in this area has
understandably been quite limited. Low-level perception poses so many problems that for now,
the modeling of full-fledged high-level perception of the real world is a distant goal. The gap
between the lowest level of perception (cells on the retina, pixels on the screen, waveforms of
sound) and the highest level (conceptual processes operating on complex structured
representations) is at present too wide to bridge.
This does not mean, however, that one must admit defeat. There is another route to the goal.
The real world may be too complex, but if one restricts the domain, some understanding may be
within our grasp. If, instead of using the real world, one carefully creates a simpler, artificial
world in which to study high-level perception, the problems become more tractable. In the
absence of large amounts of pixel-by-pixel information, one is led much more quickly to the
problems of high-level perception, which can then be studied in their own right.
Such restricted domains, or microdomains, can be the source of much insight. Scientists in all
fields throughout history have chosen or crafted idealized domains to study particular phenomena.
When researchers attempt to take on the full complexity of the real world without first having
some grounding in simpler domains, it often proves to be a misguided enterprise. Unfortunately,
microdomains have fallen out of favor in artificial intelligence. The “real world” modeling that
has replaced them, while ambitious, has often led to misleading claims (as in the case of
20
24
24
BACON), or to limited models (as we saw with models of analogy). Furthermore, while “real
world” representations have impressive labels—such as “atom” or “solar system”—attached to
them, these labels conceal the fact that the representations are nothing but simple structures in
predicate logic or a similar framework. Programs like BACON and SME are really working in
stripped-down domains of certain highly idealized logical forms—their domains merely appear to
have the complexity of the real world, thanks to the English words attached to these forms.
While microdomains may superficially seem less impressive than “real world” domains, the
fact that they are explicitly idealized worlds allows the issues under study to be thrown into clear
relief—something that generally speaking is not possible in a full-scale real-world problem. Once
we have some understanding of the way cognitive processes work in a restricted domain, we will
have made genuine progress towards understanding the same phenomena in the unrestricted real
world.
The model that we will examine here works in a domain of alphabetical letter-strings. This
domain is simple enough that the problems of low-level perception are avoided, but complex
enough that the main issues in high-level perception arise and can be studied. The model, the
“Copycat” program (Hofstadter 1984; Mitchell 1990; Hofstadter and Mitchell 1992), is capable of
building up its own representations of situations in this domain, and does so in a flexible, context-
dependent manner. Along the way, many of the central problems of high-level perception are
dealt with, using mechanisms that have a much broader range of application than just this
particular domain. Such a model may well serve as the basis for a later, more general model of
high-level perception.
This highly parallel, non-deterministic architecture builds its own representations and finds
appropriate analogies by means of the continual interaction of perceptual structuring-agents with
an associative concept network. It is this interaction between perceptual structures and the
concept network that helps the model capture part of the flexibility of human thought. The
Copycat program is a model of both high-level perception and analogical thought, and it uses the
integrated approach to situation perception and mapping that we have been advocating.
The architecture could be said to fall somewhere on the spectrum between the connectionist
and symbolic approaches to artificial intelligence, sharing some of the advantages of each. On
the one hand, like connectionist models, Copycat consists of many local, bottom-up, parallel
processes from whose collective action higher-level understanding emerges. On the other hand, it
shares with symbolic models the ability to deal with complex hierarchically-structured
representations.
We shall use Copycat to illustrate possible mechanisms for dealing with five important
problems in perception and analogy. These are:
21
25
25
• the gradual building-up of representations;
• the role of top-down and contextual influences;
• the integration of perception and mapping;
• the exploration of many possible paths toward a representation;
• the radical restructuring of perceptions, when necessary.
The description of Copycat given here will necessarily be brief and oversimplified, but further
details are available elsewhere (Hofstadter 1984; Mitchell and Hofstadter 1990; Mitchell 1990;
Hofstadter and Mitchell 1992).
The Copycat domain
The task of the Copycat program is to make analogies between strings of letters. For instance,
it is clear to most people that abc and iijjkkll share common structure at some level. The goal of
the program is to capture this by building, for each string, a representation that highlights this
common structure, and by finding correspondences between the two representations.
The program uses the result of this correspondence-making to solve analogy problems of the
following form: “If abc changes to abd, what does iijjkkll change to?” Once the program has
discovered common structure in the two strings abc and iijjkkll, deciding that the letter a in the
first corresponds to the group ii in the second and that c corresponds to ll, it is relatively
straightforward for it to deduce that the best answer must be iijjkkmm. The difficult task for the
program—the part requiring high-level perception—is to build the representations in the first
place. We will shortly examine in more detail just how these representations are built.
Before we begin a discussion of the details of Copycat, we should note that the program
knows nothing about the shapes of letters, their sounds, or their roles in the English language. It
does know the order of the letters in the alphabet, both forwards and backwards (to the program,
the alphabet is in no sense “circular”). The alphabet consists of 26 “platonic” letter entities, each
with no explicit relation to anything except its immediate neighbors. When instances of these
simple concepts, the letters, are combined into strings of various lengths, quite complex
“situations” can result. The task of the program is to perceive structure in these situations, and to
use this structure to make good analogies.
The architecture used by the program, incidentally, is applicable much more widely than to
just the particular domain used here. For instance, the architecture has also been implemented to
deal with the problem of perceiving structure and making analogies involving the dinner
implements on a tabletop (a microdomain with a more “real world” feel) (French 1988). An
application involving perception of the shapes and styles of visual letterforms, and generation of
22
26
26
new letterforms sharing the given style, has also been proposed (Hofstadter et al, 1987).
Building up representations
The philosophy behind the model under discussion is that high-level perception emerges as a
product of many independent but cooperating processes running in parallel. The system is at first
confronted with a raw situation, about which it knows almost nothing. Then a number of
perceptual agents swarm over and examine the situation, each discovering small amounts of local
structure adding incrementally to the system’s perception, until finally a global understanding of
the situation emerges.
These perceptual agents, called codelets, are the basic elements of Copycat’s perceptual
processing. Each codelet is a small piece of code, designed to perform a particular type of task.
Some codelets seek to establish relations between objects; some chunk objects that have been per-
ceived as related into groups; some are responsible for describing objects in particular ways; some
build the correspondences that determine the analogy; and there are various others. Each codelet
works locally on a small part of the situation. There are many codelets waiting to run at any
given time, in a pool from which one is chosen nondeterministically at every cycle. The codelets
often compete with each other, and some may even break structures that others have built up, but
eventually a coherent representation emerges.
When it starts to process a problem in the letter-string domain, Copycat knows very little
about the particular problem at hand. It is faced with three strings, of which it knows only the
platonic type of each letter, which letters are spatially adjacent to each other, and which letters are
leftmost, rightmost, and middle in each string. The building-up of representations of these strings
23
j
a
i i k k
bc
relations correspondences
groups
j
Figure 2. Examples of perceptual structures built by Copycat.
27
27
and of their interrelationships is the task of codelets. Given a string such as ppqqrrss, one
codelet might notice that the first and second letters are both instances of the same platonic letter-
type (“P”), and build a “sameness” bond between them. Another might notice that the physically
adjacent letters r and s are in fact alphabetical neighbors, and build a “successor” bond between
them. Another “grouping” codelet might chunk the two bonded letters p into a group, which can
be regarded at least temporarily as a unit. After many such codelets have run, a highly structured
representation of the situation emerges, which might, for instance, see the string as a sequence of
four chunks of two letters each, with the “alphabetic successor” relation connecting each chunk
with its right neighbor. Figure 2 gives an stripped-down example of Copycat’s perceptual struc-
turing.
Different types of codelets may come into play at different stages of a run. Certain types of
codelets, for example, can run only after certain types of structures have been discovered. In this
way, the codelets cause structure to be built up gradually, and in a context-sensitive manner. Due
to the highly nondeterministic selection of codelets, several directions can be simultaneously
explored by the perceptual process. Given the string abbccd, for instance, some codelets might
try to organize it as a sequence of “sameness” groups, a-bb-cc-d, while others might simulta-
neously try to organize it quite differently as a sequence of “successor” groups, ab-bc-cd.
Eventually, the program is likely to focus on one or the other of these possibilities, but because of
the nondeterminism, no specific behavior can be absolutely guaranteed in advance. However,
Copycat usually comes up in the end with highly structured and cognitively plausible representa-
tions of situations it is given.
The role of context and top-down influences
As we have seen, one of the most important features of high-level perception is its sensitivity
to context. A model of the perceptual process that proceeds in a manner that disregards context
will necessarily be inflexible.
The Copycat model captures the dependence of perception on contextual features by means of
an associative concept-network (Figure 3), the Slipnet, which interacts continually with the
perceptual process. Each node in this network corresponds to a concept that might be relevant in
the letter-string domain, and each node can be activated to a varying degree depending on the per-
ceived relevance of the corresponding concept to the given situation. As a particular feature of
the situation is noted, the node representing the concept that corresponds to the feature is activat-
ed in the concept network. In turn, the activation of this concept has a biasing effect on the per-
ceptual processing that follows. Specifically, it causes the creation of some number of associated
24
28
28
codelets, which are placed in the pool of codelets waiting to run. For instance, if the node
corresponding to the concept of “alphabetic successor” is activated in the Slipnet, then several
codelets will be spawned whose task is to look for successorship relations elsewhere in the situa-
tion.
Further, the activation of certain nodes means that it is more likely that associated perceptual
processes will succeed. If the “successor” node is highly active, for example, not only is it more
likely that codelets that try to build successorship relations will be spawned, but it is also more
likely that once they run, they—rather than some competing type of codelet—will succeed in
building a lasting relation as part of the representation. In both of these ways, perceptual
processing that has already been completed can have a contextual, top-down influence on
subsequent processing through activation of concepts in the Slipnet.
For instance, in the string kkrrtt it is likely that the two r’s will be perceived as a “sameness
group” (a group all of whose members are the same); such a perception will be reinforced by the
presence of two similar groups on either side, which will activate the node representing the
concept of “sameness group”. On the other hand, in the string abcijkpqrrst, the presence of the
groups abc and ijk will cause the node representing “successor group” (a group consisting of
alphabetically successive letters) to be active, making it more likely that pqr and rst will be
perceived in the same way. Here, then, it is more likely that the two adjacent r’s will be
perceived separately, as parts of two different “successor groups” of three letters each. The way
in which two neighboring r’s are perceived (i.e., as grouped or not) is highly dependent on the
context that surrounds them, and this contextual dependence is mediated by the Slipnet.
This two-way interaction between the perceptual process and the concept network is a
combination of top-down and bottom-up processing. The perceptual work performed by the
25
first
A B XYZ
last
successor predecessor
opposite
leftmost rightmost
Figure 3. A small portion of Copycat’s concept network, the Slipnet.
29
29
codelets is an inherently bottom-up process, achieved by competing and cooperating agents each
of which acts locally. The Slipnet, however, by modulating the action of the codelets, acts as a
top-down influence on this bottom-up process. The Slipnet can thus be regarded as a dynamic
controller, allowing global properties such as the activation of concepts to influence the local
action of perceptual agents. This top-down influence is vitally important, as it it ensures that per-
ceptual processes do not go on independently of the system’s understanding of the global context.
Integration of perception and mapping in analogy-making
We have already discussed the necessity of a fully-integrated system of perceptual processing
and mapping in analogy-making. The Copycat model recognizes this imperative. The situation-
perception and mapping processes take place simultaneously. Certain codelets are responsible for
building up representations of the given situations, while others are responsible for building up a
mapping between the two. Codelets of both types are in the pool together.
In the early stages of a run, perceptual codelets start to build up representations of the
individual situations. After some structure has been built up, other types of codelets begin to
make tentative mappings between the structures. From then on, the situation-perception and
mapping processes proceed hand in hand. As more structure is built within the situations, the
mapping becomes more sophisticated, and aspects of the evolving mapping in turn exert pressure
on the developing perceptions of the situations.
Consider, for example, two analogies involving the string ppqrss. If we are trying to find an
analogy between this and, say, the string aamnxx, then the most successful mapping is likely to
map the group of p’s to the group of a’s, the group of s’s to the group of x’s, and qr to the succes-
sor group mn. The most natural way to perceive the second string is in the form aa-mn-xx, and
this in turn affects the way that the first string is perceived, as three two-letter groups in the form
pp-qr-ss. On the other hand, if we are trying to find an analogy between ppqrss and the string
aijklx, then recognition of the successor group ijkl inside the latter string is likely to arouse per-
ceptual biases toward seeking successor relations and groups, so that the system will be likely to
spot the successor group pqrs within ppqrss, and to map one successor group to the other. This
leads to the original string being perceived as p-pqrs-s, which maps in a natural way to a-ijkl-x.
Thus we can see that different mappings act as different contexts to evoke quite different
perceptions of the same string of letters. This is essentially what was going on in the two
analogies described earlier involving DNA. In both cases, the representation of a given situation
is made not in isolation, but under the influence of a particular mapping.
We should note that the Copycat model makes no important distinction between structures
built for the purpose of situation-perception (such as bonds between adjacent letters, or groups of
26
30
30
letters), and those built for the purpose of mapping (such as correspondences between letters or
groups in the two strings). Both types of structure are built up gradually over time, and both
contribute to the program’s current understanding of the overall situation. The mapping
structures can themselves be regarded as perceptual structures: the mapping is simply an
understanding of the analogy as a whole.
Exploring different paths and converging on a solution
A model of perception should, in principle, be able to explore all of the different plausible
ways in which a situation might be organized into a representation. Many representations may be
possible, but some will be more appropriate than others. Copycat’s architecture of competing
codelets allows for the exploration of many different pathways toward a final structure. Different
codelets will often begin to build up structures that are incompatible with each other. This is
good—it is desirable that many possibilities be explored. In the end, however, the program must
converge on one particular representation of a given situation.
In Copycat, the goal of homing in on a particular solution is aided by the mechanism of
computational temperature. This is a number that measures the amount and quality of structure
present in the current representation of the situation. Relevant structures here include bonds,
groups, and correspondences, as well as some others. The term “quality of structure” refers to
how well different parts of the structure cohere with each other. Computational temperature is
used to control the amount of randomness in the local action of codelets. If a large amount of
good structure has been built up, the temperature will be low and the amount of randomness
allowed will be small. Under these circumstances, the system will proceed in a fairly
deterministic way, meaning that it sticks closely to a single pathway with few rival side-
explorations being considered. On the other hand, if there is little good structure, the temperature
will be high, which will lead to diverse random explorations being carried out by codelets.
At the start of a run, before any structure has been built, the temperature is maximally high, so
the system will behave in a very random way. This means that many different pathways will be
explored in parallel by the perceptual processes. If no promising structural organization emerges,
then the temperature will remain high and many different possibilities will continue to be
explored. Gradually, in most situations, certain structures will prove more promising, and these
are likely to form the basis of the final representation. At any given moment, a single structural
view is dominant, representing the system’s current most coherent worldview, but many other
tentative structures may be present in the background, competing with it.
As good structures build up, the temperature gradually falls and so the system’s exploratory
behavior becomes less random. This mean that structures that have already been built have a
27
31
31
lowered chance of being replaced by new ones and are thus favored. The more coherent a global
structure, the less likely parts of it are to be broken. As structure builds up and temperature falls,
the system concentrates more and more on developing the structure that exists. Eventually, the
program will converge on a good representation of a given situation. In practice, Copycat fre-
quently comes up in different runs with different representations for the same situation, but these
representations usually seem to be cognitively plausible. Its final “solutions” to various analogy
problems are distributed in a fashion qualitatively similar to the distributions found with human
subjects (Mitchell 1990; Hofstadter and Mitchell 1992).
The process of exploring many possibilities and gradually focusing on the most promising
ones has been called a “parallel terraced scan” (Hofstadter 1984; Hofstadter and Mitchell 1992).
The process is akin to the solution to the “two-armed bandit” problem (Holland 1975) where a
gambler has access to two slot machines with fixed but distinct probabilities of payoff. These
payoff probabilities are initially unknown to the gambler, who wishes to maximize payoffs over a
series of trials. The best strategy is to start by sampling both machines equally, but to gradually
focus one’s resources probabilistically on the machine that appears to be giving the better payoff.
The Copycat program has to perform an analogous task. To function flexibly, it has to sample
many representational possibilities and choose those that promise to lead to the most coherent
worldview, gradually settling down to a fixed representation of the situation. In both the two-
armed bandit and in Copycat, it takes time for certain possibilities to emerge as the most fruitful,
and a biased stochastic sampling technique is optimal for this purpose.
Radical restructuring
Sometimes representations that have been built up for a given situation turn out to be
inappropriate, in that they do not lead to a solution to the problem at hand. When people find
themselves in this situation, they need to be able to completely restructure their representations,
so that new ones can evolve that are more adequate for the current task. Maier’s two-string
experiment provides an example of radical restructuring; people have to forget about their initial
representation of a pair of pliers as a tool for bending things, and instead see it as a heavy weight.
In Copycat, when a representation has been built up, the temperature has necessarily gone
down, which makes it difficult to change to another representation. But it is obviously not advan-
tageous for the program to keep a representation that does not led to a solution. For this reason,
the program has a special set of mechanisms to deal with such situations.
For instance, when the program is given the analogy “If abc changes to abd, what does xyz
change to?”, it usually builds a detailed representation of abc and xyz as successor groups, and
quite reasonably maps one string to the other accordingly (a maps to x, c maps to z, etc). But
28
32
32
now, when it tries to carry out the transformation for xyz that it feels is analogous to abc
becoming abd, it finds itself blocked, since it is impossible to take the successor of the letter z
(the Copycat alphabet is non-circular). The program has hit a “snag”; the only way to deal with it
is to find an alternative representation of the situation.
The program deals with the problem firstly by raising the temperature. The temperature
shoots up to its maximal value. This produces a great deal of randomness at the codelet level.
Secondly, “breaker codelets” are brought in for the express purpose of destroying representations.
The result is that many representations that have been carefully built up are broken down. At the
same time, much activation is poured into the concept representing the source of the snag—the
concept Z—and much perceptual attention is focused on the specific z inside xyz (that is, it be-
comes very salient and attracts many codelets). This causes a significant change in the represen-
tation-building process the second time around. To make a long story short, the program is
thereby able to come up with a completely new representation of the situation, where abc is still
perceived as a successor group, but xyz is re-perceived as a predecessor group, starting from z
and going backwards. Under this new representation, the a in the first string is mapped to the z in
the second.
Now if the program attempts to complete its task, it discovers that the appropriate
transformation on xyz is to take the predecessor of the leftmost letter, and it comes up with the
insightful answer wyz. (We should stress that the program, being nondeterministic, does not
always or even consistently come up with this answer. The answer xyd is actually given more
often than wyz.) Further details are given by Mitchell and Hofstadter (1990).
This process of reperception can be regarded as a stripped-down model of a “scientific
revolution” (Kuhn 1970) in a microdomain. According to this view, when a field of science
comes up against a problem it cannot solve, clamor and confusion result in the field, culminating
in a “paradigm shift” where the problem is viewed in a completely different way. With the new
worldview, the problems may be straightforward. The radical restructuring involved in the above
letter-string problem seems quite analogous to this scientific process.
What Copycat doesn’t do
Some have argued that in employing hand-coded mechanisms such as codelets and the
Slipnet, Copycat is guilty of 20-20 hindsight in much the same fashion as BACON and SME. But
there is a large difference: BACON and SME use fixed representations, whereas Copycat devel-
ops flexible representations using fixed perceptual mechanisms. Whereas we have seen that the
use of fixed representations is cognitively implausible, it is clear that human beings at any given
time have a fixed repertoire of mechanisms available to the perceptual process. One might justifi-
29
33
33
ably ask where these mechanisms, and the corresponding mechanisms in Copycat, come from, but
this would be a question about learning. Copycat is not intended as a model of learning: its per-
formance, for instance, does not improve from one run to the next. It would be a very interesting
further step to incorporate learning processes into Copycat, but at present the program should be
taken as a model of the perceptual processes in an individual agent at a particular time.
There are other aspects of human cognition that are not incorporated into Copycat. For in-
stance, there is nothing in Copycat that corresponds to the messy low-level perception that goes
on in the visual and auditory systems. It might well be argued that just as high-level perception
exerts a strong influence on and is intertwined with later cognitive processing, so low-level per-
ception is equally intertwined with high-level perception. In the end, a complete model of high-
level perception will have to take low-level perception into account, but for now the complexity
of this task means that key features of the high-level perceptual processes must be studied in iso-
lation from their low-level base.
The Tabletop program (French and Hofstadter 1991; French 1992) takes a few steps towards
lower-level perception, in that it must make analogies between visual structures in a two-dimen-
sional world, although this world is still highly idealized. There is also a small amount of related
work in AI that attempts to combine perceptual and cognitive processes. It is interesting to note
that in this work, microdomains are almost always used. Chapman’s “Sonja” program (Chapman
1991), for instance, functions in the world of a video game. Starting from simple graphical infor-
mation, it develops representations of the situation around it and takes appropriate action. As in
Tabletop, the input to Sonja’s perceptual processes is a little more complex than in Copycat, so
that these processes can justifiably be claimed to be a model of “intermediate vision” (more close-
ly tied to the visual modality than Copycat’s high-level mechanisms, but still abstracting away
from the messy low-level details), although the representations developed are less sophisticated
than Copycat’s. Along similar lines, Shrager (1990) has investigated the central role of perceptu-
al processes in scientific thought, and has developed a program that builds up representations in
the domain of understanding the operation of a laser, starting from idealized two-dimensional in-
puts.
5Conclusion
It may sometimes be tempting to regard perception as not truly “cognitive”, something that
can be walled off from higher processes, allowing researchers to study such processes without
getting their hands dirtied by the complexity of perceptual processes. But this is almost certainly
a mistake. Cognition is infused with perception. This has been recognized in psychology for
decades, and in philosophy for longer, but artificial-intelligence research has been slow to pay
30
34
34
attention.
Two hundred years ago, Kant provocatively suggested an intimate connection between
concepts and perception. “Concepts without percepts”, he wrote, “are empty; percepts without
concepts are blind.” In this paper we have tried to demonstrate just how true this statement is,
and just how dependent on each other conceptual and perceptual processes are in helping people
make sense of their world.
“Concepts without percepts are empty.” Research in artificial intelligence has often tried to
model concepts while ignoring perception. But as we have seen, high-level perceptual processes
lie at the heart of human cognitive abilities. Cognition cannot succeed without processes that
build up appropriate representations. Whether one is studying analogy-making, scientific
discovery, or some other area of cognition, it is a mistake to try to skim off conceptual processes
from the perceptual substrate on which they rest, and with which they are tightly intermeshed.
“Percepts without concepts are blind.” Our perception of any given situation is guided by
constant top-down influence from the conceptual level. Without this conceptual influence, the
representations that result from such perception will be rigid, inflexible, and unable to adapt to the
problems provided by many different contexts. The flexibility of human perception derives from
constant interaction with the conceptual level. We hope that the model of concept-based
perception that we have described goes some way towards drawing these levels together.
Recognizing the centrality of perceptual processes makes artificial intelligence more difficult,
but it also makes it more interesting. Integrating perceptual processes into a cognitive model
leads to flexible representations, and flexible representations lead to flexible actions. This is a
fact that has only recently begun to permeate artificial intelligence, through such models as
connectionist networks, classifier systems, and the architecture presented here. Future advances
in the understanding of cognition and of perception are likely to go hand in hand, for the two
types of process are inextricably intertwined.
References
Bruner, J. (1957). On perceptual readiness. Psychological Review, 64: 123-152.
Burstein, M. (1986). Concept formation by incremental analogical reasoning and debugging. In
R. S. Michalski, J. G. Carbonell, and T. M. Mitchell (eds.), Machine learning: An artificial
intelligence approach, Vol. 2 (Los Altos, CA: Morgan Kaufmann).
31
35
35
Carbonell, J. G. (1986). Learning by analogy: Formulating and generalizing plans from past
experience. In R. S. Michalski, J. G. Carbonell, and T. M. Mitchell (eds.), Machine learning: An
artificial intelligence approach, Vol. 2 (Los Altos, CA: Morgan Kaufmann).
Chapman, D. (1991). Vision, instruction, and action. Cambridge, MA: MIT Press.
Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14: 179-212.
Evans, T. G. (1968). A program for the solution of a class of geometric-analogy intelligence-test
questions. In M. Minsky (ed.), Semantic information processing (Cambridge, MA: MIT Press).
Falkenhainer, B., Forbus, K. D., and Gentner, D. (1990). The structure-mapping engine.
Artificial Intelligence, 41: 1-63.
Fodor, J. A. (1983). The modularity of mind (Cambridge, MA: MIT Press).
French, R. M., and Hofstadter, D. R. (1991). Tabletop: A stochastic, emergent model of
analogy-making. Proceedings of the 13th annual conference of the Cognitive Science Society.
Hillsdale, NJ: Lawrence Erlbaum.
French, R. M. (1992). Tabletop: A stochastic, emergent model of analogy-making. Doctoral dis-
sertation, University of Michigan.
Gentner, D. (1983). Structure-mapping: A theoretical framework for analogy. Cognitive
Science, 7(2).
Hofstadter, D. R. (1984). The Copycat project: An experiment in nondeterminism and creative
analogies. M.I.T. AI Laboratory memo 755.
Hofstadter, D. R., and Mitchell, M. (1992). An overview of the Copycat project. In K. J.
Holyoak and J. Barnden (eds.), Connectionist Approaches to Analogy, Metaphor, and Case-Based
Reasoning (Norwood, NJ: Ablex).
Hofstadter, D. R., Mitchell, M., and French, R. M. (1987). Fluid concepts and creative analogies:
A theory and its computer implementation. CRCC Technical Report 18, Indiana University.
32
36
36
Holland, J. H. (1975). Adaptation in natural and artificial systems (Ann Arbor, MI: University of
Michigan Press).
Holland, J. H. (1986). Escaping brittleness: The possibilities of general-purpose learning
algorithms applied to parallel rule-based systems. In R. S. Michalski, J. G. Carbonell, and T. M.
Mitchell (eds.), Machine learning: An artificial intelligence approach, Vol. 2 (Los Altos, CA:
Morgan Kaufmann).
Holyoak, K. J. and Thagard, P. (1989). Analogical mapping by constraint satisfaction. Cognitive
Science, 13: 295-355.
James, W. (1890). The principles of psychology (Henry Holt & Co).
Kedar-Cabelli, S. (1988). Towards a computational model of purpose-directed analogy. In A.
Prieditis (ed.), Analogica (Los Altos, CA: Morgan Kaufmann).
Kuhn, T. (1970). The structure of scientific revolutions (2nd edition) (Chicago: University of
Chicago Press).
Lakoff, G. (1987). Women, fire and dangerous things (Chicago: University of Chicago Press).
Langley, P., Simon, H. A., Bradshaw, G. L., and Zytkow, J. M. (1987). Scientific discovery:
Computational explorations of the creative process (Cambridge, MA: MIT Press).
Maier, N. R. F (1931). Reasoning in humans: II. The solution of a problem and its appearance in
consciousness. Cognitive Psychology, 12: 181-194.
Marr, D. (1977). Artificial intelligence—a personal view. Artificial Intelligence, 9: 37-48.
Mitchell, M. (1990). Copycat: A computer model of high-level perception and conceptual
slippage in analogy-making. Doctoral dissertation, University of Michigan.
Mitchell, M., and Hofstadter, D. R. (1990). The emergence of understanding in a computer
model of concepts and analogy-making. Physica D, 42: 322-334.
Newell, A. and H. A. Simon (1976). Computer science as empirical inquiry: Symbols and
33
37
37
search. Communications of the Association for Computing Machinery, 19: 113-126.
Pylyshyn, Z. (1980). Cognition and computation. Behavioral and Brain Sciences, 3: 111-132.
Qin, Y., and Simon, H. A. (1990). Laboratory replication of scientific discovery processes.
Cognitive Science 14: 281-310.
Rumelhart, D. E., McClelland, J. L., and the PDP Research Group (1986). Parallel distributed
processing (Cambridge, MA: MIT Press).
Shrager, J. (1990). Commonsense perception and the psychology of theory formation. In J.
Shrager and P. Langley (eds.) Computational models of scientific discovery and theory formation
(San Mateo, CA: Morgan Kaufmann).
Simon, H. A. (1989). The scientist as problem solver. In D. Klahr and K. Kotovsky (eds.)
Complex information processing (Hillsdale, NJ: Lawrence Erlbaum).
Winston, P. H. (1982). Learning new principles from precedents and exercises. Artificial
Intelligence 19: 321-350.
34
38
38
V.S. Ramachandran and William Hirstein
The Science of Art
A Neurological Theory of Aesthetic Experience
We present a theory of human artistic experience and the neural mechanisms that mediate
it. Any theory of art (or, indeed, any aspect of human nature) has to ideally have three
components. (a) The logic of art: whether there are universal rules or principles; (b) The
evolutionary rationale: why did these rules evolve and why do they have the form that they
do; (c) What is the brain circuitry involved? Our paper begins with a quest for artistic uni-
versals and proposes a list of ‘Eight laws of artistic experience’ — a set of heuristics that
artists either consciously or unconsciously deploy to optimally titillate the visual areas of
the brain. One of these principles is a psychological phenomenon called the peak shift
effect: If a rat is rewarded for discriminating a rectangle from a square, it will respond
even more vigorously to a rectangle that is longer and skinnier that the prototype. We sug-
gest that this principle explains not only caricatures, but many other aspects of art. Exam-
ple: An evocative sketch of a female nude may be one which selectively accentuates those
feminine form-attributes that allow one to discriminate it from a male figure; a Boucher, a
Van Gogh, or a Monet may be a caricature in ‘colour space’ rather than form space. Even
abstract art may employ ‘supernormal’ stimuli to excite form areas in the brain more
strongly than natural stimuli. Second, we suggest that grouping is a very basic principle.
The different extrastriate visual areas may have evolved specifically to extract correla-
tions in different domains (e.g. form, depth, colour), and discovering and linking multiple
features (‘grouping’) into unitary clusters — objects — is facilitated and reinforced by
direct connections from these areas to limbic structures. In general, when object-like enti-
ties are partially discerned at any stage in the visual hierarchy, messages are sent back to
earlier stages to alert them to certain locations or features in order to look for additional
evidence for the object (and these processes may be facilitated by direct limbic activa-
tion). Finally, given constraints on allocation of attentional resources, art is most appeal-
ing if it produces heightened activity in a single dimension (e.g. through the peak shift
principle or through grouping) rather than redundant activation of multiple modules.
This idea may help explain the effectiveness of outline drawings and sketches, the savant
syndrome in autists, and the sudden emergence of artistic talent in fronto-temporal
dementia. In addition to these three basic principles we propose five others, constituting a
total of ‘eight laws of aesthetic experience’ (analogous to the Buddha’s eightfold path to
wisdom).
Journal of Consciousness Studies,6, No. 6-7, 1999, pp. 15–51
Correspondence: V.S. Ramachandran, Center For Brain and Cognition, University of California,
San Diego, La Jolla, CA 92093-0109, USA.
Journal of Consciousness Studies
www.imprint-academic.com/jcs
Due to file size constraints the plates accompanying the electronic
version of this article can be downloaded free of charge from
www.imprint.co.uk/rama
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
39
39
‘Everyone wants to understand art. Why not try to understand the song of a bird?’
Pablo Picasso
Introduction
If a Martian ethologist were to land on earth and watch us humans, he would be puz-
zled by many aspects of human nature, but surely art — our propensity to create and
enjoy paintings and sculpture — would be among the most puzzling. What biological
function could this mysterious behaviour possible serve? Cultural factors undoubt-
edly influence what kind of art a person enjoys — be it a Rembrandt, a Monet, a
Rodin, a Picasso, a Chola bronze, a Moghul miniature, or a Ming Dynasty vase. But,
even if beauty is largely in the eye of the beholder, might there be some sort of univer-
sal rule or ‘deep structure’, underlying all artistic experience? The details may vary
from culture to culture and may be influenced by the way one is raised, but it doesn’t
follow that there is no genetically specified mechanism — a common denominator
underlying all types of art. We recently proposed such a mechanism (Ramachandran
and Blakeslee, 1998), and we now present a more detailed version of this hypothesis
and suggest some new experiments. These may be the very first experiments ever
designed to empirically investigate the question of how the brain responds to art.
Many consider art to be a celebration of human individuality and to that extent it
may seem like a travesty to even search for universals. Indeed theories of visual art
range from curious anarchist views (or even worse, ‘anything goes’) to the idea that
art provides the only antidote to the absurdity or our existence — the only escape, per-
haps, from this vale of tears (Penrose, 1973). Our approach to art, in this essay, will be
to begin by simply making a list of all those attributes of pictures that people gener-
ally find attractive. Notwithstanding the Dada movement, we can then ask, Is there a
common pattern underlying these apparently dissimilar attributes, and if so, why is
this pattern pleasing to us? What is the survival value, if any, of art?
But first let us clear up some common misconceptions about visual art. When the
English colonizers first arrived in India they were offended by the erotic nudes in
temples; the hips and breasts were grossly hypertrophied, the waist abnormally thin
(Plate 1).1Similarly the Rajasthani and Moghul miniature paintings were considered
primitive because they lacked perspective. In making this judgement they were, of
course, unconsciously comparing Indian art with the ideals of Western representa-
tional art — Renaissance art in particular. What is odd about this criticism though, is
that it misses the whole point of art. The purpose of art, surely, is not merely to depict
or represent reality — for that can be accomplished very easily with a camera — but
to enhance, transcend, or indeed even to distort reality. The word ‘rasa’ appears
repeatedly in Indian art manuals and has no literal translation, but roughly it means
‘the very essence of.’ So a sculptor in India, for example, might try to portray the rasa
of childhood (Plate 2), or the rasa of romantic love, or sexual ecstasy (Plate 3), or
feminine grace and perfection (Plate 4). The artist is striving, in these images, to
strongly evoke a direct emotional response of a specific kind. In Western art, the
‘discovery’ of non-representational abstract art had to await the arrival of Picasso.
His nudes were also grotesquely distorted — both eyes on one side of the face for
example. Yet when Picasso did it, the Western art critics heralded his attempts to
THE SCIENCE OF ART 16
[1] Plates for the article can be downloaded free from www.imprint.co.uk/rama
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
40
40
‘transcend perspective’ as a profound new discovery — even though both Indian and
African art had anticipated this style by several centuries!
We suggest in this essay that artists either consciously or unconsciously deploy cer-
tain rules or principles (we call them laws) to titillate the visual areas of the brain.
Some of these laws, we believe, are original to this article — at least in the context of
art. Others (such as grouping) have been known for a long time and can be found in
any art manual, but the question of why a given principle should be effective is rarely
raised: the principle is usually just presented as a rule-of-thumb. In this essay we try
to present all (or many) of these laws together and provide a coherent biological
framework, for only when they are all considered simultaneously and viewed in a bio-
logical context do they begin to make sense. There are in fact three cornerstones to
our argument. First, what might loosely be called the ‘internal logic’ of the phenome-
non (what we call ‘laws’ in this essay). Second, the evolutionary rationale: the ques-
tion of why the laws evolved and have that particular form (e.g. grouping facilitates
object perception). And third, the neurophysiology (e.g. grouping occurs in extrastri-
ate areas and is facilitated by synchronization of spikes and direct limbic activation).
All three of these need to be in place — and must inform each other — before we can
claim to have ‘understood’ any complex manifestation of human nature — such as
art. Many earlier discussions of art, in our view, suffer from the shortcoming that they
view the problem from just one or two of these perspectives.
We should clarify at the outset that many aspects of art will not be discussed in this
article — such as matters concerning style. Indeed it may well be that much of art
really has to do with aggressive marketing and hype, and this inevitably introduces an
element of arbitrariness that complicates the picture enormously. Furthermore the
artistic ‘universals’ that we shall consider are not going to provide an instant formula
for distinguishing ‘tacky’ or ‘tourist’ art, that hangs in the lobbies of business execu-
tives, from the genuine thing — even though a really gifted artist could do so instantly
— and until we can do that we can hardly claim to have ‘understood’ art. Yet despite
these reservations, we do believe that there is at least a component to art — however
small — that IS lawful and can be analysed in accordance with the principles or laws
outlined here. Although we initially proposed these ‘laws’ in a playful spirit, we were
persuaded that there is enough merit in them to warrant publication in a philosophical
journal. If the essay succeeds in stimulating a dialogue between artists, visual physi-
ologists and evolutionary biologists, it will have adequately served its purpose.
The Essence of Art and the Peak Shift Principle
Hindu artists often speak of conveying the rasa, or ‘essence’, of something in order to
evoke a specific mood in the observer. But what exactly does this mean? What does it
mean to ‘capture the very essence’ of something in order to ‘evoke a direct emotional
response’? The answer to these questions, it turns out, provides the key to understand-
ing what art really is. Indeed, as we shall see, what the artist tries to do (either con-
sciously or unconsciously) is to not only capture the essence of something but also to
amplify it in order to more powerfully activate the same neural mechanisms that
would be activated by the original object. As the physiologist Zeki (1998) has elo-
quently noted, it may not be a coincidence that the ability of the artist to abstract the
‘essential features’ of an image and discard redundant information is essentially iden-
tical to what the visual areas themselves have evolved to do.
17 V.S. RAMACHANDRAN AND W. HIRSTEIN
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
41
41
Consider the peak shift effect — a well-known principle in animal discrimination
learning. If a rat is taught to discriminate a square from a rectangle (of say, 3:2 aspect
ratio) and rewarded for the rectangle, it will soon learn to respond more frequently to
the rectangle. Paradoxically, however, the rat’s response to a rectangle that is even
longer and skinnier (say, of aspect ratio 4:1) is even greater than it was to the original
prototype on which it was trained. This curious result implies that what the rat is
learning is not a prototype but a rule, i.e. rectangularity. We shall argue in this essay
that this principle holds the key for understanding the evocativeness of much of visual
art. We are not arguing that it’s the only principle, but that it is likely to be one of a
small subset of such principles underlying artistic experience.
How does this principle — the peak shift effect — relate to human pattern recogni-
tion and aesthetic preference? Consider the way in which a skilled cartoonist pro-
duces a caricature of a famous face, say Nixon’s. What he does (unconsciously) is to
take the average of all faces, subtract the average from Nixon’s face (to get the differ-
ence between Nixon’s face and all others) and then amplify the differences to produce
a caricature. The final result, of course, is a drawing that is even more Nixon-like than
the original. The artist has amplified the differences that characterize Nixon’s face in
the same way that an even skinnier rectangle is an amplified version of the original
prototype that the rat is exposed to. This leads us to our first aphorism: ‘All art is cari-
cature’. (This is not literally true, of course, but as we shall see, it is true surprisingly
often.) And the same principle that applies for recognizing faces applies to all aspects
of form recognition. It might seem a bit strange to regard caricatures as art but take a
second look at the Chola bronze — the accentuated hips and bust of the Goddess Par-
vati (Plate 1) and you will see at once that what you have here is essentially a carica-
ture of the female form. There may be neurons in the brain that represent sensuous,
rotund feminine form as opposed to angular masculine form and the artist has chosen
to amplify the ‘very essence’ (the rasa) of being feminine by moving the image even
further along toward the feminine end of the female/male spectrum (Plate 4). The
result of these amplifications is a ‘super stimulus’ in the domain of male/female
differences. It is interesting, in this regard, that the earliest known forms of art are
often caricatures of one sort or another; e.g. prehistoric cave art depicting animals like
bison and mammoths, or the famous Venus ‘fertility’ figures.
As a further example, look at the pair of nudes in Plate 5, a sculpture from Northern
India (circa 800 AD). No normal woman can adopt such contorted postures and yet the
sculpture is incredibly evocative — beautiful — capturing the rasa of feminine poise
and grace. To explain how he achieves this effect, consider the fact that certain pos-
tures are impossible (and unlikely) among men but possible in women because of cer-
tain anatomical differences that impose constraints on what can or cannot be done.
Now in our view what the artist has done here is to subtract the male posture from the
female posture to produce a caricature in ‘posture space’ thereby amplifying ‘fem-
inine posture’ and producing a correspondingly high limbic activation. The same can
be said of the dancer in Plate 6 or for the amorous couple (Plate 7). Again, even
though these particular, highly stylized anatomical poses are impossible (or unlikely)
it is very evocative of the ‘Sringara Rasa’ or ‘Kama rasa’ (sexual and amorous
ecstasy) because the artist is providing a ‘caricature’ that exaggerates the amorous
pose. It is as though the artist was been able to intuitively access and powerfully
stimulate neural mechanisms in the brain that represent ‘amorousness’.
THE SCIENCE OF ART 18
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
42
42
Aposture space might be realized in the form of a large set of remembered postures
of people one has observed. (Whether one might expect such a memory mapping to
exist in the ‘dorsal’ stream of visual processing, which connects with the agent’s own
body representations, or the ‘ventral’ stream, known to be used for face perception, is
an interesting question; perhaps the answer is, both). There is an obvious need to con-
nect these posture representations to the limbic system: it is quite imperative that I
recognize an attack posture, a posture — or body position — which beckons me, or
one which indicates sadness or depression, etc. The sculptors of Plates 5 and 6 relied
on this represented posture space in creating their works. The sculptor knows, con-
sciously or not, that the sight of those postures will evoke a certain sort of limbic acti-
vation when the posture is successfully represented in the posture space system — he
tells a story in this medium, we might say.
Until now we have considered caricatures in the form domain, but we know from
the pioneering work of many physiologists (Zeki, 1980; see also Livingstone and
Hubel, 1987; Allman & Kaas, 1971; Van Essen & Maunsell, 1980) that the primate
brain has specialized modules concerned with other visual modalities such as colour
depth and motion. Perhaps the artist can generate caricatures by exploiting the peak
shift effect along dimensions other than form space, e.g., in ‘colour space’ or ‘motion
space’. For instance consider the striking examples of the plump, cherub-faced nudes
that Boucher is so famous for. Apart from emphasizing feminine, neotonous baby-
like features (a peak shift in the masculine/feminine facial features domain) notice
how the skin tones are exaggerated to produce an unrealistic and absurd ‘healthy’
pink flush. In doing this, one could argue he is producing a caricature in colour space,
particularly the colours pertaining to male/female differences in skin tone. Another
artist, Robert, on the other hand, pays little attention to colour or even to form, but
tends to deliberately overemphasize the textural attributes of his objects, be they
bricks, leaves, soil, or cloth. And other artists have deliberately exaggerated (‘caric-
atured’ or produced peak shifts in) shading, highlights, illumination etc to an extent
that would never occur in a real image. Even music may involve generating peak
shifts in certain primitive, passionate primate vocalizations such as a separation cry;
the emotional response to such sounds may be partially hard-wired in our brains.
A potential objection to this scheme is that it is not always obvious in a given pic-
ture what the artist is trying to caricature, but this is not an insurmountable objection.
Ethologists have long known that a seagull chick will beg for food by pecking at its
mother’s beak. Remarkably, it will peck just as vigorously at a disembodied beak with
no mother attached or even a brown stick with a red dot at the end (the gull’s beak has
a vivid red spot near the tip). The stick with the red dot is an example of a ‘releasing
stimulus’ or ‘trigger feature’ since, as far as the chick’s visual system is concerned
this stimulus is as good as the entire mother bird. What is even more remarkable,
though, was Tinbergen’s discovery (Tinbergen, 1954) that a very long, thin brown
stick, with three red stripes at the end is even more effective in eliciting pecks than the
original beak, even though it looks nothing like a beak to a human observer.
The gull’s form recognition areas are obviously wired-up in such a way that Tinber-
gen had inadvertently produced a super stimulus, or a caricature in ‘beak space’ (e.g.
the neurons in the gull’s brain might embody the rule ‘more red contour the better’).
Indeed, if there were an art gallery in the world of the seagull, this ‘super beak’ would
qualify as a great work of art — a Picasso. Likewise, it is possible that some types of
19 V.S. RAMACHANDRAN AND W. HIRSTEIN
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
43
43
art such as cubism are activating brain mechanisms in such a way as to tap into or
even caricature certain innate form primitives which we do not yet fully understand.2
At present we have no idea what the ‘form primitives’ used by the human visual path-
ways are, but we suggest that many artists may be unconsciously producing height-
ened activity in the ‘form areas’ in a manner that is not obvious to the conscious mind,
just as it isn’t obvious why a long stick with three red stripes is a ‘super beak’. Even
the sunflowers of Van Gogh or the water lilies of Monet may be the equivalent — in
colour space — of the stick with the three stripes, in that they excite the visual neu-
rons that represent colour memories of those flowers even more effectively than a real
sunflower or water lily might.
There is also clearly a mnemonic component of aesthetic perception, including, the
autobiographical memory of the artist, and of her viewer, as well as the viewer’s more
general ‘cognitive stock’ brought to his encounter with the work. This general cogni-
tive stock includes the viewer’s memory of his encounters with the painting’s etio-
logical forebears, including those works that the artist himself was aware of. Often
paintings contain homages to earlier artists and this concept of homage fits what we
have said about caricature: the later artist makes a caricature of his acknowledged
predecessor, but a loving one, rather than the ridiculing practised by the editorial car-
toonist. Perhaps some movements in the history of art can be understood as driven by
a logic of peak shift: the new art form finds and amplifies the essence of a previous
one (sometimes many years previous, in the case of Picasso and African art).3
THE SCIENCE OF ART 20
[2] Another manifestation of this principle can be seen in the florid sexual displays of birds — that we find
so attractive. It is very likely, as suggested by Darwin, that the grotesque exaggeration of these
displays, for example the magnificent wings of the birds of paradise, is a manifestation of the peak shift
effect during mate choice — sexual selection caused by birds of each generation preferring caricatures
of the opposite sex to mate with (just as humans lean toward Playboy pinups and Chippendale
dancers). Indeed we have recently suggested (Ramachandran and Blakeslee, 1998) that many aspect of
morphological evolution (not just ‘secondary sexual characteristics’ or florid ‘ethological releasers’
and threat displays) may be the outcome of runaway selection, based on the peak shift principle. The
result would be not only the emergence and ‘quantization’ of new species, but also a progressive and
almost comical ‘caricaturization’ of phylogenetic trends of precisely the kind one sees in the evolution
of elephants or ankylosaurs. Even the quirks of fashion design (e.g. corsets becoming absurdly narrow,
shoes becoming smaller and smaller in ancient China, shrinking miniskirts) become more
comprehensible in terms of this perceptual principle. One wonders, also, whether the striking
resemblance between the accumulation of jewellery, shoes and other brightly coloured objects by
humans and the collections of bright pebbles, berries and feathers by bowerbirds building their
enormous nests is entirely coincidental.
[3] Lastly, consider the evolution of facial expressions. Darwin proposed that a ‘threat gesture’ may have
evolved from the real facial movements one makes before attacking a victim — i.e. the baring of
canines, etc. The same movement may eventually become divorced from the actual act and begin to
serve as a communication of intent — a threat. If the peak shift principle were to operate in the
recipient’s brain it is easy to see how such a ritualized signal would become progressively amplified
across generations. Darwin had a difficult time, however, explaining why gestures such as sadness
(instead of joy) seem to involve the opposite movement of facial features — e.g. lowering the corners
of the mouth — and he came up with his somewhat ad hoc ‘principle of antithesis’, which states that
somehow the opposite emotion is automatically linked to the opposite facial movements. We would
suggest, instead, that the principle of antithesis is, once again, an indirect result of the recipient’s brain
applying the peak shift principle. Once the organism has circuitry in its brain that says K is normal and J
is a smile, then it may follow automatically that L is the expression of the opposite emotion — sadness.
Whether this particular conjecture is correct or not we believe that emotional expressions analysed in
terms of the peak shift effect may begin to make more sense than they have in the past.
Another layer of complexity here is that even the perception of complex postures or actions may
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
44
44
Perceptual Grouping and Binding is Directly Reinforcing
One of the main functions of ‘early vision’ (mediated by the thirty or so extrastriate
visual areas) is to discover and delineate objects in the visual field (Marr, 1981;
Ramachandran, 1990; Pinker, 1998; Shepard, 1981) and for doing this the visual
areas rely, once again, on extracting correlations. For instance if a set of randomly
placed spots A is superimposed on another set of randomly placed dots B, they are
seen to mingle to form just a single enormous cluster. But if you now move one of the
clusters (say, A) then all the dots are instantly glued or bound together perceptually to
create an object that is clearly separate from the background cluster B. Similarly if
cluster A is made of red dots (and B is of green dots) we have no difficulty in segregat-
ing them instantly.
This brings us to our second point. The very process of discovering correlations
and of ‘binding’ correlated features to create unitary objects or events must be rein-
forcing for the organism — in order to provide incentive for discovering such correla-
tions (Ramachandran and Blakeslee, 1998). Consider the famous hidden face (Fig. 1)
21 V.S. RAMACHANDRAN AND W. HIRSTEIN
Figure 1
A jumble of splotches or a face?
[If you have difficulty ‘seeing’ the
face, try looking through half-closed
eyes — Editor.]
Figure 2
Initially seen as a jumble of splotches, once the Dalma-
tian is seen, its spots are grouped together — a pleasing
effect, caused perhaps by activation of the limbic sys-
tem by temporal lobe cortex.
require the observer to somehow internally re-enact or ‘rehearse’ the action before it is identified. For
instance, patients with apraxia (inability to perform complex skilled movements resulting from
damage to the left supramarginal gyrus) often, paradoxically, have difficulty perceiving and
recognizing complex actions performed by others. Also, there are cells in the frontal lobes thought to
be involved in the production of complex movements but which also fire when the animal perceives
the same movements performed by a the experimenter (di Pellegrino et al., 1992). This finding —
together with the peak shift effect — would help account for Darwin’s ‘principle of antithesis’, which
would otherwise seem completely mysterious. Such cells may also be activated powerfully when
viewing dynamic figural representations such as the ‘Dancing Devi’ (Plate 6).
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
45
45
or Dalmatian dog photo (Fig. 2). This is seen initially as a random jumble of
splotches. The number of potential groupings of these splotches is infinite but once
the dog is seen your visual system links only a subset of these splotches together and it
is impossible not to ‘hold on’ to this group of linked splotches. Indeed the discovery
of the dog and the linking of the dog-relevant splotches generates a pleasant ‘aha’
sensation. In ‘colour space’ the equivalent of this would be wearing a blue scarf with
red flowers if you are wearing a red skirt; the perceptual grouping of the red flowers
and your red skirt is aesthetically pleasing — as any fashion designer will tell you.
These examples suggest that there may be direct links in the brain between the
processes that discover such correlations and the limbic areas which give rise to the
pleasurable ‘rewarding’ sensations associated with ‘feature binding’. So when you
choose a blue matte to frame your painting in order to ‘pick up’ flecks of blue in the
painting you are indirectly tapping into these mechanisms.
How is such grouping achieved? As noted above, the primate brain has over two
dozen visual areas each of which is concerned with a different visual attribute such as
motion, colour, depth, form, etc. These areas are probably concerned with extracting
correlations in ‘higher dimensional’ spaces — such as ‘colour space’ or ‘motion
space’. In a regular topographic map — e.g., in area 17 — features that are close
together in physical space are also close together in the brain (which is all that is
meant by ‘map’). But now think of non-topographic maps — say a map of ‘colour
space’ — in which points that are close together in wavelength are mapped close
together in the colour area of the brain even though they may be distant from each
other physically (Barlow, 1986). Such proximity along different feature dimensions
may be useful for perceptual grouping and ‘binding’ of features that are similar
within that dimension.
This argument sounds plausible, but why should the outputs of separate vision
modules — space, colour, depth, motion, etc. — be sent directly to the limbic system
before further processing has occurred? Why not delay the reinforcement produced
by limbic activation until the object has actually been identified by neurons in infero-
temporal cortex? After all, the various Gestalt grouping processes are thought to
occur autonomously as a result of computations within each module itself (Marr,
1981) without benefit of either cross-module or ‘top down’ influences — so why
bother hooking up the separate modules themselves to limbic regions? One resolution
of this paradox might simply be that the serial, hierarchical, ‘bucket brigade’ model
of vision is seriously flawed and that eliminating ambiguity, segmenting the scene
and discovering and identifying objects do indeed rely on top down processes — at
least to some situations (Churchland et al., 1994). The visual system is often called
upon to segment the scene, delineate figure from ground and recognize objects in
very noisy environments — i.e., to defeat camouflage — and this might be easier to
accomplish if a limbic ‘reinforcement’ signal is not only fed back to early vision once
an object has been completely identified, but is evoked at each and every stage in
processing as soon as a partial ‘consistency’ and binding is achieved. This would
explain why we say ‘aha’ when the Dalmatian is finally seen in Fig. 2 — and why it is
difficult to revert back to seeing merely splotches once the dog is seen as a whole: that
particular percept is powerfully reinforced (Ramachandran and Blakeslee, 1998). In
other words, even though the grouping may be initially based on autonomous process
in each module (Marr, 1981), once a cluster of features becomes perceptually salient
THE SCIENCE OF ART 22
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
46
46
as a ‘chunk’ with boundaries (i.e. an object), it may send a signal to the limbic centres
which in turn causes you to ‘hold on’ to that chunk to facilitate further computation.
There is physiological evidence that grouping of features leads to synchronization of
the spikes (action potentials) of neurons that extract those features (Singer and Gray,
1995; Crick and Koch, 1998) and perhaps it is this synchrony that allows the signal to
be sent to the limbic pathways. (This, by the way, may be one reason why musical
consonance often involves harmonics — for example, a C-major chord — which, for
physical reasons would tend to emerge from a single object, whereas dissonant notes
are likely to emerge from two or more separate objects.)
The key idea, then, is the following (and it applies to many of our laws, not just
grouping). Given the limited attentional resources in the brain and limited neural
space for competing representations, at every stage in processing there is generated a
‘Look here, there is a clue to something potentially object-like’ signal that produces
limbic activation and draws your attention to that region (or feature) , thereby facili-
tating the processing of those regions or features at earlier stages. Furthermore, par-
tial ‘solutions’ or conjectures to perceptual problems are fed back from every level in
the hierarchy to every earlier module to impose a small bias in processing and the
final percept emerges from such progressive ‘bootstrapping’ (Ramachandran et al.,
1998). As noted above, consistency between partial high-level ‘hypotheses’ and ear-
lier low-level ensembles also generates a pleasant sensation — e.g. the Dalmatian
dog ‘hypothesis’ encourages the binding of corresponding splotches which, in turn,
further consolidate the ‘dog-like’ nature of the final percept and we feel good when it
all finally clicks in place. And what the artist tries to do, is to tease the system with as
many of these ‘potential object’ clues as possible — an idea that would help explain
why grouping and ‘perceptual problem solving (see below) are both frequently
exploited by artists and fashion designers.
The notion that art exploits grouping principles is of course not new (Gombrich,
1973; Arnheim, 1956; Penrose, 1973), but what is novel here is our claim that the
grouping doesn’t always occur ‘spontaneously’; that out of a temporary binding a
signal sent to the limbic system to reinforce the binding, and this is one source of the
aesthetic experience. For example, in Fig. 3, there are two possible stable organiza-
23 V.S. RAMACHANDRAN AND W. HIRSTEIN
Figure 3
Gestalt grouping principles. The tokens can be grouped either on the basis of ‘proximity’ (which
produces hourglasses), or ‘closure’. The latter organization is more stable and pleasing to the eye.
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
47
47
tions, one with hourglasses, and one with closure and most people find the latter
organization more pleasing than the former because the limbic activation is stronger
with this closure-based object-like percept. When artists speak of composition, or
grouping, they are probably unconsciously tapping into these very same principles.
One obvious prediction that emerges from this theory is that patients with Kluver-
Bucy syndrome — caused by bilateral amygdala destruction — should not only dis-
play problems in recognizing objects (visual agnosia) but also in segmenting them
out from noisy backgrounds, an idea that would be relatively easy to test
experimentally.
Isolating a Single Module and Allocating Attention
The third important principle (in addition to peak shift and binding) is the need to iso-
late a single visual modality before you amplify the signal in that modality. For
instance, this is why an outline drawing or sketch is more effective as ‘art’ than a full
colour photograph. This seems initially counterintuitive since one would expect that
the richer the cues available in the object the stronger the recognition signal and asso-
ciated limbic activation. This apparent objection can be overcome, however, once one
realizes that there are obvious constraints on the allocation of attentional resources to
different visual modules. Isolating a single area (such as ‘form’ or ‘depth’ in the case
of caricature or Indian art) allows one to direct attention more effectively to this one
source of information, thereby allowing you to notice the ‘enhancements’ introduced
by the artist. (And that in turn would amplify the limbic activation and reinforcement
produced by those enhancements). Consider a full-colour illustration of Nixon, with
depth, shading, skin tones and blemishes, etc. What is unique about Nixon is the form
of his face (as amplified by the caricature) but the skin tone — even though it makes
the picture more human-like — doesn’t contribute to making him ‘Nixon like’ and
therefore actually detracts from the efficacy of the form cues. Consequently, one
would predict that a full colour photo of Nixon would actually be less aesthetically
pleasing than a sketchy outline drawing that captures the essential ‘Nixon-like’ attrib-
utes of his face.
The idea that outlines are effective in art is hardly new. It has been repeated ad
nauseum by many authors, ever since David Hubel and Torsten Wiesel (1979) origi-
nally pointed out that this principle may reflect the fact that cells in the visual path-
ways are adequately stimulated by edges and are indifferent to homogeneous regions.
However this would only explain why one can get away with just using outlines —
not why outlines are actually more effective than a full colour half tone photo, which,
after all, has more information. We would argue that when the colour, skin texture,
etc. are not critical for defining the identity of the object in question (e.g. Nixon’s
face) then the extra redundant information can actually distract your limited atten-
tional resources away from the defining attributes of that object. Hence the aphorism
‘more is less’ in Art.4
Additional evidence for this view comes from the ‘savant syndrome’ — autistic
children who are ‘retarded’ and yet produce beautiful drawings. The animal drawings
of the eight-year old artist Nadia, for instance, are almost as aesthetically pleasing as
THE SCIENCE OF ART 24
[4] If this theory is correct, we can make the counterintuitive prediction that activation of the face area in
the brain — measured by fMRI — should, paradoxically, be greater for an outline sketch of a face than
for a full colour photo of a face.
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
48
48
those of Leonardo da Vinci! (Plate 8). We would argue that this is because the funda-
mental disorder in autism is a distortion of the ‘salience landscape’; they shut out
many important sensory channels thereby allowing them to deploy all their atten-
tional resources on a single channel; e.g., in ‘visual form representation’ channel in
the case of Nadia. This idea is also consistent with the ingenious theory of Snyder
(Snyder and Thomas, 1997), that savants are able to ‘directly access’ the outputs of
some of their early vision modules because they are less ‘concept driven’: the concep-
tual impoverishment that produces autism also, paradoxically, gives them better
access to earlier processes in vision. And finally, we would suggest that the ‘iso-
lation’ principle also explains the efflorescence of artistic talent that is occasionally
seen in fronto-temporal dementia in adults: a clinical phenomenon that is currently
being studied intensively in our laboratory.
These ideas allow us to make certain novel predictions: If you put luminous dots on
a person’s joints and film him or her walking in complete darkness, the complex
motion trajectories of the dots are usually sufficient to evoke a compelling impression
of a walking person — the so-called Johansson effect (Johansson, 1975). Indeed, it is
often possible to tell the sex of the person by watching the gait. However, although
these movies are often comical, they are not necessarily pleasing aesthetically. We
would argue that this is because even though you have isolated a cue along a single
dimension, i.e., motion, this isn’t really a caricature in motion space. To produce a
work of art, you would need to subtract the female motion trajectories from the male
and amplify the difference. Whether this would result in a pleasing work of kinetic art
remains to be seen.
Contrast Extraction is Reinforcing
Grouping, as we have already noted, is an important principle, but the extraction of
features prior to grouping — which involves discarding redundant information and
extracting contrast — is also ‘reinforcing’. Cells in the retina, lateral geniculate body
(a relay station in the brain) and in the visual cortex respond mainly to edges (step
changes in luminance) but not to homogeneous surface colours; so a line drawing or
cartoon stimulates these cells as effectively as a ‘half tone’ photograph. What is fre-
quently overlooked though is that such contrast extractions — as with grouping —
may be intrinsically pleasing to the eye (hence the efficacy of line drawings). Again,
though, if contrast is extracted autonomously by cells in the very earliest stages of
processing, why should the process be rewarding in itself? We suggest that the answer
once again has to do with the allocation of attention. Information (in the Shannon
sense) exists mainly in regions of change — e.g. edges — and it makes sense that such
regions would, therefore, be more attention grabbing — more ‘interesting’ — than
homogeneous areas. So it may not be coincidental that what the cells find interesting
is also what the organism as a whole finds interesting and perhaps in some circum-
stances ‘interesting’ translates into ‘pleasing’.
For the same reason, contrast along many other stimulus dimensions besides lumi-
nance, such as colour or texture, has been exploited by artists (for instance, colour
contrast is exploited by Matisse), and indeed there are cells in the different visual
areas specialized for colour contrast, or motion contrast (Allman and Kaas, 1971).
Furthermore, just as one can speak of a peak shift principle along very abstract dimen-
25 V.S. RAMACHANDRAN AND W. HIRSTEIN
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
49
49
THE SCIENCE OF ART 26
Figure 4
Camouflage. Notice that the boundary between the two types of texture (vertical vs. horizontal
lines) is clearly visible on the upper pattern (A), but is masked by the luminance boundaries on the
lower (B). (Based on M.J. Morgan; personal communication).
A
B
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
50
50
sions, contrast can also emerge in dimensions other than luminance or colour. For
instance, a nude wearing baroque (antique) gold jewellery (and nothing else) is aes-
thetically much more pleasing than a completely nude woman or one wearing both
jewellery and clothes, presumably because the homogeneity and smoothness of the
naked skin contrasts sharply with the ornateness and rich texture of the jewellery.
Whether the analogy between luminance contrast extracted by cells in the brain and
the contrast between jewels and naked skin is just a play of words or a deep unifying
principle is a question that cannot be answered given what we know about the brain.
But we do know that the attention grabbing effect of contrast must be a very important
principle in nature, since it is often used as a camouflage device by both predators and
their prey. For instance, in Fig. 4A, a texture border is very visible, but in Fig. 4B it is
almost ‘invisible’, camouflaged by the colour (black/white) borders that grab the
lion’s share of your attention.
At first the two principles we have just considered seem antithetical; grouping on
the basis of similarity is rewarding, but if so how can contrast (the very opposite of
grouping) also be rewarding? One clue comes from the fact that the two mechanisms
have different spatial constraints; grouping can occur between similar features (e.g.
colour or motion) even if they are far apart in space (e.g., the spots on the nose and tail
of a leopard). Contrast, on the other hand, usually occurs between dissimilar features
that are physically close together. Thus even though the two processes seem to be
inconsistent, they actually complement one another in that they are both concerned
with the discovery of objects — which is the main goal of vision. (Contrast extraction
is concerned with the object’s boundaries whereas grouping allows recovery of the
object’s surfaces and, indirectly, of its boundaries as well). It is easy to see then why
the two should be mutually reinforcing and rewarding to the organism.
Symmetry
Symmetry, of course, is also aesthetically pleasing as is well known to any Islamic
artist (or indeed to any child looking through a kaleidoscope) and it is thought to be
extracted very early in visual processing (Julesz, 1971). Since most biologically
important objects — such as predator, prey or mate are symmetrical, it may serve as
an early-warning system to grab our attention to facilitate further processing of the
symmetrical entity until it is fully recognised. As such, this principle complements
the other laws described in this essay; it is geared towards discovering ‘interesting’
object-like entities in the world.
Intriguingly, it has recently been shown experimentally that when choosing a mate,
animals and humans prefer symmetrical over asymmetrical ones and evolutionary
biologists have argued that this is because parasitic infestation — detrimental to fer-
tility — often produces lopsided, asymmetrical growth and development. If so, it is
hardly surprising that we have a built-in aesthetic preference for symmetry.
The Generic Viewpoint and the Bayesian Logic of Perception
Another less well known principle relates to what AI researchers refer to as ‘the
generic viewpoint’ principle, which is illustrated in Fig. 5A and B and Fig. 6A and B.
In Fig. 5A most people see a square occluding the corner of another square, even
27 V.S. RAMACHANDRAN AND W. HIRSTEIN
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
51
51
THE SCIENCE OF ART 28
Figure 5
One square is seen as occluding the other. It is hard to see A as B viewed from a unique vantage
point. The brain ‘prefers’ the generic view.
Figure 6
The flat hexagon with radiating spokes could be a cube but is never seen as one. The ‘generic’ inter-
pretation is again the brain’s preferred one.
AB
AB
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
52
52
29 V.S. RAMACHANDRAN AND W. HIRSTEIN
Figure 7
The brain’s abhorrence of ‘suspicious coincidences’ (a phrase used by Horace Barlow). Figure B is
pleasing, but A is distasteful to the eye.
A
B
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
53
53
though it could theoretically be Fig. B seen from a unique view point. The reason is
that there is an infinite set of viewpoints that could produce the class of retinal images
resembling A, but only a single, unique viewpoint that could produce retinal image A,
given the objects in B. Consequently, the visual system rejects the latter interpretation
as being highly improbable and prefers to see A as occlusion. (The same principle
applies to 6A and B; A could depict an outline of a cube seen from one specific van-
tage point, but people usually see it as a flat hexagon with spokes radiating from the
middle.) These examples illustrate the universal Bayesian logic of all perception:
your visual system abhors interpretations which rely on a unique vantage point and
favours a generic one or, more generally, it abhors suspicious coincidences (Barlow,
1980). For this reason, Fig. 7B is pleasing, whereas 7A is unattractive (palm tree and
hills). So if an artist is trying to please the eye, he too, should avoid coincidences,
such as those in 7A and 6B. Yet one must be cautious in saying this since every now
and then — given the perverse nature of art and artists — a pleasing effect can be pro-
duced by violating this principle rather than adhering to it. For instance, there is a
Picasso nude in which the improbability of the arm’s outline exactly coinciding with
that of the torso grabs the viewer’s attention — and is arguably attractive to him!
We hasten to add that the principles we have discussed so far certainly do not
exhaust all types of artistic experience. We have hardly touched on the purely sym-
bolic or allegorical aspects of some types of paintings or sculpture, or on surrealism
and modern abstract art (e.g., minimalists such as Kandinsky), not to mention
‘counter’ art such as the Dada movement. Also very puzzling is the question of why a
nude hidden by a diaphanous veil is more alluring than one seen directly in the flesh,
as pointed out by Ernst Gombrich (1973). It is as though an object discovered after a
struggle is more pleasing than one that is instantly obvious. The reason for this is
obscure but perhaps a mechanism of this kind ensures that the struggle itself is re-
inforcing — so that you don’t give up too easily — whether looking for a leopard
behind foliage or a mate hidden in the mist. On the other hand, we suspect that surre-
alist art really doesn’t have much to do with visual representations per se but involves
playing with links between vision and semantics, thereby taking it closer to the meta-
phorical ambiguities of poetry and language than to the purely visual appeal of a
Picasso, a Rodin, or a Chola bronze. For example, in his erotic masterpiece ‘Young
virgin autosodomised by her own chastity’ (1954), Dali has used the male penis to
represent the female buttocks and genitalia. The medium and message ‘resonate’
since they both pertain to sex but they are also in subtle conflict since they depict
‘opposite’ sexes! The result is an image pleasing on many levels simultaneously. This
playful, whimsical, aspect of art, often involving the humorous juxtaposition of com-
plementary — or sometimes even incongruous — elements, is perhaps the most enig-
matic aspect of our aesthetic experience, one which we have hardly touched upon in
this essay. Another aspect of art that we have not dealt with is style, although one can
see how once a style or trend is set in motion the peak shift principle can certainly help
amplify it.
Art as Metaphor
The use of visual metaphors in art is well known. For instance, in Plate 9, the languor-
ous, sensuous pose of the woman mimics the tree branch above — the curves match
her curves and perhaps the tree’s fertility is a metaphor for her youthfulness. (Just as
THE SCIENCE OF ART 30
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
54
54
in Plate 4, the fruit in the tree echoes the curve of the breasts as well as the abdomen.)
There are countless examples of this sort in both Eastern and Western art and yet the
question is rarely raised as to why visual ‘puns’ or allegories should be aesthetically
pleasing. A metaphor is a mental tunnel between two concepts or percepts that appear
grossly dissimilar on the surface. When Shakespeare says ‘Juliet is the sun,’ he is
appealing to the fact that they are both warm and nurturing (not the fact that they both
reside in our solar system!). But, again, why should grasping an analogy of this kind
be so rewarding to us? Perhaps the use of a simple concrete example (or one that is
easily visualised, such as the sun) allows us to ignore irrelevant, potentially distract-
ing aspects of an idea or percept (e.g. Juliet has nails, teeth and legs) and enables us to
‘highlight’ the crucial aspects (radiance and warmth) that she shares with the sun but
not with other women. Whether this is purely a device for effective communication,
or a basic cognitive mechanism for encoding the world more economically, remains
to be seen. The latter hypothesis may well be correct. There are many paintings that
instantly evoke an emotional response long before the metaphor is made explicit by
an art critic. This suggests that the metaphor is effective even before one is conscious
of it, implying that it might be a basic principle for achieving economy of coding
rather than a rhetorical device. This is also true of poetic metaphors, as when Shake-
speare says of Juliet, ‘Death, that has sucked the honey of thy breath’: the phrase is
incredibly powerful well before one becomes consciously aware of the hidden anal-
ogy between the ‘sting of death’ and the bee’s sting and the subtle sexual connota-
tions of ‘sucking’ and ‘breath’.
Classifying objects into categories is obviously vital for survival, e.g. prey vs.
predator, edible vs. inedible, male vs. female, etc. Seeing a deep similarity—acom-
mon denominator as it were — between disparate entities is the basis of all concept
formation whether the concepts are perceptual (‘Juliet’) or more abstract (‘love’).
Philosophers often make a distinction between categories or ‘types’ and ‘tokens’ —
the exemplars of a type — (e.g. ‘ducks’ vs. ‘that duck’). Being able to transcend
tokens to create types is an essential step in setting up a new perceptual category.
Being able to see the hidden similarities between successive distinct episodes allows
you to link or bind these episodes to create a single super-ordinate category, e.g., sev-
eral viewer-centred representations of a chair are linked to form a viewer-
independent abstract representation of ‘chairness’. Consequently, the discovery of
similarities and the linking of superficially dissimilar events would lead to a limbic
activation — in order to ensure that the process is rewarding. It is this basic mechan-
ism that one taps into, whether with puns, poetry, or visual art.
Partial support for this view comes from the observation that these mechanisms can
go awry in certain neurological disorders. In Capgras syndrome, for instance, connec-
tions from the visual ‘face region’ in the inferotemporal cortex to the amygdala (a part
of the limbic system where activation leads to emotions) are severed so that a familiar
face no longer evokes a warm fuzzy emotional response (Hirstein and
Ramachandran, 1997). Remarkably, some Capgras patients are no longer able to link
successive views of a person’s face to create more general perceptual category of that
particular face. We suggested that in the absence of limbic activation — the ‘glow’ of
recognition — there is no incentive for the brain to link successive views of a face, so
that the patient treats a single person as several people. When we showed our Capgras
patient DS different photos of the same person, he claimed that the photos were of dif-
31 V.S. RAMACHANDRAN AND W. HIRSTEIN
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
55
55
ferent people, who merely resembled each other! One might predict, therefore, that
patients like DS would also experience difficulty in appreciating the metaphorical
nuances of art, but such a prediction is not easy to test.
An Experimental Test
We conclude by taking up the final test of any theory: does it lead to counterintuitive
predictions that can be tested experimentally? One approach — albeit a laborious one
— would be to do ‘psychophysics’ on artistic experience: show people different types
of pictures to see what they find pretty. The principles outlined above are difficult to
test individually, but we believe that the very first one — the peak shift principle —
can be tested directly. To do so one could measure the galvanic skin response (also
known as skin conductance response, SCR) of naive experimental subjects to photos
and drawings or caricatures. When you look at any evocative picture, the image is
extracted by the ‘early’ visual areas and sent to the inferotemporal cortex — an area
specialized for detecting faces and other objects. Once the object has been recog-
nized, its emotional significance is gauged by the amygdala at the pole of the temp-
oral lobe and if it is important the message is relayed to the autonomic nervous system
(via the hypothalamus) so that you prepare to fight, flee, or mate. This in turn causes
your skin to sweat, producing changes in its electrical resistance — a skin conduc-
tance response. So every time you look at your mother or even a famous face such as
Einstein’s or Gandhi’s, you will get an SCR, but not if you look at an unfamiliar face,
or a chair or a shoe (unless you happen to have a shoe fetish!).
So the size of the SCR is a direct measure of the amount of limbic (emotional) acti-
vation produced by an image. It is a better measure, as it turns out, than simply asking
someone how much emotion he feels about what he is looking at because the verbal
response is filtered, edited, and sometimes censored by the conscious mind — so that
your answer is a ‘contaminated’ signal. Indeed there are patients with damage to the
inferotemporal cortex who cannot consciously recognize their mother, yet will still
register a larger SCR to her face than to unfamiliar people (Bauer, 1984; Tranel and
Damasio, 1985; 1988). Conversely, we have shown that another type of patient has
the opposite problem: he consciously recognizes her, but gets no emotional/limbic
response to her and hence creates the delusion that she is some sort of impostor
(Hirstein and Ramachandran, 1997). These examples suggest that measuring SCR
somehow allows you to directly access those ‘unconscious’ mental processes. The
responses we get to art objects may similarly be only partly available to conscious
experience. You may deny you are attracted to a chap for all sorts of socio-cultural
reasons but your hidden attraction to him may manifest itself as a large SCR to his
photo (or sometimes, it may spill over when you dream during REM sleep)!
Our experiment, then, is quite simple. Compare a subject’s SCR to a caricature or
even just an outline drawing of, say, Einstein or Nixon to his SCR to a photo of Ein-
stein or Nixon. Intuitively, one would expect the photo to produce a large SCR
because it is rich in cues and therefore excites more modules. One might find, para-
doxically, that the drawing actually elicits a larger SCR, and if so, this would provide
evidence for our ideas on the peak shift effect — the artist has unconsciously pro-
duced a super stimulus. As a control, one would show photos which have been
morphed to look strange to ensure that it was not merely the strangeness of the carica-
THE SCIENCE OF ART 32
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
56
56
ture which was producing the larger SCR. Similarly, one could also compare the mag-
nitude of an SCR to caricatures of women (or indeed, to a Chola bronze nude or a
Picasso nude) with the SCR to a photo of a nude woman. It is conceivable that the sub-
ject might claim to find the photo more attractive at a conscious level, while register-
ing a large ‘unconscious aesthetic response’ — in the form of a larger SCR — to the
artistic representation. That art taps into the ‘subconscious’ is not a new idea, but our
SCR measurements may be the first attempt to test such a notion experimentally.
Another ‘experiment’ on art could take advantage of the fact that many cells in the
inferotemporal cortex of monkeys respond selectively to monkey (and human!) faces
— sometimes selectively just to a single face (Tovee et al., 1996). Again, one could
try confronting the cell with a drawing or caricature of the particular monkey (or
human) face it was responding to. Would the cell respond even more vigorously to a
‘superstimulus’ of this kind?
Conclusion
In summary, we have identified a small subset of principles underlying all the diverse
manifestations of human artistic experience. There are undoubtedly many others (cf.
the principle of visual repetition or ‘rhythm’), but these eight principles are a good
place to start. We shall call them ‘the eight laws of artistic experience,’ based on a
loose analogy with the Buddha’s ‘eight-fold path’ to wisdom and enlightenment.
One, the peak shift principle; not only along the form dimension, but also along more
abstract dimensions, such as feminine/masculine posture, colour (e.g. skin tones) etc.
Furthermore, just as the gull chick responds especially well to a super beak that
doesn’t resemble a real beak, there may be classes of stimuli that optimally excite
neurons that encode form primitives in the brain, even though it may not be immedi-
ately obvious to us what these primitives are. Two, isolating a single cue helps the
organism allocate attention to the output of a single module thereby allowing it to
more effectively ‘enjoy’ the peak shift along the dimensions represented in that mod-
ule. Three, perceptual grouping to delineate figure and ground may be enjoyable in its
own right, since it allows the organism to discover objects in noisy environments.
Principles such as figure–ground delineation, closure and grouping by similarity may
lead to a direct aesthetic response because the modules may send their output to the
limbic system even before the relevant objects has been completely identified. Four,
just as grouping or binding is directly reinforcing (even before the complete object is
recognized), the extraction of contrast is also reinforcing, since regions of contrast
are usually information-rich regions that deserve allocation of attention. Camouflage,
in nature, relies partly on this principle. Five, perceptual ‘problem solving’ is also
reinforcing. Hence a puzzle picture (or one in which meaning is implied rather than
explicit) may paradoxically be more alluring than one in which the message is obvi-
ous. There appears to be an element of ‘peekaboo’ in some types of art — thereby
ensuring that the visual system ‘struggles’ for a solution and does not give up too eas-
ily. For the same reason, a model whose hips and breasts are about to be revealed is
more provocative than one who is completely naked. (E.g., in Plate 6 the necklace just
barely covers the nipples and the dress is almost sliding off the hips.) Six, an abhor-
rence of unique vantage points. Seven, perhaps most enigmatic is the use of visual
‘puns’ or metaphors in art. Such visual metaphors are probably effective because dis-
33 V.S. RAMACHANDRAN AND W. HIRSTEIN
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
57
57
covering hidden similarities between superficially dissimilar entities is an essential
part of all visual pattern recognition and it would thus make sense that each time such
a link is made, a signal is sent to the limbic system. Eight, symmetry — whose rele-
vance to detecting prey, predator or healthy mates is obvious. (Indeed, evolutionary
biologists have recently argued that detecting violations of symmetry may help ani-
mals detect unhealthy animals that have parasites.)
One potential objection might be that originality is the essence of art and our laws
do not capture this. But the flaw in this objection becomes apparent when you con-
sider the analogy with language. After all the ‘deep structure’ of language discovered
by Chomsky has enormously enriched our understanding of language even if it
doesn’t explain Shakespeare, Valmiki, Omar Khayyam or Henry James. Likewise,
our eight laws may help provide a framework for understanding aspects of visual art,
aesthetics and design, even if they don’t necessarily explain the evocativeness or
originality of individual works of art.
In conclusion, we suggest that a great deal of what we call art is based on these
eight principles. We recognize, of course, that much of art is idiosyncratic, ineffable
and defies analysis but would argue that whatever component of art is lawful — how-
ever small — emerges either from exploiting these principles or from a playful and
deliberate violation of them. We cannot resist concluding with a joke: A young man
brings his fiancée home to introduce her to his father. His father is astonished to note
that she has a clubfoot, a squint, a cleft palate and is hunchbacked, and can hardly
conceal his dismay. Noticing his father’s reaction, his son calmly tells him, ‘Well
Dad, what can I say? You either like a Picasso or you don’t.’
Acknowledgements
We thank Diane Rogers-Ramachandran, Francis Crick, Odile Crick, Julia Kindy,
Mumtaz Jahan and Niki de Saint Phalle for stimulating discussions on numerous
topics straddling the boundary between art and science. VSR also thanks All Souls
College, Oxford, for a fellowship that allowed him to complete this project.
References
Allman, J.M. and Kaas, J.H. (1971), ‘Representation of the visual field in striate and adjoining
cortex of the owl monkey’, Brain Research, 35, pp. 89–106.
Arnheim, R. (1956), Art and Visual Perception (Berkeley, CA: University of California Press).
Attneave, F. (1954), ‘Some informational aspects of visual perception’, Psychological Review,61,
pp. 183–93.
Barlow, H.B. (1986), ‘Why have multiple cortical areas?’, Vision Research,26 (1), pp. 81–90.
Bauer, R.M. (1984), ‘Autonomic recognition of names and faces in prosopagnosia: a
neuropsychological application of the Guilty Knowledge Test’, Neuropsychologia,22,
pp. 457–69.
Churchland, P.S., Ramachandran, V.S. and Sejnowski, T.J. (1994), ‘A critique of pure vision’,in
Large-scale Neuronal Theories of the Brain, ed. C. Koch and J.L. Davis (Cambridge, MA: The
MIT Press).
Crick, F. and Koch, C. (1998), ‘Consciousness and neuroscience’, Cerebral Cortex,8(2),
pp. 97–107.
di Pellegrino, G, Fadiga, L., Fogassi, L.; Gallese, V. and Rizzolatti, G. (1992), ‘Understanding
motor events: a neurophysiological study’, Experimental Brain Research, 91 (1), pp. 176–80.
Gombrich, E.H. (1973), ‘Illusion and art’, in Illusion in Nature and Art, ed. R.L. Gregory and E.H.
Gombrich (New York: Charles Scribner’s Sons).
THE SCIENCE OF ART 34
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
58
58
Hirstein, W.S. and Ramachandran, V.S. (1997), ‘Capgras Syndrome: A novel probe for
understanding the neural representation of the identity and familiarity of persons’, Proceedings
of the Royal Society of London,264, pp. 437–44.
Hubel, D.H. and Wiesel, T.N. (1979), ‘Brain mechanisms of vision’, Scientific American, 241,
pp. 150–62
Johansson, G. (1975), ‘Visual motion perception’, Scientific American, 232, pp. 76–8.
Julesz, B. (1971), Foundations of Cyclopean Perception (Chicago, IL: University of Chicago
Press).
Livingstone, M.S. and Hubel, D.H. (1987), ‘Psychophysiological evidence for separate channels
for the perception of form, color, movement and depth’, Journal of Neuroscience,7, pp. 3416–68.
Marr, D. (1981), Vision (San Francisco, CA: Freeman and Sons).
Penrose, Roland. (1973), ‘In praise of illusion’, in Illusion in Nature and Art, ed. R.L. Gregory and
E.H. Gombrich (New York: Charles Scribner’s Sons).
Pinker, S. (1998), How the Mind Works (New York: William Morrow).
Ramachandran, V.S. (1990), ‘Visual perception in people and machines’, in AI and the Eye, ed. A.
Blake and T. Troscianko (Chichester: Wiley).
Ramachandran, V.S. and Hirstein, W. (1997), ‘Three laws of qualia: Clues from neurology about the
biological functions of consciousness and qualia’, Journal of Consciousness Studies,4(5–6), pp.
429–57.
Ramachandran, V.S. and Blakeslee, S. (1998), Phantoms in the Brain (New York: William Morrow
and Co).
Ramachandran, V.S., Armell, C., Foster, C. and Stoddard, R. (1998), ‘Object recognition can drive
apparent motion perception’, Nature,395, pp. 852–3.
Shepard, R. (1981), In Perceptual Organization, ed. M. Kubovy and T. Pomerantz (New Jersey:
Lawrence Erlbaum).
Singer, W. and Gray, C.M. (1995), ‘Visual feature integration and the temporal correlation
hypothesis’, Annual Review of Neuroscience,18, pp. 555–86.
Snyder, A. and Thomas, M. (1997), ‘Autistic savants give clues to cognition’, Perception 26,
pp. 93–6.
Tinbergen, N. (1954), Curious Naturalists (New York: Basic Books).
Tovee, M.J., Rolls, E. and Ramachandran V.S. (1996), ‘Rapid visual learning in neurons in the
primate visual cortex’, Neuroreport,7, pp. 2757–60.
Tranel, D. & Damasio, A.R. (1985), ‘Knowledge without awareness: An autonomic index of facial
recognition by prosopagnosics’, Science, 228, pp. 1453–4.
Tranel, D. & Damasio, A.R. (1988), ‘Non-conscious face recognition in patients with face agnosia’,
Behavioral Brain Research, 30, pp. 235–49.
Van Essen, D.C. and Maunsell, J.H. (1980), ‘Two-dimensional maps of the cerebral cortex’, J.
Comp. Neurol.,191, pp. 255–81.
Zeki, S. (1980), ‘The representation of colours in the cerebral cortex’, Nature,284, pp. 412–18.
Zeki, S. (1998), ‘Art and the brain’, Proceedings of the American Academy of Arts and Sciences,
127 (2), pp. 71–104. Reprinted in Journal of Consciousness Studies,6(6–7), pp. 76–96.
35 V.S. RAMACHANDRAN AND W. HIRSTEIN
Copyright (c) Imprint Academic 2005
For personal use only -- not for reproduction
59
59
The eight laws of
artistic experience Cracking the code of
art’s allure
Anthony Freeman, managing editor, Journal
of Consciousness Studies
Figure 1. The
accentuated features
of a bronze statue of
the Indian goddess
Parvati (circa 11th
century A.D.),
amplify the essence
of the feminine to
produce a “peak
shift effect”.
Figure 2. Grouping:
A bold new theory to identify the
common denominator of all
visual art
‘If a Martian ethologist were to land on
earth and watch us humans, he would
be puzzled by many aspects of human
nature, but surely art—our propensity to
create and enjoy paintings and
sculpture—would be among the most
puzzling. What biological function could
this mysterious behaviour possibly
serve?
“Cultural factors undoubtedly influence
what kind of art a person enjoys. But,
even if beauty is largely in the eye of the
beholder, might there be some sort of
universal rule or ‘deep structure’,
underlying all artistic experience?”
Vilayanur S. Ramachandran, Director of
the Center for Brain and Cognition at the
University of California at San Diego,
has made a bold and controversial
attempt to answer these intriguing
questions by proposing a new scientific
theory of art. The theory explains many
familiar experiences, such as why a
cartoon squiggle can evoke a well-
known face more quickly than a full
Pa
g
e 1 of 10Crackin
g
the code of art's allure.
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\Crackin
g
the code of art's allure.htm
60
60
the Dalmatian
emerges from a
random jumble of
splotches, producing
a pleasing sensation.
Figure 3. A
caricature of Richard
Nixon by the French
cartoonist Mulatier.
Figure 4. The savant
syndrome: horses
drawn by Leonardo
da Vinci (right) and
by an eight-year old
autistic child.
colour photograph, and why many men
find the hour-glass figure of Marilyn
Monroe sexy.
Professor Ramachandran’s theory
addresses three questions: (a) What are
the “rules of art” that make something
pretty? (b) Why did these rules evolve
and have the form that they do? (c)
What is the brain circuitry involved?
Previous theories of art have looked at
one or two of these questions, but this is
the first time all three have been tackled
together.
With his colleague William Hirstein,
Ramachandran proposes a list of “Eight
laws of artistic experience . . . that artists
either consciously or unconsciously
deploy to optimally titillate the visual
areas of the brain,” in particular that part
of the brain known as the limbic system.
Of the eight (see box), three seem to be
especially significant: a psychological
phenomenon called the “peak shift
effect”; the principle of “grouping”; and
the benefit of focusing on a single visual
cue.1
The ‘peak shift effect’
The “peak shift effect” is a well-known
principle in animal discrimination
learning. For example, if a rat is taught
to discriminate a square from a rectangle
and rewarded for the rectangle, it will
soon learn to respond more frequently to
the rectangle. Moreover, if the rat is
trained with a prototype rectangle of,
say, aspect ratio 3:2, it will respond even
Pa
g
e 2 of 10Crackin
g
the code of art's allure.
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\Crackin
g
the code of art's allure.htm
61
61
The eight
laws
of artistic
experience
1
The ‘peak shift
principle’ makes
exaggerated
elements
attractive
2
Isolating a single
cue helps to
focus attention
3
Perceptual
grouping makes
objects stand out
from
background
4
Contrast is
reinforcing
5
Perceptual
‘problem
more positively to a longer and skinnier
figure—say, of aspect ratio 4:1. This
curious result implies that what the rat is
learning to value is not a particular
rectangle but a rule: rectangles are
better than squares. So the greater the
ratio between the long and the short
sides, i.e. the less square-like it is, the
“better” the rectangle is in the rat’s eyes.
This is the “peak shift effect”.
Ramachandran argues that this principle
holds the key for understanding the
evocativeness of much of visual art.
How does the peak shift effect relate to
human pattern recognition and aesthetic
preference? Consider the way in which a
skilled cartoonist produces a caricature
of a famous face, say the late U.S.
President Richard Nixon’s. What a
cartoonist does (unconsciously) is to
take the average of all faces, subtract it
from Nixon’s face (to get the difference
between Nixon’s face and all others) and
then amplify the differences to produce a
caricature. The final result, of course, is
a drawing that is even more Nixon-like
than the original. The artist has amplified
the differences that characterize Nixon’s
face in the same way that an even
skinnier rectangle is an amplified version
of the original prototype that the rat is
exposed to. Hence Ramachandran’s
aphorism that “All art is caricature”. (This
is not literally true, as he admits, but it
applies surprisingly often.) In other
words, what the artist tries to do is not
only capture the essence of something
Pa
g
e 3 of 10Crackin
g
the code of art's allure.
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\Crackin
g
the code of art's allure.htm
62
62
solving’ is also
reinforcing
6
Unique vantage
points are
suspect
7
Visual ‘puns’ or
metaphors
enhance art
8
Symmetry is
attractive
but amplify it in order to activate neural
mechanisms more powerfully than the
original object.
Look at the Chola bronze—the
accentuated hips and bust of the
Goddess Parvati (Fig. 1) and you will
see at once that this is essentially a
caricature of the female form. Here the
artist has chosen to amplify the “very
essence” (called the rasa by Hindu
artists) of being feminine, by moving the
image abnormally far toward the
feminine end of the female/male
spectrum. The artistic amplification
produces a “super stimulus” to which,
Ramachandran conjectures, certain
brain circuits respond. Artists may also
try to evoke a strong direct emotional
response by exploiting the peak shift
effect along dimensions other than form.
For instance, a Boucher, a Van Gogh, or
a Monet may be thought of as a
caricature in “colour space”.
Perceptual grouping and binding
A second basic principle suggested by
Ramachandran is “grouping” (or
binding). The way this works can be
illustrated by the Dalmatian dog picture
shown in Fig. 2. This is seen initially as a
random jumble of splotches. The
number of potential groupings of these
splotches is infinite, but once the dog is
seen, your visual system links only a
subset of these splotches together and it
is impossible not to “hold on” to this
group of linked splotches. Indeed, the
Pa
g
e 4 of 10Crackin
g
the code of art's allure.
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\Crackin
g
the code of art's allure.htm
63
63
discovery of the dog and the linking of
the dog-relevant splotches generates a
pleasant “aha” sensation. In “colour
space” the equivalent of this would be
wearing a blue scarf with red flowers if
you are wearing a red skirt; the
perceptual grouping of the red flowers
and your red skirt is aesthetically
pleasing. Artists understand the pleasure
given by such effects.
The evolutionary value of such grouping
of stimuli to pick out objects is obvious: it
makes the detection of both prey and
predators much easier. But how is such
grouping achieved? The key idea is as
follows. Given the brain’s limited
attentional resources and shortage of
neural space for competing
representations, every stage in the
processing of visual information offers
an opportunity to generate a signal that
says, “Look, here is a clue to something
potentially object-like!” Partial solutions
or conjectures to perceptual problems
are fed back from every level in the
hierarchy to every earlier module to
impose a small bias in processing,
allowing the final percept to emerge from
such progressive “bootstrapping”.
Consistency between partial high-level
“hypotheses” and earlier low-level
ensembles generates a pleasant
sensation—e.g. the Dalmatian dog
“hypothesis” encourages the binding of
corresponding splotches which, in turn,
further consolidate the “dog-like” nature
of the final percept and we feel good
Pa
g
e 5 of 10Crackin
g
the code of art's allure.
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\Crackin
g
the code of art's allure.htm
64
64
when it all finally clicks in place. And
what the artist tries to do is to tease the
system with as many of these “potential
object” clues as possible—an idea that
would help explain why grouping and
“perceptual problem-solving” are both
frequently exploited by artists and
fashion designers.
Isolating a visual signal
The third principle (in addition to peak
shift and binding) emphasized by
Ramachandran is the need to isolate a
single visual modality before you amplify
the signal in that modality. The brain’s
ability to do this explains why an outline
drawing or sketch is more effective as
“art” than a full colour photograph.
Consider a full-colour illustration of
Nixon, with depth, shading, skin tones
and blemishes, etc. What is unique
about Nixon is the form of his face (as
amplified by the caricature), but the skin
tone—even though it makes the picture
more human-like—
doesn’t contribute to making him “Nixon-
like” and therefore actually detracts from
the efficacy of the form cues. This
explains why one not only “gets away”
with just using outlines—they are
actually more effective than a full-colour
half-tone photo, despite its having more
information. Hence the aphorism “more
is less” in Art.
Additional evidence for this view comes
from the “savant syndrome”—autistic
children who are “retarded” and yet
Pa
g
e 6 of 10Crackin
g
the code of art's allure.
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\Crackin
g
the code of art's allure.htm
65
65
produce beautiful drawings. The animal
drawings of the eight-year-old artist
Nadia, for instance, are almost as
aesthetically pleasing as those of
Leonardo da Vinci (Fig. 4).
Ramachandran argues that this is
because the fundamental disorder in
autism is a distortion of the “salience
landscape”; savants shut out many
important sensory channels, thereby
allowing them to deploy all their
attentional resources on a single
channel.
‘Lie detector’ testing
Ramachandran believes that the “peak
shift principle” can be tested directly.
The method would employ skin
conductance response (SCR), the
technology used in “lie detectors”. The
size of the SCR is a direct measure of
the amount of limbic (emotional)
activation produced by an image. It is a
better measure, as it turns out, than
simply asking someone how much
emotion they feel about what they are
looking at, because the verbal response
is filtered, edited, and sometimes
censored by the conscious mind.
Measuring SCR allows direct access to
“unconscious” mental processes.
The experiment would compare a
subject’s SCR to a caricature of, say,
Einstein or Nixon, to his SCR to a photo
of the same individuals. Intuitively, one
would expect the photo to produce a
large SCR because it is rich in cues and
Pa
g
e 7 of 10Crackin
g
the code of art's allure.
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\Crackin
g
the code of art's allure.htm
66
66
therefore excites more modules. If one
found, paradoxically, that the caricature
actually elicited a larger SCR, this would
provide evidence for the operation of the
peak shift effect—the artist would have
unconsciously produced a super
stimulus.
Critical responses
Ramachandran says that one could also
compare the magnitude of an SCR to
caricatures of women (or indeed, to a
Chola bronze nude or a Picasso nude)
with the SCR to a photo of a nude
woman. It is conceivable that the subject
might claim to find the photo more
attractive at a conscious level, while
registering a large “unconscious
aesthetic response”—in the form of a
larger SCR—to the artistic
representation. That art taps into the
“subconscious” is not a new idea, but
our SCR measurements may be the first
attempt to test such a notion
experimentally.
Not surprisingly, Ramachandran’s
attempt to reduce aesthetic experience
to a set of physical or neurobiological
laws has already met with stout criticism.
A symptom of trouble to come has been
seen in his use of the term “pretty”. If
used at all by serious art critics, the word
damns with faint praise. Prettiness is not
seen as a synonym for beauty, but as a
shallow impostor. Yet Ramachandran
uses it without irony as a positive
attribute. This one piece of unfortunate
Pa
g
e 8 of 10Crackin
g
the code of art's allure.
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\Crackin
g
the code of art's allure.htm
67
67
terminology might be forgiven, but other
elements in his work are all of a piece
with it.
First, there is the heavy reliance on the
female form and upon the erotic in his
examples. Then he seems to equate
“arousal” (as measured by SCR) with a
positive aesthetic response—an
assumption felt by critics to presuppose
the reductionist case he is trying to
prove. Taken together, these points
seem to some critics to confuse
pornography with high art.
The “science of art” has also been
attacked from the scientific side, on the
grounds that its proponents have not yet
conducted any serious empirical tests of
their ideas. At best they have offered a
manifesto for a research programme and
made some suggestions for possible
lines of investigation. Even then, it has
been pointed out that the narrow range
of examples used hardly justifies the
lofty claims to be dealing with the whole
of art, let alone to have uncovered the
“laws of aesthetic experience”. Nor has
hitching his bandwagon to the Buddhist
train, by associating his “eightfold laws”
with the Buddha’s eightfold noble path
won Ramachandran any friends, though
to be fair, he admits himself that this is a
slightly whimsical association.
Criticism has centred on the lack of
proportion between the narrow approach
to art taken by Ramachandran and the
grandiloquent claims he makes for his
theory—although, as he admits, he
Pa
g
e 9 of 10Crackin
g
the code of art's allure.
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\Crackin
g
the code of art's allure.htm
68
68
initially proposed it “in a playful spirit”.
But the very audacity with which its
author propounds it guarantees that we
shall be hearing a lot more of it in the
coming months.
1. V.S. Ramachandran and William Hirstein
are publishing a fuller exposition of this
subject later this year in the Journal of
Consciousness Studies under the title “The
Science of Art, A Neurological Theory of
Aesthetic Experience”.
The UNESCO Courier
Pa
g
e 10 of 10Crackin
g
the code of art's allure.
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\Crackin
g
the code of art's allure.htm
69
69
Joseph A. Goguen
Art and the Brain
Editorial Introduction
This essay has three main goals: to introduce and somewhat contextualize the
p
apers that follow; to bring out some implicit (and generally unintended)
dialogues among them; and, in order to encourage further discussion in future
issues of the Journal of Consciousness Studies, to introduce some new
p
erspectives, including the historical development of Western art, cognitive
linguistics, and the sacred. The remarks pertaining to the third goal are
necessarily personal, and should not be confused with Journal of
Consciousness Studies editorial policy, except insofar as their goal is to
p
romote debate across the gap that separates the sciences and the humanities
—
the ‘two cultures’. This gap was perhaps first pointed out in the late 1950s
by C.P. Snow, who was noted as both a novelist and scientist (Snow, 1959).
The contributions in this volume divide into four main groups: the focus paper
by V.S. Ramachandran and William Hirstein with its associated
commentaries; the paper by Semir Zeki, from which the volume draws its
title; the paper by Nicholas Humphrey with its commentaries; and the papers
b
y Erich Harth, Ralph Ellis and Jason Brown. The approach of Ramachandran
and Hirstein and their commentators generally has a psychological flavour,
while Zeki takes a more neurobiological approach, Humphrey combines
anthropological and biological ideas, Harth takes a cognitive-evolutionary
view, Ellis calls upon Gibsonian affordances and Brown combines process
theory with clinical pathology.
Perhaps the largest question confronted in this volume is, what does it mean to
be human? Like most big questions, this one is addressed in very different
ways by the disciplines of psychology, neurophysiology, anthropology,
evolutionary biology and philosophy, to say nothing of art. But by exploring
some collisions among these multiple perspectives, perhaps we can learn more
than through any single perspective, and we might even begin to see some
ways to bridge C.P. Snow’s infamous gap.
Pa
g
e 1 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
70
70
I: The Focus Paper and Its Commentaries
The focus paper by Ramachandran and Hirstein is firmly on the scientific side
of the gap, proposing a bold experimental exploration of the aesthetic
response. Their approach is decidedly and unapologetically materialist and
reductionist. In his commentary on this paper, the eminent psychologist
Richard Gregory says that he expects this paper to ‘stimulate a lively debate’
and, indeed, the commentaries are delightfully energetic and diverse; Gregory
also suggests considering this debate a test of whether ‘there has been any
p
rogress towards a rapprochement between [the] opposing camps’ of the
sciences and the humanities ‘in the half century since the time of Snow’s
original complaint’. Readers should form their own view on this, but I would
encourage them to consider the entire volume as relevant evidence.
While many readers from the humanities side of the gap will doubtless share
the disquiet of several commentators on the focus paper, reductionism
certainly has its advantages, even in the humanities; for example, as brilliantly
argued by E.O. Wilson in a recent popular book (Wilson, 1999), what we
learn from science has a resilient, stable and cumulative quality that seems
notably absent in the humanities. Of course, results from science are always
open to a variety of philosophical and cultural interpretations, and
reductionism itself is a philosophical position. Therefore scientific work in
humanistic areas can never eliminate the humanities. Moreover, because the
arts play an entirely different kind of role in society, they are in an even more
resistant position. But this only means that both sides have strengths from
which to make their contributions to the great debate, not that the humanities
are necessarily in the stronger position.
Testing the outer limits of logical positivism in the late 1920s and early 1930s,
the then well-known mathematician G.D. Birkhoff undertook a programme to
reduce aesthetics to mathematics, by defining the aesthetic measure of an
object to be the ratio of its symmetry to its complexity (Birkhoff, 1928; 1933).
While this possibly could be useful for evaluating simple figures, such as
company logos, it has generally been regarded as a failure, because it seems
so difficult to apply to more complex objects. Still, it can be said that this
brave if misguided attempt succeeded in advancing our understanding of the
limitations of approaches of that kind.
It seems to me that this is the very least we should expect from reductionist
Pa
g
e 2 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
71
71
forays into the humanities, and that in fact Ramachandran and Hirstein, along
with Zeki, Humphrey and other authors in this volume, have made
contributions that go well beyond this minimum, and will withstand the test of
time.
V.S. Ramachandran is a distinguished cognitive neuropsychologist, winning
recognition for his work on the so-called ‘phantom limb’ phenomenon, and
for a recent book arguing against the metaphysical self, in favour of
consciousness as a brain-based biological phenomenon (Ramachandran &
Blakeslee, 1998). His contribution to this volume with Hirstein argues that
‘underlying all the diverse manifestations of human artistic experience’ are
eight ‘laws of artistic experience’, among which the peak shift effect is
p
rominent, both in the focus paper and its commentaries. A special strength of
this paper is the experimental programme that it proposes for validating its
eight laws. Because of the experimental methodology, involving physiological
measurements such as galvanic skin response, it could be argued that the
implicit view of what it means to be human taken here is constrained by this
narrow ‘channel’ through which flows a rather small amount of information,
having a highly specific character.
Three of the commentaries are by authors from the humanities side of the gap.
The distinguished art historian, Partha Mitter doubts the validity of the
‘eliding of cognition and pleasure’ by Ramachandran and Hirstein, and also
questions whether they have taken sufficient account of cultural factors, or
used a sufficiently broad sample of art on which to base their conclusions. The
curator Julia Kindy, though enthusiastic about the paper, questions the
conclusion that ‘all art is caricature’
b
y giving some fascinating examples, and
also questions the notion of beauty used by Ramachandran and Hirstein. The
artist Ruth Wallen attacks the focus paper on several grounds, including its
lack of attention both to culture, and to academic debates about the nature of
art; she also objects to the parallel with Buddhism, in support of which she
develops some Buddhist concepts. Jaron Lanier is well regarded as both a
musician and computer scientist. In harmony with several other
commentators, he suggests that the focus paper might better be entitled ‘The
Science of Design’, arguing that crucial aspects of art are overlooked. He also
gives a pertinent discussion of Indian aesthetic theory, comparing it to higher
mathematics, and once again points to the importance of cultural factors in art,
as well as to the basis of culture in learning. Of course, Ramachandran mounts
Pa
g
e 3 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
72
72
a vigorous defence against all this in his final reply, and readers will no doubt
find it more enjoyable if I do not attempt to summarize it here.
There are four commentaries from the science side of the gap. Colin
Martindale provides interesting technical details on the peak shift effect,
discusses its application to evolution (via sexual selection), and cites some of
his own research on the psychology of art, showing that untrained viewers do
not like cubist art and do not prefer veiled nudes. Richard Gregory notes that
there is ‘more and more evidence of feedback loops — and an enormous
richness of downgoing fibres’ and considers that ‘the role of knowledge —
both knowledge of the world and experience of art — is greatly
underestimated in this paper’. He also questions whether perceptual grouping
and binding are directly reinforcing, suggesting an evolutionary explanation
instead. Bruce Mangan argues that, for the study of consciousness, the
p
henomenology of art is fundamental, particularly its ability to enhance and
intensify experience; he also argues against the ‘all art is caricature’
hypothesis of Ramachandran and Hirstein, and against their single dimension
hypothesis; in addition, he suggests that the ineffability of art may arise from
the fact that most of the work is unconscious parallel distributed processing,
and he cites work of Berlyne from 1971 applying the peak shift effect to art.
Finally, Bernard Baars joins in calling attention to the role of information in
understanding the aesthetic response, noting in particular that emotion can
reduce redendancy effects, in which repeated stimuli fade from consciousness;
he also claims that the aesthetic ‘is not just a luxury, but a compelling
biological adaptation’. Once again, I encourage reading the commentaries
themselves and the responses of Ramachandran before settling on any
evaluation of all these positions.
II: Other Papers
Semir Zeki is an eminent neurophysiologist, who takes an orthodox
reductionist approach to art when he says that he wants a ‘theory of
aesthetics . . . based on an understanding of the workings of the brain’.
However, he employs the unusual rhetorical device of claiming that ‘artists
are neurologists, studying the brain with techniques that are unique to them
and reaching interesting but unspecified conclusions about the organization of
the brain’. Although he gives several arguments, those that I find most
compelling have the form:
Pa
g
e 4 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
73
73
Artists who said they are especially interested in X have found
ingenious ways to partially isolate X from Y, having a clear basis in
known neuroanatomy.
In the case of kinetic art, X is movement and Y is colour, while in the case of
fauvism, X is colour and Y is form, noting that in his pioneering prior work,
Zeki had already shown that movement, colour and form are processed in
different ways, that involve different areas of the brain (see pp. 78–9 below
for references and more detail). Moreover, arguments of the above type would
be reinforced if the claim of Ramachandran and Hirstein that art is more
effective when restricted to one modality (p. 24 below) were to be established.
An important fact about vision is the massive feedback from ‘higher’ to
‘lower’ centres, including the retina itself; this suggests that vision, far from
being a passive reception of ‘what’s out there’, is an active search for ‘what’s
important’ to the organism, based on expectations and prior experience. In
p
articular, work by Zeki, as well as by David Hubel at Harvard, and others,
has shown that there is significant feedback among the areas of the brain
associated with visual processing. Zeki’s approach to art is a natural
application of his pioneering discoveries of the spatial modularity of visual
p
erception, and of the temporal asynchrony among various perceptual
modalities and submodalities. Perhaps the most basic insight that an outsider
can gain from this research is that a great deal of parallel distributed
p
rocessing is needed in order to create perceptual constancy from the chaos of
sensory inputs, and that most of this processing is unconscious. It seems
highly likely that we will see further significant insights into the neural bases
of various visual phenomena, as the result of future experiments using the
rapidly advancing technologies of brain imaging.
Despite the variety of disciplines entering into Humphrey’s paper and its
commentaries, it seems especially important to recognize the biological issues
that are involved, among which perhaps the most central is the origin and
nature of modern human beings. There is widespread agreement that
anatomically modern humans first appeared roughly 100,000 years ago, but
there is still debate about when cognitively modern humans first appeared. It
has been widely believed that Cro-Magnon humans had a capacity of
symbolization and communication that set them apart from the other hominids
that they were gradually displacing a few tens of thousands of years ago, and
Pa
g
e 5 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
74
74
the stunning cave art dating from about 10,000 to 35,000 years ago is usually
taken as strong evidence for this belief. But Humphrey argues against this,
giving evidence that it may have been the Neanderthals who produced this art.
It is interesting to notice that much of the debate around Humphrey’s paper is
in fact aesthetic, rather than overtly biological. For example, in the response to
his commentaries, Humphrey argues that Spanish-Levantine rock art, Greek
vase painting, and Roman murals have an inferior ‘copy-book’ quality, in
comparison with the cave art of Chauvet and Lascaux; I wonder though if he
would be willing to say the same about Greek sculpture? In any case, this is a
fascinating example of the humanities making a substantive contribution to a
significant scientific debate.
There is more at stake here than might at first be thought, including the
uniqueness of humans, the origin of language, and perhaps even Darwinian
evolution. Let’s take these one at a time. Although most discussion about the
uniqueness of ‘man’ has a more rhetorical than scientific flavour, there are
some substantive points, including how the cognitive capabilities of
N
eanderthals, Cro-Magnons and modern humans (homo sapiens) might differ;
Humphrey’s paper and the debate around it are directly relevant to this point,
with Humphrey arguing against one kind of uniqueness, and most of his
commentators disagreeing with him. In a recent provocative contribution, the
geographer Jerome Dobson, of the Oak Ridge National Laboratory, has
suggested that Neanderthals may actually have been a kind of modern human
suffering from chronic iodine deficiency (Dobson, 1998), a view that has been
sharply challenged by many anthropologists. So it seems that the debate about
the origin and nature of modern man is intensifying, as it continues to twist
through a widening range of disciplines.
The origin of language is clearly related to cognitive capability, and among
the major issues that have been debated here are Noam Chomsky’s dual
claims that (1) language has a biological basis, the constraints of which lead to
certain universals in human languages; and (2) nevertheless, this biological
basis did not evolve to support language, but rather language makes use of
cognitive abilities that were already there, perhaps for no purpose at all, or just
for their beauty. Chomsky has gone so far as to say that ‘this poses a problem
for the biologist, since if true, it is an example of true “emergence” — the
appearance of a qualitatively different phenomenon at a specific stage of
complexity of organization’ (Chomsky, 1972). By ‘a problem for the
Pa
g
e 6 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
75
75
biologist’, Chomsky presumably means a challenge to the neo-Darwinian
theory of evolution, or at least, to some strict interpretations of that theory.
The relevance to this volume is that language requires cognitive capabilities
similar to those involved in art and, moreover, the biological origins of art and
language are equally mysterious.
The paper by Erich Harth in this volume goes more deeply into several
aspects of this area. Harth applies ideas from cognitive science (in particular,
his own ‘sketch-pad’ approach), evolutionary biology, neurophysiology and
linguistics, to the origins of art. In fact, the argument largely proceeds by
discussing the origins of language, and then considering art as a correlative
development. This paper includes a nice discussion of the binding and related
p
roblems, with a proposed solution. In addition, there is an instructive joke
featuring Pablo Picasso.
Ellis also takes a cognitive approach, suggesting that art plays with our
expectations, which are based in efferent brain activity, by offering a variety
of Gibsonian emotional ‘affordances’. Meaning is then created in the context
of a total ongoing dynamic life process; hence, for Ellis, meaning is far from
being the causal result of simple stimuli. This view of art is an application of
his general approach to cognition as an expectancy-led interactive and
embodied process. Ellis also takes a strong stand for the fundamental
importance of emotion in art, and he makes explicit suggestions about its
neurophysiological basis.
Brown explores conceptual, intentional, and emotional dimensions of art with
some sophistication, drawing on ideas from process theory (in the tradition of
Whitehead), clinical neuropathology and phenomenology. Like Ellis, Brown
cites the interdependence of emotion and perception, a point also made in
several of the commentaries on Ramachandran and Hirstein. But Brown puts
greater emphasis on the more general role of knowledge in guiding
p
erception. There seems to be almost a consensus among our contributors on
the importance of emotion and knowledge in perception, and especially in art;
hence this may well be a fruitful area for future scientific exploration. Readers
may also enjoy comparing Brown’s clinically-oriented discussion of brain
damage (p. 146 below) and art with the various views in the paper of
Humphrey and its commentaries; Brown’s comparisons of music and visual
art are also interesting.
Pa
g
e 7 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
76
76
III: Diverse Perspectives
We now consider some themes not represented elsewhere in this volume, but
that may nevertheless be relevant. These include the curious parallel between
some phases in the historical development of Western art and in our scientific
and philosophical understandings of perception, a preliminary view of what
might make some works of art great, and the perhaps surprising anticipation
of some recent neuroscientific findings by ancient Indian thought. There is
also a brief discussion of cognitive linguistics.
Favouring long-term trends, and risking serious over-simplification, we might
try to see the recent history of Western art as a gradual exploration of more
and more contextual aspects of seeing. Medieval art, consisting primarily of
scenes from the stories of Christ, the saints, etc., presents a pure, eternal, self-
existing world (much as in traditional Indian religious art); it is non-
p
erspectival in form, and symbolic in content; it does not directly address the
realm of human experience — for example, the size of objects tends to
correlate with their importance rather than with their location relative to a
viewer. Renaissance art introduces perspective, and hence an individual
observer, with a much wider range of subject matter, and (passing through the
baroque) develops into the classical period, with its interest in realism and its
fascination with form, especially symmetry, as perhaps best exemplified in the
architecture and music of this period.
Art in the romantic period makes emotion a more explicit focus, while the
impress- ionists and cubists can perhaps be described as interested in the
p
rocesses of perception, the former more concerned with colour and design,
and the latter more concened with form. Even critical social realism can be
found, for example in the ashcan school. The expressionists, especially
abstract expressionists, explored the emotional content of pure form, while
surrealists (like Magritte) and conceptual and pop artists explored the effect of
our conceptions and preconceptions on art. More recently still, many
p
erformance artists have reached out for more direct involvement with their
audiences.
Western theories of art have tended to follow a similar development. Though
fixing and comparing their ordering is more difficult, there does seem to be a
tendency to take more and more account of context. For example, the realist
vision of the early Renaissance tried to confine context to the location of an
Pa
g
e 8 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
77
77
observer, following the geometrical theory of perspective. This realism later
gave way to broader views, e.g., in mannerism. The Renaissance use of
p
erspective also illustrates how scientific theories of art have often been the
most prestigious, perhaps because of their precision; another example is the
influence of advances in anatomy on Renaissance art. Contemporary aesthetic
theories tend to challenge earlier absolutist theories, and emphasize concepts
like ‘intertextuality’, which asserts the interdependence of art objects with
other objects (Derrida, 1976). Recent progress in neuroscience has inspired a
significant increase of interest in theories of aesthetics, with often exciting
results, as I hope readers of this volume will be able to affirm, but it is not yet
clear that science is able keep pace with the expanding contextualization
evinced by artists and by humanist theories of art.
We can also compare the progression of artistic styles with the various visual
centres discussed in the paper by Zeki. The visual processing areas and
p
athways among them are complex, and it seems fair to say that what is
known today is only a very small part of what could be known. Still, much of
what is known is relevant to the picture that we are trying to paint in this
essay. First, the two retinas and two lateral geniculate bodies (LGBs) in the
thalamus seem mainly to be doing low-level processing, using neural
structures whose connectivity reflects the contiguity of incoming signals.
Subsequent processing is done in the visual cortex, where what we think of as
reality begins to emerge from some two dozen areas, of which V1 through V5
are the best understood, due to work of Zeki, Hubel, and others; these centres
extract lines, colour, brightness and motion in complex but fascinating ways:
V1 and V2 seem to prepare information for V3 to V5, with V3 especially
sensitive to lines, V4 to colour (including relative brightness, which is
computed by comparing an area with its surround), and V5 to motion; there is
also a good deal of feedback to V1 and V2. ‘Symbolic’ recognition, e.g., that
a certain pattern of lines and circles is a go board, comes further along in the
neural circuitry, and is little understood; but it is natural to suppose that this is
where the organism’s experience, including culture, is brought to bear on
p
erception.
To summarize, there is a broadening of context as we move from retina and
LGB to V1, V2, then V3, V4, V5, and on to the still higher areas, and it
should once again be emphasized that more highly contextual information is
constantly being fed back to ‘lower’ areas, to help improve their performance.
Pa
g
e 9 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
78
78
So we seem to have a suggestive parallel between the neural architecture of
visual processing and (aspects of) stylistic development in Western visual art.
The similarity between some cave art and some autistic art noticed by
Humphrey fits into the physiological and historical themes being developed
here, in that the ‘jittery’ outlines seen in both kinds of art are in some ways
closer to what our eyes actually take in, due to the rapidity of saccadic eye
movements, of which we are generally unconscious. But what this tells us
about (so called) primitive man and (so called) primitive art is not necessarily
very clear, since it can be argued that the art of pre-colonial Africa, Australia
and India is in some ways more sophisticated than much art in the formal
Western sense, e.g., because it takes more account of cultural context (other
arguments are given in the paper by Ramachandran and Hirstein).
An embodied, enacted view of perception is also taken by work in the
relatively new field of cognitive linguistics, which considers metaphor as a
key to understanding conceptual aspects of the human mind, thus placing it
squarely in the contested borderland between the sciences and the humanities.
Here metaphor is defined as ‘mapping across conceptual domains’, in which
‘the image schemata structure of the source domain is projected onto the
target domain in a way that is consistent with inherent target domain
structure’ (Lakoff, 1993), and image schemata are defined as ‘recurring
structures of, or in, our perceptual interactions, bodily experiences and
cognitive operations’ (Johnson, 1987). Work by Lakoff, Johnson, Fauconnier,
Turner and others demonstrates that (contrary to Brown) metaphor has deep
logical structure (Lakoff & Turner, 1989), involving the newly discovered
fundamental cognitive operation of blending (Turner & Fauconnier, 1995);
this logic can even be expressed precisely in the language of modern
mathematics called category theory (Goguen, 1999). Masako Hiraga has
successfully applied blending to classical Japanese haiku (Hiraga, 1999a,b).
One example of an image scheme is HIGHER IS BETTER, which enables us to
understand sentences like ‘his art reached new heights’. Such image schemas
are closely bound up with the contexts provided by the embedding of our
bodies in our physical and social worlds.
With an appropriate humility and trepidation, let us now approach the difficult
question of what distinguishes great art from merely good art. I suggest that it
may be helpful to take a spiritual perspective on this question, and would like
to begin by recalling from classical Buddhism the three marks of existence,
Pa
g
e 10 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
79
79
which are suffering, impermenance, and non-ego. Here ‘suffering’ refers to a
general sense of the unsatisfactoriness of human existence, that our hopes and
expectations are often frustrated, our goals often unachieved, and even when
they are achieved, there is a tendency to escalate them, raising the stakes, and
thus increasing the likelihood of later frustration. ‘Impermanence’ refers to the
temporary nature of all experience, achievements, and relationships;
eventually, we will all die; eventually, even the sun will die. The most
difficult of these three concepts is ‘non-ego’, which (in part) refers to the
constructed nature of the objects of our attention, including ourselves. For
example, a large square rock could be a bench, a coffee table, or an altar,
depending on the context; the situation is similar for all the objects of our
cognition — they take on a certain conceptual identity by virtue of their
context, though in general we simply take this for granted. Non-ego is also
one aspect of so called ‘emptiness’ (sunyata in sanskrit and muku in Japanese
—
there is no good western equivalent for this word); here can be found
several deeper aspects of non-ego.
Among the three marks of existence, we can perhaps consider non-ego to be
the abode of the sublime, in the sense of Kant, a kind of profound aesthetic
experience that goes beyond beauty, inspiring awe, perhaps even fear. This
can be explained by noting that the inner meaning of emptiness is a vast
p
otential for freedom and creativity, which at the same time is threatening,
because of its denial of ego. See also the last section of Brown, especially its
lovely final sentences, and the profound paragraph on great art in Ellis (p.172
below). Symbols of impermanence are often found in art; for example,
flowers play this role in much oriental art (for some reason, I am particularly
thinking of cherry blossoms falling into a stream, in one of Kurasawa’s films),
and much recent art questions our easy presupposition of everyday reality, in
various ways (e.g., dadaism, surrealism, pop art and conceptual art). Of
course, we should not neglect what Kant called ‘the beautiful’, which is
characterized by stillness, peacefulness and harmony, since the ability to
appreciate it properly serves as a ground for appreciating the sublime
p
roperly.
When deeply understood, the three marks of existence manifest as their polar
opposites: bliss, permanence and self-existence. Within the Western tradition,
these qualities seem to be most clearly expressed in medieval art, and they are
abundantly present in what the Western tradition has disdainfully called
Pa
g
e 11 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
80
80
‘primitive’ art.
Such an approach to art from religion is by no means incompatible with
science, because its fundamental paradigm is to examine human experience in
a state of meditation; the resulting philosophy is no more than an attempt (or
really, many different attempts) to bring some coherence to the results of this
examination. For example, the various philosophical assertions of Buddhism
are not seen as objective truths, but rather as a kind of phenomenological
classification of data gathered through meditation experience. Perhaps one day
science will be able to say some interesting things about this kind of
experience, much as it is now beginning to say some interesting things about
art. It should also be noted that the purpose of such philosophy is
soteriological, that is, aimed at improving human beings, rather than at
obtaining some kind of disembodied knowledge; I like to think this is also the
p
urpose of many great works of art.
From this perspective, it can be argued that there has been a progressive
diminution of the sacred in Western art from medieval times into the twentieth
century, with some recent attempts to rediscover the sacred in ways that differ
greatly from traditional Christianity, such as the close observation of the
textures of natural objects; a revival of medieval musical forms can also be
seen in the work of Arvo Part and others. However, most contemporary
artistic production is in the service of commercialism and consumerism, much
as most European art up to the nineteenth century was in the service of the
Church or the nobility.
In a different kind of connection between art and religion, many have been
surprised to learn that the recently discovered and much debated delay
between perception and consciousness was noticed and explained centuries
ago by Buddhist meditators, who further noted that an emotional evaluation
p
receeds conscious experience; see for example the lucid discussion of the
five skandhas (form, feeling, perception, concept and consciousness) in
Varela et al. (1991). This tradition also recognized that we do not perceive an
‘objective world’ that is ‘out there’, but rather, we construct our own world,
based on our values and expectations. When that ‘world’ is transformed
through the practice of meditation, it becomes ‘pure appearance’, also called
mahamudra, which is experience liberated as ‘self symbolic’, luminous and
transparent, without our usual overlays of projection and attachment; in this
way, all experience can become heightened aesthetic experience. All the
Pa
g
e 12 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
81
81
world’s great religions seem to have produced mystics who have described
roughly similar experiences, for example, Meister Eckhart (see, e.g., Forman,
1991).
The view that great art is related to the sacred stands in sharp contrast to
reductionist views. For example, Richard Taylor and others (1999) claim that
fractal dimension (Mandelbrot, 1977) is ‘an essential tool for determining the
fundamental content of the abstract paintings produced by Jackson Pollock in
the late 1940s’, and have shown that this metric gradually increases, ‘from
close to 1 in 1943 to 1.72 in 1952’. Their claim that ‘fractal analysis could be
used as a quantitative, objective technique both to validate and date Pollock’s
drip paintings’ seems perfectly valid, but it also seems clear that poor art
could easily be produced having any desired fractal dimension, e.g., by
choosing ugly colours and textures. It seems to me that the phenomeno-
logical dimension of great art is of key importance, and that this can never be
captured by any simple metric, whether fractal dimension, galvanic skin
response, or Birkhoff’s aesthetic ratio.
Finally, let us return to Richard Gregory’s question about whether there has
been any progress towards closing the gap between the ‘two cultures’ since
the time of C.P. Snow. One salient fact is that the disciplines of sociology,
linguistics and anthropology have each achieved a blend of scientific and
humanistic methods. After going through periods in which strongly
reductionist approaches were tried and then found wanting, it seems fair to say
that contemporary humanistic methods now predominate in these fields. Of
course methodological debates still rage, but it is also clear that some genuine
integration of the two cultures has been achieved in these fields. This essay
has also pointed out several interconnections among papers in this volume that
bridge the gap in various ways.
On the other hand, the presupposition seems widespread that science has a
p
rivileged position, in the sense that scientific assertions automatically
dominate assertions from other fields. While this is valid enough for those
assertions that have a thoroughly scientific character, it tends to obscure the
fact that knowledge obtained by the humanities often has a different quality;
for example, it may be phenomeno- logical, i.e., data of the ‘how it feels’
kind. This privileging of science, which can be seen in the editorial policies of
many journals, seems to me especially damaging to genuine dialogue between
the sciences and the humanities, and if anything, seems to be on the increase,
Pa
g
e 13 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
82
82
p
resumably due to the ongoing amazing successes of science and technology.
But by abandoning such artificial constraints, I believe that discourse like that
found in this volume can flourish, and demonstrate clearly that in some
important respects the gap between the two cultures has been significantly
narrowed.
Joseph A. Goguen, Dept. of Computer Science, UCSD, La Jolla, CA 92093-
0114, USA
Acknowledgements
I wish to thank Keith Sutherland and Anthony Freeman for their hard work on
this volume, without which it could never have come into being, to the extent
that they could well be considered its co-editors, though I carry full
responsibility for this essay. I also wish to thank Ruth Wallen for valuable
comments, and the Japan Society for the Promotion of Science for support
during the period in which this was written, mainly in various cafes and trains
around Kanazawa, Kyoto and Tokyo.
References
Birkhoff, G.D. (1928), Atti Congressi Bologna, 1, pp. 315–33.
Birkhoff, G.D. (1933), Aesthetic Measure (Cambridge: Cambridge University Press).
Chomsky, N. (1972), Language and Mind (San Diego, CA: HBJ College & Schools
Division).
Derrida, J. (1976), Of Grammatology, tr. G.C. Spivak (Baltimore, OH: John Hopkins
Univ. Press).
Dobson, J. (1998), ‘The iodine factor in health and evolution’, Geographical Review, 88
(1), pp. 1–28.
Forman, R.K.C. (1991), Meister Eckhart: Mystic As Theologian (Rockport, MA:
Element).
Goguen, J. (1999), ‘An introduction to algebraic semiotics, with applications to user
interface design’, in Computation for Metaphor, Analogy and Agents, ed. Chrystopher
N
ehaniv (Berlin: Springe
r
-Verlag, in press).
Pa
g
e 14 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
83
83
Hiraga, M. (1999a), ‘“Blending” and an interpretation of haiku: a cognitive approach’,
P
oetics Today (in press).
Hiraga, M. (1999b), ‘Rough sea and the milky way: “blending” in a haiku text’, in
Computation for Metaphor, Analogy and Agents, ed. Chrystopher Nehaniv (Berlin:
Springer-Verlag, in press).
Johnson, M. (1987), The Body in the Mind: The Bodily Basis of Meaning, Imagination,
and Reason (Chicago, IL: University of Chicago Press).
Lakoff, G. (1993), ‘The contemporary theory of metaphor’, in Metaphor and Thought,
ed. Andrew Ortony (Cambridge: Cambridge University Press, 2nd edition).
Lakoff, G. and Turner, M. (1989), More Than Cool Reason (Chicago, IL: University of
Chicago Press).
Mandelbrot, B.B. (1977), The Fractal Geometry of Nature (Oxford: W.H. Freeman).
Ramachandran, V.S. and Blakeslee, S. (1998), Phantoms in the Brain (New York:
William Morrow).
Snow, C.P. (1959), The Two Cultures and the Scientific Revolution (Cambridge
University Press).
Taylor, R.P., Micolich, A.P. and Jonas, D. (1999), ‘Fractal analysis of Pollock’s drip
p
aintings’, Nature, 399, p. 422.
Turner, M. and Fauconnier, G. (1995), ‘Conceptual integration and formal expression’,
Metaphor and Symbolic Activity, 10 (3), pp 183–204.
Varela, F., Thompson, E. and Rosch, E. (1991), The Embodied Mind (Cambridge, MA:
MIT Press).
Wilson, E.O. (1999), Consilience (London: Vintage).
Pa
g
e 15 of 15intro
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\intro.htm
84
84
V. S. Ramachandran and William Hirstein
Three Laws of Qualia
What Neurology Tells Us about the Biological
Functions of Consciousness, Qualia and the Self
Neurological syndromes in which consciousness seems to malfunction, such as temporal lobe
epilepsy, visual scotomas, Charles Bonnet syndrome, and synesthesia offer valuable clues
about the normal functions of consciousness and ‘qualia’. An investigation into these syn-
dromes reveals, we argue, that qualia are different from other brain states in that they possess
three functional characteristics, which we state in the form of ‘three laws of qualia’ based on a
loose analogy with Newton’s three laws of classical mechanics. First, they are irrevocable: I
cannot simply decide to start seeing the sunset as green, or feel pain as if it were an itch; sec-
ond, qualia do not always produce the same behaviour: given a set of qualia, we can choose
from a potentially infinite set of possible behaviours to execute; and third, qualia endure in
short-term memory, as opposed to non-conscious brain states involved in the on-line guidance
of behaviour in real time. We suggest that qualia have evolved these and other attributes (e.g.
they are ‘filled in’) because of their role in facilitating non-automatic, decision-based action.
We also suggest that the apparent epistemic barrier to knowing what qualia another person is
experiencing can be overcome simply by using a ‘bridge’ of neurons; and we offer a hypothe-
sis about the relation between qualia and one’s sense of self.
Introduction
Nothing is more chastening to human vanity than the realization that the richness of
our mental life — all our thoughts, feelings, emotions, even what we regard as our
intimate self — arises exclusively from the activity of little wisps of protoplasm in the
brain. The distinction between mind and body, illusion and reality, substance and
spirit has been a major preoccupation of both eastern and western thought for millenia
(Aristotle, 1961; Descartes, 1986; Fodor, 1975; Dennett, 1978; Searle, 1980). And
although these distinctions have generated an endless number of debates among phil-
osophers, little of lasting value seems to have emerged. As Sutherland (1989) has
said, ‘Consciousness is a subject on which much has been written but little is known.’
Our primary goal in this paper is to forge a fresh approach to the problem, by treating
it not as a philosophical, logical, or conceptual issue, but rather as an empirical prob-
lem. Our focus is on showing the form a scientific theory of consciousness might take,
something which is independent of the truth of all of the more detailed claims and
suggestions we will make. Our essay will consist of two sections. In part one, which
www.imprint-academic.com/rama
copyright © Journal of Consciousness Studies, 4, No. 5-6, 1997, pp. 429–58
85
85
philosophers can profitably skip, we describe some thought experiments to illustrate
the problem of qualia, since in our experience, most neuroscientists and even most
psychologists dispute the very existence of the problem. In part two, we offer numer-
ous examples from neurology and perceptual psychology that, together with a new
theoretical framework we offer, will help eventually solve the problem of conscious-
ness. Our theory should be seen as complementing rather than replacing a host of
other recent biological approaches to the problem such as those of Crick and Koch
(1992), Pat Churchland (1986), Baars (1988), Edelman (1989), Llinás (Llinás & Paré,
1991), Plum (Plum & Posner, 1980), Bogen (1995a,b), Gazzaniga (1993), Humphrey
(1993), Damasio (1994) and Kinsbourne (1995).
Much of our discussion will focus on the notion of qualia. It is our contention, how-
ever, that the problem of the self and the problem of qualia are really just two sides of
the same coin. In part, our argument is that the self is indeed something that arises
from brain activity of a certain kind and in certain brain areas, and that this activity is
also closely tied to functions related to qualia. In contrast to the idea that qualia are
private, subjective, and unsharable properties belonging exclusively to a private self,
we suggest two thought experiments to show that there is no insurmountable barrier
to sharing them. We then explore various issues involved in how qualia are generated
and managed by neural systems, and by examining pathological and experimental
cases that clarify these functions, we propose at the same time to clarify the nature of
the self. We conclude that the self, or the thing that leads to the illusion of a unitary,
enduring self, is neither a separable subject of consciousness nor a homunculus, but it
can be mapped anatomically to limbic and other associated structures which ‘drive’
frontal executive processes. This view contrasts sharply with the widely held view
that consciousness is based on the frontal processes themselves.
Part I: Epistemological Prolegomena
The qualia problem
We will illustrate the problem of giving an account of conscious experience, referred
to by philosophers as the problem of qualia,1with two simple thought experiments.
First, imagine that you are a future superscientist with a complete2knowledge of
the workings of the brain. Unfortunately however, you are a rod monochromat: you
don’t have any cone receptors in your eyes to delineate the different colours; you are
colour blind. For the sake of argument, however, let’s also assume that the central
processing mechanisms for colour in your brain are intact, they haven’t withered
away. This is not an illogical assumption; it’s fanciful perhaps, but not illogical.
430 V.S. RAMACHANDRAN AND W. HIRSTEIN
[1] Qualia are the ‘raw feels’ of conscious experience: the painfulness of pain, the redness of red. Qualia
give human conscious experience the particular character that it has. For instance, imagine a red
square; that conscious experience has (at least) two qualia: a colour quale, responsible for your
sensation of redness, and a shape quale, responsible for the square appearance of the imagined object.
[2] The assumption that anyone could ever have a complete knowledge of the brain is questionable,
depending of course on what one means by ‘complete’. All we mean by this is that the super- scientist’s
theory has no obvious explanatory gaps in it, and that it allows him to predict behaviour with an
extremely high level of accuracy. This example borrows liberally from Jackson’s ingenious ‘Mary’
scenario (Jackson, 1986).
86
86
You, the superscientist, study the brain of X, a normal colour perceiver, as he ver-
bally identifies colours he is shown. You’ve become very interested in this curious
phenomenon people call colour; they look at objects and describe them as red or
green or blue, but the objects often all look like shades of grey to you. You point a
spectrometer at the surface of one of the objects and it says that light with a wave-
length of 600nm is emanating from the object, but you have no idea what colour this
might correspond to, or indeed what people mean when they say ‘colour’. Intrigued,
you study the pigments of the eye and so on and eventually you come up with a com-
plete description of the laws of wavelength processing. Your theory allows you to
trace out the entire sequence of neural events starting from the receptors all the way
into the brain until you monitor the neural activity that generates the word ‘red’. Now,
once you have completely understood the laws of colour vision (or more strictly, the
laws of wavelength processing), and you are able to predict correctly which colour
word X will utter when you present him with a certain light stimulus, you have no rea-
son to doubt the completeness of your account.
One day you come up with a complete diagram. You show it to X and say, ‘This is
what’s going on in your brain.’ To which he replies, ‘Sure that’s what’s going on, but
I see red, where is the red in this diagram?’ ‘What is that?’ you ask. ‘That’s part of the
actual experience of the colour which it seems I can never convey to you,’ he says.
This is the alleged epistemological barrier which you confront in trying to understand
X’s experience. Our thought experiment is also useful in that it allows us to put for-
ward a clear definition of qualia: they are that aspect of X’s brain state that seems to
make your scientific description incomplete from X’s point of view.
Second, imagine there is a species of electric fish in the Amazon which is very
intelligent, in fact as intelligent and sophisticated as us. But it has something we lack:
the ability to sense electrical fields, using special organs in its skin. You can study the
neurophysiology of this fish and figure out how the electrical organs on the sides of its
body transduce electrical current, how this is conveyed to the brain, what part of the
brain analyses this information, how it uses this information to dodge predators, find
prey, and so on. If the electric fish could talk, however, it would say, ‘Fine, but you’ll
never know what it feels like to sense electricity.’ These two thought experiments ex-
emplify the problem of qualia. They are vaguely similar to Nagel’s ‘what is it like to
be a bat’ problem (‘You’ll never know what it’s like to be a bat’, Nagel, 1974), except
that our examples are better, for the following reason. In the Nagel version, it’s the
whole bat experience, the qualia produced by the bat’s radar system along with every-
thing else in its conscious mental life, which Nagel claims we cannot know. But this
misses the point. Most people would agree that you couldn’t know what it is like to be
a bat unless you are a bat — after all, the bat’s mental life is so completely, utterly dif-
ferent. In our electric fish example, however, we are deliberately introducing a crea-
ture which is similar to us in every respect, except that it has one type of qualia that we
lack. And the point is, even though your description of the fish is complete scientifi-
cally, it will always be missing something, namely the actual experience of electrical
qualia. This seems to suggest that there is an epistemological barrier between us and
the fish. What we have said so far isn’t new, except that we have come up with a
thought experiment which very clearly states the problem of why qualia are thought
to be essentially private. It also makes it clear that the problem of qualia is not neces-
sarily a scientific problem, because your scientific description is complete. It’s just
BIOLOGICAL FUNCTIONS OF CONSCIOUSNESS, QUALIA & SELF 431
87
87
that the description is incomplete epistemologically because the experience of elec-
tric current is something you never will know.
This is what philosophers have assumed for centuries, that there is a barrier which
you simply cannot get across. But is this really true? We think not; it’s not as though
there is this great vertical divide in nature between mind and matter, substance and
spirit. We will argue that this barrier is only apparent,3and that it arises due to lan-
guage. In fact, this barrier is the same barrier that emerges when there is any trans-
lation. The language of nerve impulses (which neurons use to communicate among
themselves) is one language; a spoken natural language such as English is a different
language. The problem is that X can tell you about his qualia only by using an inter-
mediate, spoken language (when he says, ‘Yes but there’s still the experience of red
which you are missing’), and the experience itself is lost in the translation. You are
just looking at a bunch of neurons and how they’re firing and how they’re responding
when X says ‘red’, but what X is calling the subjective sensation of qualia is supposed
to be private forever and ever. We would argue, however, that it’s only private so long
as he uses spoken language as an intermediary. If you, the colour blind superscientist,
avoid that and take a cable made of neurons from X’s area V4 (Zeki, 1993) and con-
nect it directly to the same area in your brain, then perhaps you’ll see colour after all
(recall that the higher-level visual processing structures are intact in your brain). The
connection has to bypass your eyes, since you don’t have the right cone cells, and go
straight to the neurons in your brain without an intermediate translation. When X
says ‘red’, it doesn’t make any sense to you, because ‘red’ is a translation, and you
don’t understand colour language, because you never had the relevant physiology and
training which would allow you to understand it. But if you skip the translation and
use a cable of neurons, so that the nerve impulses themselves go directly to the area,
then perhaps you’ll say, ‘Oh my God, I see what you mean.’ The possibility of this de-
molishes the philosophers’ argument (Kripke, 1980; Searle, 1980; 1992) that there is
a barrier which is insurmountable. Notice that the same point applies to any instru-
ments I might use to detect activity in your brain — the instrument’s output is a sort of
translation of the events it is actually detecting.
In principle, then, you can experience another creature’s qualia, for example even
the electric fish’s. It’s not inconceivable that you could find out what that part of the
brain is doing in the fish and that you could somehow graft it onto the relevant parts of
your brain with all the associated connections, and that you would then start experi-
encing the fish’s electrical qualia.4Now we could get into the philosophical debate
over whether you need to be a fish to experience it, or whether as a human being you
could experience it, but we’ve already made the distinction between the entire experi-
ence of being a fish, and the qualia themselves, which are just part of that experience.
Thus qualia are not the private property of a particular self; other selves can experi-
ence a creature’s qualia.
432 V.S. RAMACHANDRAN AND W. HIRSTEIN
[3] This idea emerged in discussions with F.H.C. Crick. See the acknowledgements at the end of this
paper.
[4] The same thought experiment can be performed within a single subject. Anaesthetize the corpus
callosum of a human at birth, expose the right brain alone to colours, then at age twenty-one
de-anaesthetize the callosum, in order to see if the left brain then begins to experience the right brain’s
qualia.
88
88
What are qualia for?
So far we’ve talked about the epistemology of qualia and we’ve suggested that there
is no barrier, and that you can in principle experience someone else’s qualia by using
a bridge of neurons — this problem may simply be a translation problem. We now
want to address the question of why qualia evolved. Many others have raised this
question before and come up with a wide range of different answers. One could also
put on the sceptic’s hat and say, ‘Since you have already shown that the scientific des-
cription is complete without qualia, it is meaningless to ask why it evolved or what its
function is. Doing so would entail converting a closed system — the physical uni-
verse — into an open one, and that would be a logical fallacy.’ We could, however,
temporarily set aside scepticism5and instead search for a reply to the questions ‘Why
did qualia emerge in evolution; or, why did some brain events come to have qualia?’
Is it a particular style of information processing that produces qualia, or is it a particu-
lar neural locus, or perhaps only some types of neurons are associated with qualia?
Crick (1996; Crick & Koch, 1992) has made the ingenious suggestion that the neural
locus of qualia is a set of neurons in the lower layers of the primary sensory areas,
because these are the ones that project to the frontal lobes. His approach has galva-
nized the entire scientific community (cf. Horgan, 1994) and has served as a catalyst
for those seeking biological explanations for qualia. Similarly, people have suggested
that it’s the synchronization of oscillations that leads to conscious awareness (Paré &
Llinás, 1995; Purpura & Schiff, 1997). This seems somewhat ad hoc, however —
why this rather than something else? These approaches are attractive, if only for one
BIOLOGICAL FUNCTIONS OF CONSCIOUSNESS, QUALIA & SELF 433
[5] Epiphenomenalism cannot be rejected on strictly logical grounds and can be defended on grounds of
parsimony; we may not need qualia for a complete description of the way the brain works. Since when,
however, has Occam’s razor been useful for scientific discovery? In fact, all of science begins with a
bold conjecture of what might be true. The discovery of relativity, for example, was not the product of
applying Occam’s razor to our knowledge of the universe at that time. The discovery came from
rejecting Occam’s razor and asking what if some deeper generalization were true, which is not required
by the available data, but which makes unexpected predictions (which later turn out to be parsimonious
after all). It is ironic that most scientific discoveries come not from brandishing (or sharpening)
Occam’s razor — despite the view to the contrary held by the great majority of scientists and
philosophers — but from generating seemingly ad hoc and ontologically promiscuous conjectures
which are not called for by the current data.
For the same reason, we are sympathetic to Penrose’s (1994) view that some hitherto undiscovered
physical principles may be required for explaining conscious experience. Although his particular
theory may turn out to be wrong (see, e.g., Grush and Churchland, 1995), we would argue that his idea
should not be rejected on the grounds of parsimony alone. The fact that nothing we know about
consciousness demands the postulation of new physical principles is not a sound argument against
seeking such principles.
In general then, although philosophical scepticism may be logically justified (just as we cannot
prove with complete logical certainty that we are not dreaming, or that your ‘red’ is not my ‘green’), it
is misplaced in the scientific realm, where one is concerned most often with what is likely to be true
‘beyond reasonable doubt’ — rather than with absolute certainty. Unless we set aside such misgivings
one is trapped in an intellectual stalemate. In this respect we are in complete agreement with Crick and
Koch (1992).
Another famous sceptic’s challenge (also known as Molyneux’s question) is ‘Can a person blind
from birth ever experience visual qualia?’ Although this is often posed as a conceptual dilemma, we
believe that it can be solved empirically by simply delivering localized transcranial magnetic
stimulation to visuotopic V1 in blind human volunteers, to see whether it evokes completely novel, yet
visuotopically organized visual qualia. (There is a paper by Ramachandran, Cobb & Hirstein, on this
topic in preparation.)
89
89
reason — that reductionism has been the single most successful strategy in science.
Unfortunately however, it is not always easy to know a priori what the appropriate
level of reductionism is for a given scientific problem (Churchland, 1996). Elucida-
tion of the role of the double helix in heredity turned out to be the most important sci-
entific discovery in this century (Medawar, 1969), because Crick and Watson had the
foresight and genius to realize that the molecular level was the appropriate one. Had
they chosen the quantum level, they would have failed! In a similar vein, we wouldn’t
expect an exhaustive description of the molecular structure of a mousetrap to reveal
its function. Nor would a parthenogenetic (asexual) Martian scientist understand how
the testicles worked by simply studying their structure, unless he knew about sex!
And yet this is precisely the strategy adopted by the vast majority of neuroscientists
trying to understand the functions of the brain.
Part II: The Biological Functions and Neural Basis of Qualia
In this essay we would like to try something different. We will deliberately begin at a
‘higher’ level of analysis, and use simple introspection as a strategy for elucidating
the biological functions of consciousness. Toward this end we will first present some
simple demonstrations of the ‘filling in’ of the natural blind spot of the eye
(Ramachandran, 1992) and argue that this can provide some strong hints about the
functions of qualia. Following these demonstrations we will examine a number of
neurological syndromes in which qualia seem to malfunction, which raises the possi-
bility that far from being a holistic property of the entire brain, qualia are indeed asso-
ciated with the activity of a small subset of neural structures, as suggested by Crick
(1994; 1996). We do not claim to have solved the problem of qualia, but at the very
least the examples and thought experiments should provide food for thought.
First, consider the well-known example of the blind spot corresponding to the optic
disc — the place where the optic nerve exits the back of the eyeball. To demonstrate
the blind spot to yourself, shut your right eye and hold Figure 1 about 10 inches away
from your face while looking at the small fixation star on the right. Now move the
page toward or away from your eye very slowly, and you will find that there is a criti-
cal distance at which the spot on the left completely disappears. Notice, however, that
when the spot disappears, it does not leave a gap or a dark hole behind in the visual
field. Indeed, the entire field looks homogeneous, and the region corresponding to the
blind spot is ‘filled in’ with the same texture as the background. Sir David Brewster,
who discovered filling in, believed it was evidence for a benevolent deity (1832):
‘The Divine Artificer has not thus left his works imperfect...thespot, in place of
being black has always the same colour as the ground.’ Curiously, Sir David was not
troubled by the question of why the Divine Artificer should have created an imperfect
eye to begin with!
Now close your right eye and aim the blindspot of your left eye at the middle of
your extended finger. The middle of the finger should disappear, and yet the finger
looks continuous. In other words, the qualia are such that you do not merely deduce
intellectually that the finger is continuous — ‘after all, my blind spot is there’ — you
literally see the missing piece of your finger. A dramatic demonstration of this phe-
nomenon is the following: if you show someone a donut shape so that the donut is
‘around’ the blind spot, say a yellow donut, and if the inner diameter of the donut is
434 V.S. RAMACHANDRAN AND W. HIRSTEIN
90
90
BIOLOGICAL FUNCTIONS OF CONSCIOUSNESS, QUALIA & SELF 435
Figure 1. The eye’s natural blind spot
Close your right eye and fixate the star with your left eye. Slowly move the page back and forth
about ten inches from your eye until the dark circle on the left disappears.
Figure 2a. Filling in
Cover your right eye and fixate your left eye on the small white cross. Move the figure back and
forth until your blind spot encompasses the centre of the ring on the left. Visual processes fill in the
centre of the ring so that it looks like a solid disc.
Figure 2b. Salience of filled-in objects
Cover you right eye and fixate your left eye on the small white square. Move the figure back and
forth until your blind spot encompasses the centre of the ring on the left of the square. The solid
filled-in disc will perceptually ‘pop out’ from the other rings.
91
91
slightly smaller than the blind spot, the donut will look like a complete, homogeneous
disk. In fact, the size of the donut can be such that you’re actually seeing three times
as much yellow now as you did before (see Figure 2a), which in turn means that your
brain actually ‘filled in’ your blind spot with qualia. The reason we emphasize this is
that there are some who have argued you simply ignore the blind spot and don’t notice
what’s going on (Dennett, 1991), so that there really is no filling in. But this can’t be
right, because if you show someone several rings, one of which alone is concentric
with the blind spot, that single one will look like a disc and will actually ‘pop out’ per-
ceptually (see Figure 2b). How can something you are ignoring pop out at you? This
means that not only does the blindspot have qualia associated with it, but that the
qualia can provide ‘sensory support’and therefore are being filled in preattentively,
so to speak.
As we have emphasized in previous papers (Ramachandran, 1993; 1995a,b;
Churchland and Ramachandran, 1993) we use the phrase ‘filling in’ in a somewhat
metaphorical sense. We certainly do not wish to imply that there is a pixel-by-pixel
rendering of the visual image on some internal neural screen, which would defeat the
whole purpose of vision (and would imply a ‘Cartesian theatre’, an idea which Den-
nett has brilliantly demolished). We disagree, however, with Dennett’s specific claim
that there is no ‘neural machinery’ corresponding to the blind spot. (There is, in fact, a
patch of cortex corresponding to each eye’s blind spot that receives input from the
other eye as well as the region surrounding the blind spot in the same eye; Fiorini et
al., 1992; see below.) What we mean by ‘filling in’ is simply this: that one quite liter-
ally sees visual stimuli (e.g. patterns and colours) as arising from a region of the vis-
ual field where there is actually no visual input. This is a purely descriptive,
theory-neutral definition of filling in and one does not have to invoke — or debunk —
homunculi watching screens to accept it. We would argue that the visual system fills
in not for the benefit of a homunculus but in order to make some aspects of the infor-
mation explicit for the next level of processing (Ramachandran, 1993). In the last sec-
tion we will argue that filling in is just one example of a general coherencing of
consciousness, which perceptual systems undertake in order to prepare representa-
tions to interact with limbic executive structures, an interaction from which both the
experience of qualia and intentionality emerge.
Now consider a related example. Suppose I put one finger in front of another finger
and look at the two fingers. Of course I see the occluded finger as continuous. I know
it’s continuous. I sort of see it as continuous. But if you ask me, do you literally see
the missing piece of finger, I would say ‘no’ — for all I know, someone could have
actually sliced two pieces of finger and put them on either side of the finger in front to
fool me. I don’t literally see that missing part.
Compare these two cases, the blind spot and the occluded finger, which are in fact
quite similar in that they are both cases where there is missing information which the
brain supplies. What is the difference, however? What difference does it make to you,
the conscious person, that the representation of the yellow donut now has qualia in the
middle and that the representation of the occluded finger part does not? The differ-
ence, we suggest, is that you cannot change your mind about the yellow in the middle
of the donut. In other words, you can’t think ‘Maybe it’s yellow, oh well, maybe it’s
pink, maybe it’s blue. You can’t think ‘Well, it’s probably yellow, but who knows, it
may be pink.’ No, it’s shouting at you ‘I am yellow’, with an explicit representation of
436 V.S. RAMACHANDRAN AND W. HIRSTEIN
92
92
yellowness in its centre. In other words, the filled-in yellow is not revocable, not
changeable by you. In the case of the occluded finger, however, you can think ‘there’s
a high probability that there is a finger there, but some malicious scientist could have
pasted two half-fingers on either side of it’, or, ‘there could be a little Martian sitting
there for all I know’. These scenarios are highly improbable, but not inconceivable.
Another way, then, to capture the difference between the two types of cases is that I
could choose to assume that there is something else behind the occluding finger, but
that I cannot do that with the filled-in region of the blind spot.
Thus the crucial difference between a qualia-laden percept and one that doesn’t
have qualia is that the qualia-laden percept is irrevocable, whereas the one which
lacks qualia is flexible; you can choose any one of a number of different ‘pretend’
inputs using top-down imagery. Once a qualia-laden percept has been created, you’re
stuck with it. A good example of this is that high-contrast photo of the dalmatian dog
(Figure 3). Initially, as you look it, it’s all fragments, then suddenly everything clicks
and you see the dog, you’ve got the dog qualia. The next time you see it, there’s no
way you can avoid it, and not see the dog. Indeed, we have recently shown that neur-
ons in the brain have permanently altered their connections once you have seen the
dog (Tovee et al., 1996).
Three laws of qualia
We now describe three laws of qualia (with apologies to Sir Isaac Newton) which we
hope will serve as guideposts for future inquiry. The examples we have just described
demonstrate an important feature of qualia: if something is revocable, it isn’t a quale
(or has only weak qualia associated with it). To put it less strongly, there is a link
between the strength or vividness of a quale and the degree of its irrevocability, i.e.,
this may be quantitative, rather than a qualitative distinction. However, although
something’s being irrevocable may be necessary, it is certainly not sufficient for the
presence of qualia. Why? Well, imagine that I shine a light into the eye of someone
who is in a coma. If the coma is not too deep, the patient’s pupil will constrict, even
though she will have no subjective awareness of any qualia caused by the light. The
entire reflex arc is irrevocable, and yet there are no qualia associated with it. You
BIOLOGICAL FUNCTIONS OF CONSCIOUSNESS, QUALIA & SELF 437
Figure 3
The irrevocability of
shape qualia
Once you see the dalma-
tion dog in the picture on
the left, it is impossible
to go back to the state of
not seeing it.
93
93
can’t change your mind about it, you can’t do anything about it, just like you couldn’t
do anything about the yellow filling in your blind spot in the donut example. So why
is it that only the latter has qualia? The key difference, we submit, is that there are no
qualia in the case of the pupil’s constriction because there is only one output avail-
able. But in the case of the yellow, even though the representation which was created
is irrevocable, what you can do with the representation is open-ended; you have the
luxury of choice. This is the second important feature of qualia: sensations which are
qualia-laden afford the luxury of choice. So now we have identified two functional
features of qualia: irrevocability on the input side, and flexibility on the output side.
There is a third important feature of qualia. In order to make decisions on the basis
of a qualia-laden representation, the representation needs to exist long enough for
executive processes to work with it. Your brain needs to hold the representation in an
intermediate buffer, in other words, in ‘working memory’. Again this condition is not
enough in itself, because there could be other reasons why a neural system needs to
hold some information in a buffer where qualia are not involved (e.g. spinal cord
‘memory’). Typically in these cases, however, there is only one output possible, in
which case the second important feature of qualia would be missing, on our scheme.
There is some physiological evidence for such a connection between qualia and mem-
ory. Goodale has reported a certain type of ‘blindsight’ patient who can correctly
rotate an envelope to post it in a horizontal or a vertical slot, even though he does not
consciously perceive the slot’s orientation and cannot tell you whether the slot is ver-
tical or horizontal (Milner & Goodale, 1995). But if the room lights are switched off
just before he puts the letter in, ‘he’ forgets the orientation of the slot almost immedi-
ately and is unable to get the letter in. This suggests that the unconscious ‘dorsal
stream’ visual system which discerns orientation and affects arm movements accord-
ingly is not only devoid of qualia but also does not have memory; it is the ‘ventral
stream’ visual system that is conscious and has memory. We would maintain that the
reason the qualia-laden ventral system has memory is because it is involved in mak-
ing choices based on perceptual representations. In contrast, the system without
qualia engages in continuous real-time processing running in a tightly closed loop
and consequently doesn’t need memory — it is not involved in the making of choices.
This suggests a testable prediction: in patients with blindsight, and in Goodale’s
visual zombie, if you give the patient a choice, the system should go haywire. Not
only should it not have short-term memory as Goodale showed, but also it should be
incapable of making choices. For example if the person is asked to mail a letter and
shown two orthogonal slots simultaneously, he should fail, being unable to choose
between the two (or alternatively, the system might always go for the first one it
detects). This is consistent with the Crick-Koch view that the neurons which project
to the frontal lobes are the qualia neurons because, obviously, the frontal lobes are
important for the execution of choices. We would argue, however, that what we think
of as the choice itself is really the work of a limbic executive system consisting of the
amygdala, anterior cingulate cortex, and other areas, and that the frontal lobes are
needed only for fully working out the long-range implications and possible alterna-
tives which the decision entails, and for dealing with complications arising as the
decision is executed (more on this in the final section).
Let’s extend the account to the qualia associated with pain. Say you prick some-
body with a pin. It’s well known that there are two components: there is an immediate
438 V.S. RAMACHANDRAN AND W. HIRSTEIN
94
94
withdrawal, involving no qualia, followed a couple of seconds later by the experience
of pain qualia. This dissociation is itself striking evidence for our view because the
non-qualia-laden pathway is irrevocable, but has a fixed output (withdrawal) and
therefore doesn’t have qualia in our scheme. The pain you experience, on the other
hand, is irrevocable, and what you do about it is flexible. You can put some medica-
tion on it, or you can run away from whatever caused it. This is a nice example
because it’s a case of the same stimulus producing two different streams of process-
ing, one involving qualia and the other not.
Bistable percepts
Let’s take bistable figures; how would our account apply to them? Here, the sensory
stimulus can specify two qualia with equal certainty, so the output system can only
choose between those two in creating an intermediate-level representation (Figure 4).
Once you settle on an interpretation, however, it clicks and if it’s revocable it’s only
in favour of a single other percept. You can only see that famous ambiguous figure as
a duck or a rabbit, for instance. But when you finally do see it, the implications are
infinite — this fulfills our criterion about output flexibility. In the spinal cord on the
other hand there are neural circuits that display a type of bistability, but the implica-
tions are finite. So for qualia to exist you need potentially infinite implications, but a
stable, finite, irrevocable representation as a starting point. But if the starting point is
revocable, then the representation will not have strong, vivid qualia. Good examples
of this are something seen behind an occluder, or imagining that there is a monkey sit-
ting on that chair. These do not have strong qualia, for good reason, because if they
did you wouldn’t be able to survive long, given the way your cognitive system is
structured. As Shakespeare said: ‘You cannot cloy the hungry edge of appetite by
bare imagination of a feast.’ Very fortunate, for otherwise you wouldn’t go eat, you
would just generate the qualia associated with satiety in your head. In a similar vein,
one could argue that a mutant creature that could imagine having orgasms is unlikely
BIOLOGICAL FUNCTIONS OF CONSCIOUSNESS, QUALIA & SELF 439
Figure 4
Bistable drawings
‘Ambiguous figures’
such as this one are
designed to allow two
possible interpretations.
Such figures offer a sort
of limited revocability:
one set of shape qualia
is revocable only in
favour of the other.
95
95
to pass on its genes to the next generation. Therefore (real perceptual) qualia are pro-
tected; they are partially insulated from top-down influences.
At the same time, however, you occasionally need to run a virtual reality simula-
tion using less vivid qualia generated from memory representations in order to make
appropriate decisions in the absence of the objects which normally provoke those
qualia. The memories one normally evokes in this case are not fully laden with qualia;
they have qualia which are just vivid enough to allow you to run the simulation. If
they possessed full-strength qualia, again, that would be dangerous; indeed that’s
called a hallucination. Presumably that’s what happens in temporal lobe seizures;
some mechanism has gone awry, and the virtual reality simulation has now become
like real sensory input. The simulation loses its revocability and generates pathologi-
cal qualia.
Why don’t these internally generated images, or beliefs for that matter, have strong
qualia? We can explain that. Percepts need to have qualia because they are driving
ongoing, decision-laden behaviour. You can’t afford the luxury of hesitating over the
percept itself, however. The stimulus ensemble determines it, and you don’t have
time to say, ‘Maybe it determines something else.’ You need to ‘plant a flag’ and say
‘This is it.’ Beliefs and internal images on the other hand should not be qualia-laden,
because they should not be confused with real perception; you need to be constantly
aware of their tentative nature. And by virtue of their tentative status beliefs lack
strong qualia — they are indefinitely revocable. So you believe — and you can imag-
ine — that under the table there is a cat because you see a tail sticking out, but there
could be a pig under the table with a transplanted cat’s tail. You must be willing to
entertain that hypothesis, however implausible, because every now and then you
might be surprised.
What is the computational advantage to making qualia irrevocable? One answer is
stability. If you constantly change your mind about qualia, then the number of poten-
tial outputs will literally be infinite; there will be nothing constraining your behav-
iour. At some point you need to say ‘this is it’ and plant a flag on it, and it’s that
planting of the flag that we call qualia. The perceptual system follows a rationale
something like this: given the available information, it is 90% certain that the object
perceived is red. Therefore for the sake of argument, I’ll assume that it is red and act
accordingly, because if I keep saying ‘maybe it’s not red’, I won’t be able to take the
next step. In other words, if I treated percepts like beliefs, I would be blind. Qualia
are irrevocable in order to eliminate hesitation and to confer certainty to decisions.
Charles Bonnet syndrome
This system can break down, however. For example, consider the curious neurologi-
cal disorder known as Charles Bonnet syndrome. Patients with this disorder typically
have damage to the retina, to the optic nerve, optic radiations, or sometimes even to
area 17, producing blindness in either a large portion or in the entire visual field. But
remarkably, instead of seeing nothing, they experience vivid visual hallucinations.
Typically these are ‘formed’ hallucinations rather than abstract patterns; i.e., the
patients claim to see little circus animals, or Lilliputian beings walking around. No
adequate explanation of the syndrome has been proposed to date, although the hallu-
cinations are sometimes referred to as ‘release hallucinations’ in the older clinical
literature.
440 V.S. RAMACHANDRAN AND W. HIRSTEIN
96
96
We recently had the opportunity to examine two patients with this syndrome, both
of whom present certain novel features, which may help to elucidate the neural
mechanisms underlying this disorder. These patients had a sharply circumscribed
region in the visual field where they were completely blind; i.e., they had a blind spot,
or scotoma. The remarkable thing is that their hallucinations are confined entirely to
the blind region. For example, patient MB had a left paracentral scotoma, about the
size of her palm (held out at arm’s length), caused probably by damage to area 17 and
the optic radiations, as a result of laser surgery to destroy an arteriovenous malforma-
tion. She was of course completely blind in this region, and yet as often as twenty or
thirty times a day she would experience the most vivid hallucinations confined
entirely to the blind spot. Surprisingly, these were static, outline drawings, like car-
toon drawings, filled in with colour, but having no depth or motion.
We suggest that the hallucinations associated with Charles Bonnet syndrome arise
because of the massive feedback projections (Ramachandran, 1993) that are known
to exist from higher cortical areas to visual areas that precede them in the hierarchy;
for example from area 17 to the LGN, or from IT and MT to areas 17 and 18 (Zeki,
1978; van Essen, 1979; Churchland et al., 1994). When a normal person imagines
something, such as a rose, we usually assume that some sort of activity is evoked in
the higher centres such as the temporal lobes, where the memory of this rose is stored
in the form of altered synaptic weights (and perhaps new synaptic connections). So
when you imagine a rose, one expects activity in the temporal lobes. But there is a
great deal of evidence now to suggest that in addition to the expected activity in IT,
there is also activity in area 17, as though somehow this information was being pro-
jected back onto your ‘neural screen’ corresponding to area 17 (Cohen et al., 1996;
Farah, 1989). It’s as though, to enable you to make certain fine spatial discrimina-
tions, your brain needs to run a sort of virtual reality simulation, and for some reason
this requires the participation of area 17. (In particular, discrimination of topological
features of the image, for example, may require that it be represented again in area
17).
However, when a normal person imagines a rose, she does not literally hallucinate
a rose; what she experiences is typically a faint, ghostlike impression of one. Why?
One possibility is that the normal person, unlike the Charles Bonnet patient, has real
visual input coming in from the retina and optic nerve. This is true, by the way, even
when the eyes are closed, because there is always spontaneous activity in the retina,
which may function to provide a null signal informing the higher centers that there is
no rose here, and this prevents her from literally hallucinating the rose. (Indeed, this
may be one reason why spontaneous activity in the peripheral receptors and nerves
evolved in the first place.) Again, all this is very fortunate, otherwise your mind
would be constantly flooded with internally generated hallucinations, and if you
begin confusing internal images with reality, you will be quickly led astray.
In the Charles Bonnet patient the visual input is completely missing, therefore the
internally generated images which are sent back to V1, or perhaps V2 (areas 17 and
18), assume a degree of vividness and clarity not seen in normal people. This explains
why the images are confined entirely to the scotoma, why they are so extremely vivid
(one patient told us that the colours ‘look more real than real colours’), and why they
have the irrevocable quality of genuine, stimulus-evoked qualia. In other words, ordi-
narily your top-down imagery will produce only weak images because there is com-
BIOLOGICAL FUNCTIONS OF CONSCIOUSNESS, QUALIA & SELF 441
97
97
peting real visual input (or spontaneous activity), but when the input goes away, then
you start confusing your internal images with external reality.
It is not clear why in the case of MB the images lacked depth and motion. One pos-
sibility is that for some reason the feedback information arises only from the ventral
stream (the IT-V4 pathway), which is concerned primarily with colour and form, and
there was no feedback from the dorsal stream and MT which would have conferred
the appropriate spatial attributes, such a depth and motion, to the image.
Perhaps a more important general implication of this syndrome which has been
overlooked in the past is that it is strong evidence for the idea that vision is not the
one-way cascade or flow of information which it is often thought to be. For example,
one simple-minded view of vision (Marr, 1982) holds that visual processing is
sequential, modular, and hierarchical: each box computes something and sends it to
the subsequent box, a model proposed frequently by AI researchers. This is clearly
not how human vision works (Edelman, 1989); instead, there seems to be a constant
echo-like back-and-forth reverberation between different sensory areas within the
visual hierarchy and indeed (as we shall see) even across modalities. To deliberately
overstate the case, it’s as though when you look at even the simplest visual scene, you
generate an endless number of hallucinations and pick the one hallucination which
most accurately matches the current input — i.e., the input seems to select from an
endless number of hallucinations.6There may even be several iterations of this going
on, involving the massive back-projections — a sort of constant questioning,asina
game of twenty questions, until you eventually home in on the closest approximation
to reality (a partially constrained hallucination of this sort is, of course, the basis of
the well-known Rorschach ink blot test). Thus what you finally see is the result of a
compromise between top-down processes and bottom-up processes, a very different
view from the conventional one in which vision is seen as involving a hierarchical
upward march of information; a bucket brigade.
Synesthesia
A second illustration of breakdown in the functions of qualia is provided by the
extraordinary phenomenon of synesthesia, where sensations evoked through one
modality produce vivid qualia normally associated with another modality. Many of
these cases tend to be a bit dubious — the claims of ‘seeing’ a sound or ‘tasting’ a col-
our turn out to be mere metaphors. However, we recently examined a patient who had
relatively normal vision up until the age of seven, then suffered progressive deteriora-
tion in his sight due to retinitis pigmentosa, until finally at the age of forty he became
442 V.S. RAMACHANDRAN AND W. HIRSTEIN
[6] This is analogous to the way in which the immune system works. When I inject you with killed or
denatured smallpox virus (antigens), they generate antibodies and lymphocytes that are specific to
smallpox. It was once believed by medical scientists (and is still believed by many laypeople and
philosophers) that upon entering your blood, the smallpox antigens — a protein molecule — instruct
the formation of specific antibodies by acting as a template. We know now that this view, while
intuitively plausible or obvious, is wrong. In fact, your body has antibody-producing cells for every
conceivable antigen; even ‘martian’ antigens, so to speak. What the antigenic challenge (smallpox, for
example) does is simply to select the appropriate clone of cells causing them to multiply and produce
the specific anti-smallpox antibody. This is a useful analogy, but there is of course a difference: the
random gene shuffling that leads to a multiplicity of antibodies has already been accomplished in the
fetus, and no longer goes on in the adult. In the case of perception, on the other hand, the random
combinations are tried out online even as you watch the stimulus.
98
98
completely blind. After about two or three years, he began experiencing visual hallu-
cinations similar to those experienced by Charles Bonnet patients. For example, he
would see little spots of red light which initially lacked depth, but which coalesced
over time to form the clear visual impression of a face, including depth and shading.
More interestingly however, this patient began to notice that whenever he palpated
objects while negotiating the visual environment, or held an object in his hand, or
even just read braille, this would conjure up the most vivid visual images, sometimes
in the form of unformed flashes, sometimes a movement or ‘pulsation’ of pre-existing
hallucinations, or sometimes the actual shape of the object he was palpating (e.g. a
corner). These images were highly intrusive, and actually interfered with his braille
reading and object palpation. We suggest that in this patient, as indeed in normal peo-
ple, palpating an object evokes visual memories of that object, as a result of a previ-
ously established Hebbian association.7Of course, when a normal person closes his
eyes and palpates a ruler, he doesn’t hallucinate a ruler, even though he will typically
visualize it. The reason, again, is because of the presence of normal, countermanding
visual input in the form of spontaneous activity from the retina and visual pathways.
But when this information is removed, as with the Charles Bonnet patient, our patient
begins hallucinating. This can be verified by directly recording evoked potentials
from his visual cortex while he is palpating objects (Cobb et al., in preparation).
Finally, this line of speculation is also consistent with what we have observed in
amputees with phantom limbs. After amputation, many of these patients experience a
vivid phantom arm, and while most of them are able to ‘move’ their phantom, a subset
of them find that the phantom is in a fixed position, i.e., their phantom is paralysed.
But what would happen if one were to somehow create the visual illusion that the
phantom had come back, and could move? To do this, we placed a vertical mirror on
the table in front of the patient in the sagittal plane. The patient then puts his normal
(say) right hand on the right side of the mirror and ‘puts’ his phantom left hand on the
left side of the mirror. He then looks at the mirror reflection of his right hand, and
moves his right hand around until its reflection is exactly superimposed on the felt
position of the phantom limb. If he now starts making movements with his right hand,
he gets the distinct visual illusion that his phantom hand is moving. Remarkably, this
also seems to produce vivid sensations seeming to come from joints and muscles in
the phantom limb, i.e., the patient experiences a curious form of synesthesia.
Such effects do not occur in normal individuals, supporting our conjecture that the
presence of real (somatosensory) input somehow prevents such synesthesia. In a nor-
mal person, even though there is a visual impression that their left hand is moving
(when they are actually looking at the mirror image of their right hand) this is contra-
dicted by somatic sensations which inform the brain that the left hand is not in fact
BIOLOGICAL FUNCTIONS OF CONSCIOUSNESS, QUALIA & SELF 443
[7] A second possibility is ‘remapping’. We have previously shown that upon amputation of an arm in a
human patient the brain area corresponding to the missing hand gets ‘invaded’ by sensory input from
the face. Consequently, touching the face evokes sensations in the missing phantom hand
(Ramachandran et al., 1995).
In a similar vein, when the visual areas — either cortical or subcortical — are deprived of input it is
not inconceivable that input from the somatosensory area ‘invades’ the vacated territory so that
touching stimuli begins to evoke visual sensations. The two hypotheses, haptically-induced visual
imagery vs ‘remapping’ can be distinguished by measuring the latency of evoked MEG responses
(Cobb et al., in preparation).
99
99
moving. The fact that this does not happen in the phantom limb patient may imply that
the visual signals are causing activation to travel back all the way to the primary
somatosensory areas concerned with proprioception. Again, this can be tested using
imaging techniques.
Filling in the blind spot
Is there an absolute, qualitative distinction between qualia-laden percepts and those
which are not; between perception and conception? Let us illustrate this point with
three thought experiments. Consider the obvious phenomenological distinction
between the region corresponding to my blind spot, where I can’t see anything, and
another sort of ‘blind spot’: the region behind my head, where I also can’t see any-
thing. In other words, each of us actually has three blind spots, one in the field of view
of each eye, and a third behind our heads, which is much larger. Now, ordinarily you
don’t walk around experiencing an enormous gap behind your head, and therefore
you might be tempted to jump to the conclusion that you are in some sense filling in
the gap. But obviously, you don’t: there simply is no visual neural representation in
the brain corresponding to this area behind your head. You fill it in only in the trite
sense that, for example if you are standing in a bathroom with wallpaper in front of
you, you assume that the wallpaper continues behind your head. But the important
point to emphasize here is, even though you assume (imagine, believe) that there is
wallpaper behind your head, you don’t literally see it. In other words, any ‘filling in’
is purely metaphorical and does not fulfill our criterion of being irrevocable. In this
fundamental sense there is an important distinction between filling in of the blind
spot, and our failure to notice the presence of a big gap behind your head (even though
it is the conceptual similarity between these two cases that has misled many psycho-
logists and philosophers to conclude that the eye’s blind spot is not filled in). Put very
simply, this means that in the case of the blind spot, as we said earlier, you can’t
change your mind about areas which have been filled in, whereas in the region behind
your head, you are free to think, ‘In all likelihood there is wallpaper there, but who
knows, maybe there is an elephant there.’
It would appear then, that filling in of the blind spot is fundamentally different,
both phenomenologically and in terms of what the neurons are doing, from your fail-
ure to notice the gap behind your head.8But the question remains, is the distinction
between what is going on behind your head and the blind spot qualitative or quantita-
tive, and is the dividing line completely arbitrary (cf. ‘Is a man bald if he has only
three hairs on his head?’)? To answer this, let us consider the following thought
experiment. Imagine we continue evolving in such a way that our eyes migrate
toward the sides of our heads, while at the same time preserving the binocular visual
field. The fields of view of the two eyes encroach further and further behind our heads
444 V.S. RAMACHANDRAN AND W. HIRSTEIN
[8] Gattass et al. (1992) showed that there is a patch of neurons in area 17 corresponding to the blind spot.
The neurons in this patch fire when there are two bars on either side of the blind spot, creating an
irrevocable representation in area 17. That is about as close as you can get to arguing that there is a
neural mechanism for filling in. To argue otherwise is pedantic.
The converse of qualia-laden filling in would be qualia-less ‘repression’ or inhibition of irrelevant,
confusing, or destabilizing information that would otherwise clutter up consciousness and ‘distract’
executive structures (Ramachandran, 1995b). Analogously, one might leave non-urgent mail sitting in
one’s mailbox lest it clutter up one’s desktop and distract one from more pressing matters.
100
100
until they are almost touching. At that point let’s assume you have a blind spot behind
your head (between your eyes) which is identical in size to the blind spot which is in
front of you. The question then arises: Would the completion of objects across the
blind spot behind your head be true filling in of qualia, as with the real blind spot, or
would it still be conceptual, revocable imagery or guesswork of the kind that you and
I experience for the region behind the head? The answer to this question, we suggest,
is that there will be a definite point when the images become irrevocable, and when
representations are created, or at least recreated and fed back to the early visual areas,
and at that point it becomes functionally equivalent to the blind spot. If this account is
true, there is indeed a fundamental qualitative change, both in the phenomenology
and in the corresponding information-processing strategies in the nervous system that
are used to create the representations.
Thus, even though blind-spot completion and completion behind the head can be
regarded as two ends of a continuum, evolution has seen fit to partition this contin-
uum in order to adequately ‘prepare’ the completed data for subsequent processing in
the case of blind spot completion. We suspect that the motive behind the partition has
to do with balancing the need to reduce the workload of higher-level processes by
passing them definite, perspicuous, gap-free representations on the one hand, with the
need to avoid error on the other. In the case of the eye’s blind spot, the chance that
something significant is lurking there is small enough that it pays simply to treat the
chance as zero. In the case of the blind area behind my head, however, the odds of
something important being there are high enough that it would be dangerous to fill in
this area with wallpaper or whatever pattern is in front of the eyes.
The second experiment might again be used to undermine the case for a strong
qualitative distinction between qualia-laden percepts and conceptual representations,
however. Let us go back to the example of the finger occluded by another finger. We
argued there that the region behind the occluder is at least partially revocable. How-
ever, consider the following intermediate case: a cat behind a picket fence. Or even
better, a cube hidden by three slats (Figure 5). It is very hard not to see a cube in this
figure. Here you have an intermediate case where the representation seems to be filled
BIOLOGICAL FUNCTIONS OF CONSCIOUSNESS, QUALIA & SELF 445
Figure 5
Intermediate case
There is a strong impression
that there is a complete cube
underneath the three slates,
but is this due to genuine
filling-in, or to conceptual
‘amodal completion’?
(After Kanizsa, 1979, and
Bregman, 1981.)
101
101
in and yet not filled in. However, the existence of such intermediate cases should not
forbid us from arguing that there may be separate neural mechanisms at the two ends
of the spectrum.
It is very unlikely that the visual system has evolved dedicated neural machinery
for the specific purpose of filling in the blind spot. What we are seeing here, instead,
may be a manifestation of a very general visual process — one that we may call sur-
face interpolation (Ramachandran, 1992; 1993; 1995b). It is very likely that the pro-
cess may have much in common with — and may involve some of the same neural
machinery as — the sort of filling in one sees in the example of the occluded finger
(which Kanizsa (1979) termed ‘amodal completion’). There are nevertheless impor-
tant differences between completion across the blind spot and amodal completion
(e.g. the occluded finger example), which implies that, although the two processes are
similar, they are not identical (contrary to the views of Durgin et al., 1995). The most
important difference, of course, is that filling in across the blind spot is modal,
whereas filling in behind occluders is amodal. What this means is simply that in one
case you literally see the filled-in sections, in the other case you don’t. (This distinc-
tion will not appeal to behaviourists but should be obvious to anyone who has care-
fully observed such stimuli and is not wholly devoid of common sense.)
A second difference between genuine filling in and conceptual or amodal comple-
tion is that the corner of a square or the arc of a circle will get completed amodally
behind an occluder but will not get completed modally across the blind spot
(Ramachandran, 1992; 1993). In fact, subjects sometimes report the corner or arc
being completed amodally behind an ‘imaginary’ occluder corresponding to the blind
spot; the occluder is usually reported to resemble an opaque smudged ‘cloud’.
In spite of the differences, it is very likely that the two completion processes share
some neural activity up to a certain stage in visual processing. Evidence for this
comes from the work of Gattass et al. (1992). They found that neurons in the patch of
area 17 corresponding to (say) the left eye’s blind spot respond not only to the right
eye (as expected) but also to two collinear line segments lying on either side of the left
eye’s blind spot — as though they were filling in this segment. Intriguingly, they also
noted that similar effects could sometimes be seen in the rest of the normal visual
field if a small occluder was used instead of the blind spot. The implication is that, at
least in the early stages of processing, both modal completion across the blind spot
(i.e. the filling in of qualia) and amodal completion behind occluders may be based on
similar neural mechanisms. But if so, why is there such a compelling phenomenologi-
cal difference between the two? One possibility is that the presence of the occluder
itself might be signalled by a different set of neurons which vetoes the modal comple-
tion process. This makes good functional sense, for if you were to hallucinate some-
thing in front of the occluder you might be tempted to grab it!
Consider a third example: the peculiar mental diplopia or ‘multilayer’ qualia asso-
ciated with locating objects in a mirror. Assume you’re looking into the rearview mir-
ror of your car, when suddenly you see the reflection of a red car zooming towards
you from behind. You accelerate rather than brake, even though, optically, the image
is in front of you and expanding. It is as if, when you look at the rearview mirror, you
are dealing with bilayered qualia. There is a sense in which you continue to localize
the image in front of you, and there is a sense in which you localize it behind you. This
raises an interesting question, namely, does the ‘location quale’ represent the object
446 V.S. RAMACHANDRAN AND W. HIRSTEIN
102
102
as being in front of you or behind you? (Qualia represent an object as being red or
square, but they also represent it as being in a certain location, egocentrically
specified.)
Now imagine that, instead of a rearview mirror, there is a small window in front of
you, and through that window you see a missile being hurled at you. Now of course,
you duck backwards. Even though the two situations are exactly equivalent optically
— there is an expanding retinal image — in the former case you accelerate forward
because somehow, at some level, your visual system performs the appropriate
transformation.
Another possibility is that when you look into the mirror you accelerate forward,
not because the location qualia are now actually behind you, but because you’ve
learned a reflex avoidance manoeuver, using the dorsal stream system alone. (So, in
this situation, the input is irrevocable, and the output is also not open-ended, i.e. it’s a
single behaviour.) On the other hand, the high-level revocable aspect of the experi-
ence — where you think, ‘Hey this is a mirror, so the object must be behind me,’
doesn’t have qualia either — it is more conceptual in nature. But at the critical inter-
mediate level, which is still qualia-laden, the object is still represented as being in
front of you. You look at the red object, and it’s clearly in front of you, and if a fly
appears on the mirror, it is right next to the red object. You certainly don’t experience
it behind your head. So what initially seems to be a disturbing borderline case, in fact
can be readily explained in terms of our overall conceptual scheme. But even so, the
example is thought-provoking and it leads to experimental questions,9such as: If
someone were to hurl a missile at you from behind, as you watched in a mirror, would
you duck forward or backward?
A fourth example of ‘bilayered’ (or bistable, really) qualia is shown in Figure 6
(below). What you see initially is a grey rectangle occluded partially by an opaque
white square with Swiss-cheese like holes in it. Obviously the grey of the rectangle is
not seen where it is occluded, but with a bit of practice you can get yourself to see this
as a transparent grey film stuck in front of the white square with holes. When you see
it this way, the film does have qualia because you ‘choose’ to see it in front and to flag
it with the appropriate qualia — preparing it for further processing, as it were.
Anosognosia, schizophrenia, and other delusional states
Notice that in the ‘cognitive’ realm this sort of completion or filling in is not unlike
the confabulations that right hemisphere stroke patients generate to ‘deny’ that they
are paralysed — an anomaly is simply explained away (Ramachandran, 1995c).
Some process located in the left hemisphere fills in gaps and smooths over contradic-
tions in the patient’s belief system (e.g., the contradiction between ‘I can use both
arms’ and ‘I can’t see my left arm moving’). We have suggested elsewhere that such
psychological defences evolved mainly to stabilize behaviour (they prevent your hav-
ing to orient to every kind of anomaly that threatened the status quo) and should be
seen as part of a general strategy for the ‘coherencing’ of consciousness: they help
BIOLOGICAL FUNCTIONS OF CONSCIOUSNESS, QUALIA & SELF 447
[9] Intriguingly, we have recently described a new neurological sign of right hemisphere disease, which
we call the looking glass syndrome, in which patients, while looking at a mirror, will reach for objects
‘inside’ the mirror, and assert that the object is inside or behind the mirror, even though they realize
they are looking into a mirror (Ramachandran et al., 1997).
103
103
avoid indecisive vacillation and serve to optimize resource allocation, and to facili-
tate rapid, effective action. Similarly, perceptual filling in occurs to keep conscious
qualia coherent, perspicuous, and distraction-free — it is another example of a gen-
eral strategy of coherencing consciousness.
The cognitive styles of the two hemispheres might be fundamentally different;
when faced with an ‘anomaly’ or discrepancy in sensory input, the left hemisphere
tries to ‘smooth over’ the discrepancy (employing denial, repression, or confabula-
tion) in the interest of preserving stability, whereas an ‘anomaly detector’ in the right
hemisphere tends to orient to the discrepancy and generate a paradigm shift in the
brain’s representation of the situation (Ramachandran, 1995c).
The dialectic between the opposing tendencies of the two hemispheres that we are
proposing also bears a tantalizing resemblance to what physicists refer to as the ‘edge
of chaos’ in dynamical systems: the emergence of ‘complexity’ at the boundary
between stability and chaos. Chaos arises in deterministic systems that show a highly
sensitive dependence on initial conditions. This is not unlike the sensitivity to pertur-
bation (or ‘anomalies’) that we have postulated for the cognitive style of the right
hemisphere. In marked contrast, the left hemisphere is relatively insensitive to change
and tries to preserve stability. Interesting or complex types of behaviour, on the other
hand, seem to emerge spontaneously at the boundary between the two — a place
where there is just enough novelty to keep things interesting and predictable but also
just enough stability to avoid complete anarchy and instability. And it is precisely
these little eddies of complexity at the border zone that may correspond roughly to
what we call human caprice, innovation and creativity.
There is a similarity between anosognosics and schizophrenics who have ‘positive’
symptoms. In the former, we have argued, there is a failure to register a mismatch
between expectation and current sensory input — leading to hallucinations (e.g. ‘I
can see my arm moving’) as well as delusions (‘My left arm is fine’) and memory dis-
448 V.S. RAMACHANDRAN AND W. HIRSTEIN
Figure 6. Bilayered qualia
Look at the figure and try to see the grey areas as part of a single translucent rectangle. This repre-
sents a higher level of filling in located somewhere between the filling in of the blind spot and amo-
dal filling in. (After Kanizsa, 1979)
104
104
tortions (‘I know that I am paralysed now; therefore I never denied that I was para-
lysed’). We suggest that such a failure results from damage to an anomaly or
mismatch detector in the right frontoparietal cortex of anosognosics but it is entirely
possible that some similar pathology may also underlie schizophrenia. Indeed, Frith
& Dolan (1997) have recently performed an ingenious experiment using the same
mirror box we use on our phantom limb patients (Ramachandran & Rogers-
Ramachandran, 1996) to demonstrate that a mismatch between vision and proprio-
ception results in right frontal activation (during a PET scan), independently of
whether the mismatch occurred on the right or left side of the body!
Surprisingly, there is no neurological syndrome in which one sees exactly the same
types of ‘positive symptoms’ — i.e. combinations of hallucinations and delusions —
that occur in schizophrenia (Frith & Dolan, 1997). We would venture to predict, how-
ever, that if someone developed Charles Bonnet syndrome (from ocular pathology)
combined with a right frontoparietal lesion (causing a failure to register a mismatch
between fantasy and reality) then you would come pretty close to a neurological
equivalent of schizophrenia, for such a patient would take his hallucinations quite lit-
erally — not recognizing them to be illusory. Surprisingly, we have recently seen a
phantom limb patient who precisely fits this general description. He lost his left arm
in a car accident in which he also suffered bilateral frontal damage. While most peo-
ple who lose an arm experience the illusion of a persisting arm, i.e., a phantom limb,
they obviously do not literally see the arm or believe that the arm still exists. Our
patient (DS), on the other hand, insisted that his arm was still there and had not been
lost, even though he was quite lucid mentally in other domains (Hirstein and
Ramachandran, 1997).
Qualia of percepts vs. qualia of beliefs
Beliefs are also associated with ‘partial qualia’ and conscious awareness, once they
are made explicit in ‘working memory’. In the absence of sensory support, however,
the qualia associated with beliefs are fleeting and less robust than the real qualia-
laden percepts associated with sensory stimuli. Therefore the distinction between
qualia associated with percepts and those associated with explicit (or occurrent)
beliefs may be quantitative rather than qualitative. Tacit beliefs, on the other hand, are
completely qualia-free.
As an analogy, consider the distinction between ‘knowing’ and ‘remembering’ or
between ‘procedural’ and ‘episodic’ memory in humans. We know that episodic
memories are partially qualia-laden, whereas skills are not (Tulving, 1983). How-
ever, when a bee does a waggle dance, it is communicating an episodic memory. Why
does this not qualify as an episodic memory analogous to human episodic memory?
To argue that the bee is not conscious, and true episodic memories are conscious,
would be circular, and does not answer the question. The problem is readily solved in
our scheme, however, since in the bee, the alleged episodic memory is available for
the production of only one (or two) outputs, and hence the bee lacks the second of our
defining features of consciousness: flexibility of output.10
BIOLOGICAL FUNCTIONS OF CONSCIOUSNESS, QUALIA & SELF 449
[10] One prediction here is that a non-conscious zombie, such as Milner’s visual zombie, or perhaps certain
sleepwalkers should not have episodic memories.
105
105
This is one advantage that our scheme has over other theories of consciousness: it
allows us to unambiguously answer such questions as, Is a sleepwalker conscious? Is
the spinal cord of a paraplegic conscious? Is a bee conscious? Is an ant conscious
when it detects pheromones? In each of these cases, instead of the vague assertion that
one is dealing with various degrees of consciousness, which is the standard answer,
one should simply apply the three criteria we have specified. For example, Can a
sleepwalker make choices? Does he have short-term memory? Does a patient with
akinetic mutism have short-term memory? Can the bee use its waggle dance for more
than one output?, etc., thereby avoiding endless semantic quibbles over the exact
meaning of the word ‘consciousness’.
The importance of the temporal lobes for consciousness and qualia
‘Does any of this yield clues as to where in the brain might qualia might be?’, you ask.
It is ironic that people have often thought that the seat of consciousness is the frontal
lobes, because nothing dramatic happens to consciousness if you damage the frontal
lobes. We suggest instead that most of the action is in the temporal lobes. Admittedly
this allows us only a fourfold reduction in the problem space, since the brain has only
four lobes; but at the very least it may help us narrow down the problem by allowing
us to focus on specific neural structures and their functions. In particular, we suggest,
one needs the amygdala and other parts of the temporal lobes for seeing the signifi-
cance of things to the organism. Without this structure you are like Searle’s Chinese
room (Searle, 1980): capable of giving a single correct output in response to a
demand, but with no ability to sense the meaning of what you are doing or saying.11
Our claim that qualia are based primarily in the temporal lobes is consistent with
the idea put forward by Jackendoff (1987) and Crick (1996) that qualia and con-
sciousness are associated not with the early stages of perceptual processing (at the
level of the retina, for instance), where (in our scheme) obviously multiple choices
are not possible. Nor are they associated with the final stages of perceptual processing
and behaviour planning, where behavioural programs are executed. Rather, they are
associated with the intermediate stages of processing. The temporal lobes are in fact
the interface between perception and action.
Another piece of evidence for the idea that the temporal lobes are the neural locus
of consciousness and qualia is that the brain lesions which produce the most profound
disturbances in consciousness are those which generate temporal lobe seizures.
Researchers who electrically stimulate the temporal lobes of epileptics prior to per-
forming lobectomies have found the temporal lobes to be the best place for producing
conscious experiences in their subjects (Penfield & Perot, 1963; Gloor et al., 1982;
Gloor, 1992; Bancaud et al., 1976). Stimulating primary sensory areas, such as the
visual cortex, can produce strange, unformed qualia, such as phosphene flashes, but
only, we suspect because the events set in place by the stimulation eventually follow
the natural course of processing into the temporal lobes, and produce (weak) effects
there. Stimulating the amygdala is the surest way to ‘replay’ a full, vivid experience,
such as an autobiographical memory complete with intense emotions, or a vivid hal-
450 V.S. RAMACHANDRAN AND W. HIRSTEIN
[11] This reminds one of the old quip in which one behaviourist zombie turns to his mate after passionate
lovemaking and says, ‘I know it was good for you, but was it good for me?’, a question which
encapsulates the entire Searle/Dennett debate.
106
106
lucination (Gloor, 1992). The seizures which temporal lobe epilepsy (TLE) sufferers
endure are associated not only with alterations in consciousness in the sense of per-
sonal identity, personal destiny, and personality, but also with vivid, qualia-laden hal-
lucinations such as smells and sounds (MacLean, 1990; Bear, 1979; Waxman &
Geschwind, 1975; Gloor, 1992; Bancaud et al., 1994). If these are mere memories as
they are sometimes claimed to be, why would the person say ‘I literally feel like I’m
reliving it’? What characterizes these seizures is the vividness of the qualia they pro-
duce. So the smells, the pains, the tastes and the emotional feelings, all of which are
generated in the temporal lobes, suggest that they are in fact the seat of consciousness.
Another reason for choosing the temporal lobes — especially the left temporal lobe
— as the main player in generating conscious experience is that this is where much of
language — especially semantics — is represented. If I see an apple, it is the activity
in the temporal lobes that allows me to apprehend all its implications almost simulta-
neously. Recognition of it as a fruit of a certain type occurs in IT (infero- temporal
cortex), the amygdala gauges its significance for my well-being, and Wernicke’s and
other areas alert me to all the nuances of meaning that the mental image — including
the word ‘apple’ — evokes; I can eat the apple, I can smell it, I can bake a pie, remove
its pith, plant its seeds, use it to ‘keep the Doctor away’, tempt Eve, and on and on.
If one enumerates all of the attributes that we usually associate with the words
‘consciousness’ or ‘awareness’, each of them, you will notice, has a correlate in tem-
poral lobe seizures:
(1) Sensory Qualia — the raw feel of sensations, such as colour or pain. TLE: Vivid
visual and auditory hallucinations; the patient always notices that these look and
feel like the real thing — they do not merely have the fleeting qualia of memories
(Penfield & Jasper, 1954).
(2) The attachment of emotional significance and value labels to objects and events.
TLE (especially seizures involving the amygdala): The patient may see cosmic
significance in everything around him (Waxman and Geschwind, 1975), or feel
intense fear (Strauss et al., 1982). Conversely, bilateral damage to the amygdala
may lead to a loss of emotion and empathy, or to the ‘psychic blindness’ and
unthinking, automatic behaviour characteristic of the Kluver-Bucy syndrome
(Lilly et al., 1983). It is a moot point whether such a person would have any vis-
ual qualia. (One could regard the zombie-like behaviour of Goodale’s patient as
an extreme example of this.)
(3) Body image — the sense of being corporeal and of occupying a specific location
in space.TLE: Autoscopic hallucinations (Devinsky et al., 1989), ‘out of body’
experiences. Also, the temporal lobes and the limbic system receive a more mas-
sive projection form the viscera than any other part of the brain. The construction
of a body image is one of the foundations of our sense of self but, as we will show
in the next section, the body image is a merely a temporary construct, and in the
next section we will describe experiments that clearly demonstrate its transitory
nature.
(4) Convictions of truth or falsehood. TLE: An absolute sense of omnipotence or
omniscience (Bear, 1979; Trimble, 1992). It seems ironic that our convictions
about the absolute truth or falsity of a thought should depend not so much on the
BIOLOGICAL FUNCTIONS OF CONSCIOUSNESS, QUALIA & SELF 451
107
107
propositional language system but on much more primitive limbic structures
which add a form of emotional qualia to thoughts, giving them a ‘ring of truth’.
This would explain why the more dogmatic assertions of priests as well as scien-
tists are so notoriously resistant to correction through intellectual reasoning!
(5) Unity — the sense of being a single person despite experiencing a lifetime of
diverse sensory impressions. TLE: Synesthesia; doubling of consciousness; mul-
tiplication of personal identity, e.g. in Capgras syndrome (which we have argued
is due primarily to a temporal lobe lesion, see Hirstein & Ramachandran, 1997)
and other reduplicative paramnesias, the patient may come to regard himself as
more than one person. Similarly, multiple personality disorder (MPD) is often
seen in association with TLE (Schenk & Bear, 1981; Ahern et al., 1993).
(6) Free will — the sense of being able to make a decision or control one’s move-
ments. TLE: Even though the ability to engage in long range-planning is lost
mainly in frontal disease, it is damage to the cingulate (which is part of the limbic
system) that often results in something like ‘disorders of the will‘ (e.g. the alien
hand syndrome (Goldberg et al., 1981), akinetic mutism: ‘loss of will’ (Nielson
and Jacobs, 1951). Zombie-like automatisms are a frequent concomitant of TLE
seizures, and also result from stimulation of the anterior cingulate gyrus (Ban-
caud et al., 1976). It would be interesting to find out whether the patient can make
actual choices during such states (we would argue that they cannot).
Furthermore, one frequently sees profound alterations in conscious experience —
such as loss of contact with reality (de-realizations) and dream-like trance states dur-
ing TLE seizures. While each of the disorders listed above can also be seen when
other brain areas are damaged (e.g. body image distortions in parietal lobe syn-
dromes), almost all of them can be seen in various combinations when the temporal
lobes are damaged. Thus if there is a single brain region that can be regarded as criti-
cal for generating conscious experience, it would be the temporal lobes and various
interconnected parts of the amygdala, the inferotemporal cortex, Wernicke’s area and
other associated structures (e.g. the cingulate gyrus). Remove these and you have the
prototypical zombie of philosophers’ thought experiments.
A new illusion of decapitation
We will now describe an illusion which demonstrates how the body image — despite
its apparent durability and permanence — is an entirely transitory internal construct
that can be profoundly altered by the stimulus contingencies and correlations that one
encounters. Consider the following two illusions, the ‘phantom nose’ and the ‘phan-
tom head’ that we recently discovered in our laboratory. In the first experiment, the
subject sits in a chair blindfolded, with an accomplice sitting at his right side, or in
front of him, facing the same direction. The experimenter then stands near the sub-
ject, and with his left hand takes hold of the subject’s left index finger and uses it to
repeatedly and randomly tap and stroke the nose of the accomplice, while at the same
time, using his right hand, he taps and strokes the subject’s nose in precisely the same
manner, and in perfect synchrony. After a few seconds of this procedure, the subject
develops the uncanny illusion that his nose has either been dislocated, or has been
stretched out several feet forwards or off to the side, demonstrating the striking plas-
452 V.S. RAMACHANDRAN AND W. HIRSTEIN
108
108
ticity or malleability of our body image. The more random and unpredictable the tap-
ping sequence the more striking the illusion. We suggest that the subject’s brain
regards it as highly improbable that the tapping sequence on his finger and the one on
his nose are identical simply by chance and therefore ‘assumes’ that the nose has been
displaced — applying a universal Bayesian logic that is common to all sensory sys-
tems. Interestingly, once the illusion is in place, if a drop of ice-cold water is now ap-
plied to the subject’s nose, the cold is sometimes felt in the new location of the nose.
The phantom nose illusion is a very striking one, and we were able to replicate it on
twelve out of eighteen naive subjects.12 Rather surprisingly, the illusion sometimes
works even if the accomplice sits facing the subject; the logical absurdity of the situa-
tion seems not to veto the effect. This simple experiment demonstrates the single
most important principle underlying the mechanisms of perception and conscious
experience: that they may have evolved exclusively for extracting statistical regulari-
ties from the natural world.
In the second experiment we had a naive subject looking at his own reflection in a
half-silvered mirror, and placed a dummy’s head on the other side of the mirror, opti-
cally superimposed in exact registration on the subject’s own reflection. The lights
are switched off and the upper half of the dummy’s face, including the nose, is illumi-
nated with one spotlight and the lips alone of the subject are illuminated separately
with a different light source. When the subject looks at the mask, he sees a combina-
tion of the top of the mask and, reflected in the glass, the bottom of his face. If the sub-
ject is asked to make large lip and tongue movements (and baring of the teeth), he
develops the uncanny experience of being in direct control of the dummy’s facial
movements, as though his ‘will’ was manifesting itself through the dummy’s mouth.
It is as though the brain regards it highly improbable that the lips of the dummy should
be so perfectly synchronized with his own motor speech commands, and therefore
assumes that the subject’s own free will has taken over the dummy.13
To test this objectively, we pinched the dummy’s face and found that this evoked a
striking increase in the subject’s skin conductance response, whereas simply pinch-
ing the dummy without the initial lip movements evoked a much smaller response
(Ramachandran et al., in preparation). The extraordinary implication is that, using
this relatively simple procedure, we had successfully ‘decapitated’ the subject, induc-
ing the self to temporarily cast off its mortal coil to inhabit the dummy. The subject
comes to experience the dummy’s head as being his own to such an extent that it is
now hooked up to his own limbic system and autonomic output. Even intermittent,
unpredictable tactile stimulation (touch, cold, pain) delivered to the subject’s face
were occasionally referred to the dummy in a modality-specific manner (in a manner
analogous to the referral of tactile stimulation to visually resurrected phantom limbs;
BIOLOGICAL FUNCTIONS OF CONSCIOUSNESS, QUALIA & SELF 453
[12] It has not escaped our notice that if a willing accomplice were available, the effect could also be
produced using other body parts.
[13] The sceptic could ask, How is this situation fundamentally different from ordering another human
being — such as a valet — to perform an elaborate series of actions, or controlling a marionette on
strings with one’s fingers? The answer is that in the former case there is no perfect temporal synchrony
between the orders issued and the actions performed by your subordinate; and in the latter case, even
though there is some degree of synchrony, the movement trajectories and the body parts involved in the
marionette are different from those of the puppeteer. This explains why the transfer of free will
requires an experimental setup similar to the one we describe.
109
109
Ramachandran and Rogers-Ramachandran, 1996). The observation also lends credi-
bility to the reports of the self temporarily deserting the body: out of the body experi-
ences and ‘autoscopic hallucinations’ in parietal lobe syndrome and ketamine
anesthesia.
Qualia and ‘the self’
We have discussed qualia and the body image,14 but what about the self? Even though
the notion of a unitary, enduring self may turn out to be a form of adaptive self- decep-
tion or delusion (Ramachandran, 1995b) we must consider why the illusion arises.
We also need to consider the question of who the so-called observer is in the two
thought experiments we began with. Since qualia-laden percepts are generated for
someone or something — presumably ‘the self’ — the problem of the self and the
problem of qualia are really just two sides of the same coin.
One way to approach the question of how our account of qualia relates to the ques-
tion of the self is to ask from a scientific point of view why something like filling in of
the blind spot with qualia-laden representations occurs. The original motive many
had for arguing that the blind spot is not filled in was that there is no one there to fill
them in for — that no homunculus is there looking at them (Dennett, 1991). This is an
argument against the following line of reasoning: ‘If qualia are filled in, they must be
filled in for some viewer, i.e., a homunculus.’
There is reason to think that the conclusion is false (i.e., there is no homunculus), it
was argued, and hence reason to think that the antecedent is also false: qualia are not
in fact filled in, and that the appearance that they are is an illusion (Dennett, 1991).
Now, since we have argued that qualia are in fact filled in (Ramachandran, 1992;
1993; 1995a; Ramachandran & Gregory, 1991), does this mean that we believe they
are filled in for a homunculus? Of course not, but the fallacy may not be in the form of
the reasoning, just in the illegitimate specificity with which the conclusion is stated.
The above argument is really a ‘straw man’; the line of reasoning should run: ‘If
qualia are filled in, they are filled in for something.’
Now, what is the ‘something’ here? There exists in certain branches of psychology
the notion of an executive, or a control process (McKay, 1969). These processes are
generally taken to be frontal, or prefrontal, but we would like to suggest that the
something which qualia are filled in for is a sort of executive process, but a limbic15
454 V.S. RAMACHANDRAN AND W. HIRSTEIN
[14] Our ‘phantom nose’ effect is quite similar to one reported by Lackner (1988) except that the underlying
principle is different. In Lackner’s experiment, the subject sits blindfolded at a table, with his arm
flexed at the elbow, holding the tip of his own nose. If the experimenter now applies a vibrator to the
tendon of the biceps, the subject not only feels that his arm is extended — because of spurious signals
from muscle stretch receptors — but also that his nose has actually lengthened. Lackner invokes
Helmholtzian ‘unconscious inference’ as an explanation for this effect (I am holding my nose; my arm
is extended; therefore my nose must be long). The illustration we have described, on the other hand,
does not require a vibrator and seems to depend entirely on a Bayesian principle — the sheer statistical
improbability of two tactile sequences being identical. (Indeed, our illusion cannot be produced if the
subject simply holds the accomplice’s nose.) Not all subjects experience this effect, but that it happens
at all is astonishing: that a lifetime’s evidence concerning your nose can be negated by just a few
seconds of intermittent tactile imput.
[15] The limbic system includes the hypothalamic nucleii, amygdala insula, interstitial nuclei of the striae
terminalis, fornix and fimbria, septum, mamillary bodies and cingulate gyrus, but the exact definition
is not critical to our argument. The cholinergic lateral dorsal tegmental and pedunculopontine nuclei
and the intralaminar thalamic nucleii that project to limbic structures may be an integral part of the
110
110
one, rather than a frontal one. This would be a process involved in connecting motiva-
tion and emotion with the choice of actions to perform, based on a certain definite
incoming set of qualia — very much the sort of thing which the self was traditionally
supposed to do. A control process is not something which has all the properties of a
full human being, of course — it is not at all a homunculus. All the notion of a control
process entails, as we are employing it, is that control processes are guided by some
brain areas (i.e. perceptual areas and motivational areas) as they control the activities
of other brain areas (i.e. motor and planning areas).
Seen this way, filling in is a kind of treating and preparing of qualia in order for
them to interact properly with limbic executive structures. Qualia may need to be
filled in before they causally interact with these structures because gaps interfere with
the proper working of these executive structures. To speak metaphorically, perhaps
the control structures are prone to be distracted by gaps in a way which greatly
reduces their efficiency and their ability to select appropriate output. The processes
involved in generating qualia smooth over anomalies in their product in the same way
in which the president’s advisers might remove any little confusions or fill in any
gaps in the data they give him, inconsequential confusions and gaps which might u-
nnecessarily distract his attention from the main message of the data, causing him to
take longer to make a decision, or worse, to make the wrong decision.
Where in the limbic system are these control processes? Perhaps a system involv-
ing the amygdala and the anterior cingulate, given the amygdala’s central role in emo-
tion (LeDoux, 1992; Halgren, 1992), and the anterior cingulate’s apparent executive
role (Posner & Raichle, 1994; Devinsky et al., 1995), and the connection between its
damage and disorders of the will, such as akinetic mutism and alien hand syndrome. It
is not difficult to see how such processes could give rise to the mythology of a self as
an active presence in the brain — a ‘ghost in the machine’.
Acknowledgments: We thank P.S. Churchland, F.H.C. Crick, R.L. Gregory, D.C. Dennett,
M. Kinsbourne and J. Smythies for stimulating discussions, and the NIMH for support. The idea
that the Charles Bonnet syndrome might arise from the activity of feedback pathways projecting
back to areas 17 and 18 was first suggested by one of us (VSR) in two interviews (Grady, 1993;
Nash, 1995). The notion that the epistemic barrier to sensing another person’s qualia results entirely
from a translation problem emerged from conversations (and correspondence) with F.H.C. Crick in
1984. Our ideas about bees emerged from discussions with M. Hauser.
References
Ahern, G.L., Herring, A.M., Tackenberg, J. et al. (1993), ‘The association of multiple personality and
temporolimbic epilepsy’, Archives of Neurology,50, pp. 1020–5.
Aristotle (1961), DeAnima (Oxford: Clarendon Press).
Baars, B. (1988), A Cognitive Theory of Consciousness (New York: Cambridge University Press).
Bancaud, J. et al. (1976), ‘Manifestations comportmentales induites par la stimulation electrique du gyrus
cingulaire anterieur chez l’homme’, Revue Neurologique,132, pp. 705–24.
BIOLOGICAL FUNCTIONS OF CONSCIOUSNESS, QUALIA & SELF 455
qualia-linked circuitry, but it remains to be seen whether they merely play a ‘supportive’ role for qualia
(as indeed the liver and heart do!) or whether they are a part of the actual circuitry that embodies qualia;
i.e. are they analogous to the power supply of a VCR or television set, or to the actual magnetic recording
head and cathode ray tube? Hyperactivity of this system may contribute to peduncular hallucinosis. Also
the doubling of dorsal tegmental and pedeuneulopontine cell numbers which is known to occur in schizo-
phrenia may help explain hallucinations. (This idea emerged in conversation with John Smythies.)
111
111
Bancaud, J., Brunet-Bourgin, F., Chauvel, P. & Halgren, E. (1994), ‘Anatomical origin of deja vu and
vivid ‘’memories’’ in human temporal lobe epilepsy’, Brain,117, pp. 71–90.
Bear, D.M. (1979), ‘Personality changes associated with neurologic lesions’, in Textbook of Outpatient
Psychiatry, ed. A. Lazare (Baltimore, MD: Williams and Wilkins Co.).
Bogen, J.E. (1995a), ‘On the neurophysiology of consciousness: Part I. An overview’, Consciousness and
Cognition,4, pp. 52–62.
Bogen, J.E. (1995b), ‘On the neurophysiology of consciousness: Part II. Constraining the semantic prob-
lem’, Consciousness and Cognition,4, pp. 137–58.
Bregman, A. (1981), ‘Asking the ‘’what for’’ question’, in Perceptual Organization, ed. M. Kubovy &
J. Pomerantz (Hillsdale, NJ: Lawrence Erlbaum Associates).
Brewster, D. (1832), Letters In Natural Magic (London: John Murray).
Churchland, P.S. (1986), Neurophilosophy (Cambridge, MA: The MIT Press).
Churchland, P.S. (1996), ‘The hornswoggle problem’, Journal of Consciousness Studies,3(5–6),
pp. 402–8.
Churchland, P.S. & Ramachandran, V.S. (1993), ‘Filling in: Why Dennett is wrong’, in Dennett and His
Critics: Demystifying Mind, ed. B. Dahlbom (Oxford: Blackwell Scientific Press).
Churchland, P.S., Ramachandran, V.S. & Sejnowski, T.J. (1994), ‘A critique of pure vision’, in Large-
scale Neuronal Theories of the Brain, ed. C. Koch & J.L. Davis (Cambridge, MA: The MIT Press).
Cobb S., Ramachandran, V.S. & Hirstein, W. (in preparation), ‘Evoked potentials during synesthesia’.
Cohen, M.S., Kosslyn, S.M., Breiter, H.C. et al. (1996), ‘Changes in cortical activity during mental rota-
tion. A mapping study using functional MRI’, Brain,119, pp. 89–100.
Crick, F. (1994), The Astonishing Hypothesis: The Scientific Search for the Soul (New York: Simon and
Schuster).
Crick, F. (1996), ‘Visual perception: rivalry and consciousness’, Nature,379, pp. 485–6.
Crick, F. & Koch, C. (1992), ‘The problem of consciousness’, Scientific American,267, pp. 152–9.
Damasio, A.C. (1994), Descartes’ Error (New York: Putnam).
Dennett, D.C. (1978), Brainstorms (Cambridge, MA: The MIT Press).
Dennett, D.C. (1991), Consciousness Explained (Boston, MA: Little, Brown and Co.).
Descartes, R. (1986), Meditations on First Philosophy, trans. J. Cottingham (Cambridge: Cambridge
University Press).
Devinsky, O., Feldmann, E., Burrowes, K. & Broomfield, E. (1989), ‘Autoscopic phenomena with sei-
zures’, Archives of Neurology,46, pp. 1080–8.
Devinsky, O., Morrell, MJ, Vogt, BA. (1995) ‘Contribution of anterior cingulate cortex to behavior’,
Brain,118, pp. 279–306.
Durgin, F.H., Tripathy, S.P. & Levi, D.M. (1995), ‘On the filling in of the visual blind spot: some rules of
thumb’, Perception,24, pp. 827–40.
Edelman, G. (1989), The Remembered Present (New York: Basic Books).
Farah, M.J. (1989), ‘The neural basis of mental imagery’, Trends in Neurosciences,10, pp. 395–9.
Fiorini, M., Rosa, M.G.P., Gattass, R. & Rocha-Miranda, C.E. (1992), ‘Dynamic surrounds of receptive
fields in primate striate cortex: A physiological basis’, Proceedings of the National Academy of Sci-
ence 89, pp. 8547–51.
Fodor, J.A. (1975), The Language of Thought (Cambridge, MA: Harvard University Press).
Frith, C.D. & Dolan, R.J. (1997), ‘Abnormal beliefs: Delusions and memory’, Paper presented at the
May, 1997, Harvard Conference on Memory and Belief.
Gattass, R., Fiorini, M., Rosa, M.P.G, Pinon, M.C.F., Sousa, A.P.B., Soares, J.G.M. (1992), ‘Visual
responses outside the classical receptive field RF in primate striate cortex: a possible correlate of percep-
tual completion’, in The Visual System from Genesis to Maturity, ed. R. Lent (Boston, MA: Birk-
hauser).
Gazzaniga, M.S. (1993), ‘Brain mechanisms and conscious experience’, Ciba Foundation Symposium,
174, pp. 247–57.
Gloor, P., Olivier, A., Quesney, L.F., Andermann, F., Horowitz, S. (1982), ‘The role of the limbic system
in experiential phenomena of temporal lobe epilepsy’, Annals of Neurology,12, pp. 129–43.
Gloor, P. (1992), ‘Amygdala and temporal lobe epilepsy’, in The Amygdala: Neurobiological Aspects of
Emotion, Memory and Mental Dysfunction, ed J.P. Aggleton (New York: Wiley-Liss).
Goldberg, G., Mayer, N. & Toglis, J.U. (1981), ‘Medial frontal cortex and the alien hand sign’, Archives
of Neurology,38, pp. 683–6.
Grady, D. (1993), ‘The vision thing: Mainly in the brain’, Discover, June, pp. 57–66.
Grush, R. & Churchland, P.S. (1995), ‘Gaps in Penrose’s toilings’, Journal of Consciousness Studies,
2(1), pp. 10–29.
Halgren, E. (1992), ‘Emotional neurophysiology of the amygdala within the context of human cognition’,
in The Amygdala: Neurobiological Aspects of Emotion, Memory and Mental Dysfunction, ed J.P.
Aggleton (New York: Wiley-Liss).
Hirstein, W. & Ramachandran, V.S. (1997), ‘Capgras syndrome: A novel probe for understanding the
neural representation of the identity and familiarity of persons’, Proceedings of the Royal Society of
London,264, pp. 437–44.
456 V.S. RAMACHANDRAN AND W. HIRSTEIN
112
112
Horgan, J. (1994), ‘Can science explain consciousness?’, Scientific American,271, pp. 88–94.
Humphrey, N. (1993), A History of the Mind (London: Vintage).
Jackendoff, R. (1987), Consciousness and the Computational Mind (Cambridge, MA: The MIT Press).
Jackson, F. (1986), ‘What Mary did not know’, Journal of Philosophy,83, pp. 291–5.
Kanizsa, G. (1979), Organization In Vision (New York: Praeger).
Kinsbourne, M. (1995), ‘The intralaminar thalamic nucleii’, Consciousness and Cognition,4,
pp. 167–71.
Kripke, S.A. (1980), Naming and Necessity (Cambridge, MA: Harvard University Press).
Lackner, J.R. (1988), ‘Some proprioceptive influences on perceptual representations’, Brain,111, pp. 281–97.
LeDoux, J.E. (1992), ‘Emotion and the amygdala’, in The Amygdala: Neurobiological Aspects of Emo-
tion, Memory and Mental Dysfunction, ed J.P. Aggleton (New York: Wiley-Liss).
Lilly, R., Cummings, J.L., Benson, D.F. & Frankel, M. (1983), ‘The human Kluver-Bucy syndrome’,
Neurology,33, pp. 1141–5.
Llinás, R.R. & Paré, D. (1991), ‘Of dreaming and wakefulness’, Neuroscience,44, pp. 521–35.
MacLean, P.D. (1990), The Triune Brain in Evolution (New York: Plenum Press).
MacKay, D.M. (1969), Information, Mechanism and Meaning (Cambridge, MA: The MIT Press).
Marr, D. (1982), Vision (San Francisco: Freeman).
Medawar, P. (1969), Induction and Intuition in Scientific Thought (London: Methuen).
Milner, A.D. & Goodale, M.A. (1995), The Visual Brain In Action (Oxford: Oxford University Press).
Nagel, T. (1974), ‘What is it like to be a bat?’, Philosophical Review,83, pp. 435–50.
Nash, M. (1995), ‘Glimpses of the mind’, Time, pp. 44–52.
Nielson, J.M. & Jacobs, L.L. (1951), ‘Bilateral lesions of the anterior cingulate gyri’, Bulletin of the Los
Angeles Neurological Society,16, pp. 231–4.
Paré, D. & Llinás, R. (1995), ‘Conscious and preconscious processes as seen from the standpoint of
sleep-waking cycle neurophysiology’, Neuropsychologia,33, pp. 1155–68.
Penfield, W.P. & Jasper, H. (1954), Epilepsy and the Functional Anatomy of the Human Brain (Boston,
MA: Little, Brown & Co.).
Penfield, W.P. & Perot, P. (1963), ‘The brain’s record of auditory and visual experience: a final summary
and discussion’, Brain,86, pp. 595–696.
Penrose, R. (1994), Shadows of the Mind (Oxford: Oxford University Press).
Plum, F. & Posner, J.B. (1980), The Diagnosis of Stupor and Coma (Philadelphia: F.A. Davis and Co.).
Posner, M.I. & Raichle, M.E. (1994), Frames of Mind (New York: Scientific American Library).
Purpura K.P. & Schiff, N.D. (1997), ‘The thalamic intralaminar nuclei: a role in visual awareness’, The
Neuroscientist,3, pp. 8–15.
Ramachandran, V.S. (1992), ‘Blind spots’, Scientific American,266, pp. 85–91.
Ramachandran, V.S. (1993), ‘Filling in gaps in logic: Some comments on Dennett’, Consciousness and
Cognition,2, pp. 165–8.
Ramachandran, V.S. (1995a), ‘Filling in gaps in logic: Reply to Durgin et al.‘, Perception,24, pp. 41-845.
Ramachandran, V.S. (1995b), ‘Perceptual correlates of neural plasticity’, in Early Vision and Beyond, ed.
T.V. Papathomas, C. Chubb, A. Gorea and E. Kowler (Cambridge, MA: The MIT Press).
Ramachandran, V.S. (1995c), ‘Anosognosia in parietal lobe syndrome’, Consciousness and Cognition,4,
pp. 22–51.
Ramachandran, V.S. & Gregory, R.L. (1991), ‘Perceptual filling in of artificially induced scotomas in
human vision’, Nature,350, pp. 699–702.
Ramachandran, V.S., Rogers-Ramachandran, D. & Cobb, S. (1995), ‘Touching the phantom limb’,
Nature,377, pp. 489–90.
Ramachandran, V.S. & Rogers-Ramachandran, D. (1996), ‘Synaesthesia in phantom limbs induced with
mirrors’, Proceedings of the Royal Society of London,263, pp. 377–86.
Ramachandran, V.S, Altschuler, E.L. & Hillyer, S. (1997), ‘Mirror agnosia’, Proceedings of the Royal
Society of London,264, pp. 645–7.
Ramachandran, V.S., Hirstein, W. & Stoddard, R. (in preparation), ‘The phantom head: An illusion of
decapitation’.
Schenk, L. & Bear, D. (1981), ‘Multiple personality and related dissociative phenomena in patients with
temporal lobe epilepsy’, American Journal of Psychiatry,138, pp. 1311–16.
Searle, John R. (1980), ‘Minds, brains, and programs’, Behavioral and Brain Sciences,3, pp. 417–58.
Searle, John R. (1992), The Rediscovery of the Mind (Cambridge, MA: The MIT Press).
Strauss, E., Risser, A. & Jones, M.W. (1982), ‘Fear responses in patients with epilepsy’, Archives of Neu-
rology,39, pp. 626–30.
Sutherland, N.S. (1989), The International Dictionary of Psychology (New York: Continuum).
Tovee, M.J., Rolls, E.T. & Ramachandran, V.S. (1996), ‘Rapid visual learning in neurones of the primate
temporal visual cortex’, Neuroreport,7, pp. 2757–60.
Tulving, E. (1983), Elements of Episodic Memory (Oxford: Clarendon Press).
Trimble, M.R. (1992), ‘The Gastaut-Geschwind syndrome’, in The Temporal Lobes and the Limbic Sys-
tem, ed. M.R. Trimble and T.G. Bolwig (Petersfield: Wrightson Biomedical Publishing Ltd.).
BIOLOGICAL FUNCTIONS OF CONSCIOUSNESS, QUALIA & SELF 457
113
113
van Essen, D.C. (1979), ‘Visual areas of the mammalian cerebral cortex’, Annual Reviews of Neurosci-
ence,2, pp. 227–63.
Waxman, S.G. & Geschwind, N. (1975), ‘The interictal behavior syndrome of temporal lobe epilepsy’,
Archives of General Psychiatry,32, pp. 1580-6.
Zeki, S.M. (1978), ‘Functional specialisation in the visual cortex of the rhesus monkey’, Nature,274,
pp. 423–8.
Zeki, S.M. (1993), A Vision of the Brain (Oxford: Oxford University Press).
458 V.S. RAMACHANDRAN AND W. HIRSTEIN
114
114
Ivar Hagendoorn
Some Speculative Hypotheses about
the Nature and Perception of
Dance and Choreography
I: Introduction
Ever since I first saw a dance performance I have wondered why it is that I am some-
times fascinated and touched by some people moving about on a stage, while at
other times it leaves me completely indifferent. I will argue that an answer to this
question has to be searched for in the way sensory stimuli are processed in the brain.
After all, all our actions, perceptions and feelings are mediated and controlled by the
brain. The thoughts and feelings evoked by a dance performance are no exception
and thus they too have a neural substrate in the brain.
In music and the visual arts what has been called ‘neuroaesthetics’ (Zeki,
2001a) and ‘neuromusicology’ has already yielded some interesting insights. As
one study showed, the aesthetic appeal of Mondrian’s paintings can be related to
certain psychophysical properties of the visual system (Latto et al., 2000). Ear-
lier studies had found that the perception of oblique lines is slightly inferior to
the perception of horizontal and vertical lines. Making clever use of the fact that
some of Mondrian’s paintings have an oblique frame, the authors showed that
people also prefer horizontal and vertical to oblique lines. Findings like this sug-
gest that we have an aesthetic preference for those stimuli that are closely tuned
to the respective sensory areas in the brain.
The present article draws an itinerary through various brain structures and
shows how these may combine to ultimately give rise to the sensations we expe-
rience when watching a dance performance. Since watching dance is essentially
a visual experience the present analysis concentrates on visual processing. This
is not to deny that music is an integral part of most dance performances or that
movements produce noise, which may influence visual processing and in the
absence of a visual component elicit visual images. But, to state the obvious, if
the stage lights go out the audience’s experience of the dancers’ movements will
Journal of Consciousness Studies,11, No. 3–4, 2004, pp. 79–110
Correspondence:Ivar Hagendoorn, Marnixstraat 18-A, 2518 PZ Den Haag, The Netherlands,
Email: ivar@ivarhagendoorn.com
115
115
be impaired. I should emphasize that the itinerary chosen here is only one of
many routes that participate in the processing of dance, although I believe it to be
a main route. At the start of this itinerary lies the observation that neural process-
ing delays interfere with both perception and action. This problem is at once
illustrated and brought to the fore in both the perception and practice of dance.
Several authors have proposed in various forms that the brain compensates for
these delays by creating predictions of forthcoming sensory and motor events
(e.g. Berthoz, 2000; Kawato et al., 1987; Wolpert & Flanagan, 2001; Engel et
al., 2001). Based on these considerations I will advance two hypotheses. I will
argue that the deviation from and correspondence between the actual motion tra-
jectory of a moving object and the trajectory as predicted by the brain of the
observer, gives rise to two distinct emotional responses, analogous to the euphoria
and frustration of catching or missing a ball. Through their sequential interplay
these responses may reinforce each other to give rise to the feelings one can experi-
ence when watching dance. As a corollary I will argue that in forming a prediction
of a moving object’s motion trajectory the brain engages in a form of motor imag-
ery which, through a different route, may contribute to a state of arousal.
Much of the present article is devoted to a discussion of the brain regions
involved in the above processes (Figure 1). The unsuspecting reader should
therefore be warned of some tough neuroscience ahead. (Some readers may wish
to jump straight to section VIII where I present my two main hypotheses and then
read back to how I got there). I will argue that mirror neurons, which become
active both when a movement is perceived and performed, may constitute a neu-
ral bridge between action and perception. Interestingly, as I will show, one of the
brain regions where mirror neurons have been found has also been associated
with perceptual anticipation. I will then relate these findings to the literature on
prediction and reward and argue that brain regions associated with prediction
errors may also account for at least some of the emotions we experience when
watching dance.
Different people will have a different understanding of the word ‘dance’,
some may recall a ballet or a musical they once saw, while others may think of
80 I.G. HAGENDOORN
Figure 1.
Schematic representation
of the main brain regions
mentioned in the article
and their approximate
locations.
116
116
the movements they and others perform at a party. By dance or ‘dance perfor-
mance’, I will here refer to a sequence of movements, not necessarily choreo-
graphed, of any length, from two seconds to two hours, whereby the goal of the
movement is the movement itself. This is not so much a definition of dance, or
dance as an art form, but an operational delimitation of the question I will be
addressing here: why is it that it can be interesting to watch someone ‘just danc-
ing around’ or, to avoid circular reasoning, going through a series of movements
without any apparent goal, other than ‘just’ performing the movements? And
what is it that choreographers do when they compose such a series of move-
ments, with no other purpose than to be performed and seen? What goes on in
their mind when they adjust a position or movement sequence? What are the neu-
ral processes that guide and constitute aesthetic judgment? My primary refer-
ences here are works by choreographers such as George Balanchine, William
Forsythe, Merce Cunningham and Jiri Kylián, often referred to as ‘abstract’
dance. However, movement sequences from martial arts when rehearsed ‘as
such’ also fit the present description, so do Indian dances, Japanese butoh and
various other dance traditions.
Having outlined how a movement sequence may bring about the feelings it
does in certain dance performances, I will argue that choreographers like all other
artists when creating a work, are implicitly led by the brain mechanisms underlying
sensory experience and emotion. By making explicit some of these implicit consid-
erations they may eventually be put to creative use, a proposition I will illustrate
with some examples from my own choreographic work. I would like to emphasize
from the outset that the ideas expressed here are tentative and not uncontroversial
(e.g., Zeki, 2001b; Ione, 2001). However, it is only with their propagation and by
inviting feedback from critics that they may be further developed.
II: From the Retina to the Brain
What we see is light and light reflected from surfaces. The difference between
light hitting the eye or an arm, or a wall, is that inside the eye are light-sensitive
receptors that transform the energy carried by the light into electrical signals.
After some pre-processing these electrical signals are relayed along separate
pathways to various specialized areas in the brain, where they are either used to
form a visual representation of the stimulus or immediately translated into an
action such narrowing the pupil.
Of the various pathways emanating from the retina only two are directly
involved in processing visual information for perception. One conveys informa-
tion contributing primarily to the perception of movement, while the other is
associated with the processing of colour and shape. Before reaching the visual
areas of the brain, both pathways pass through the thalamus, the brain’s central
relay centre. From thereon the visual information is relayed to the primary visual
cortex, where, after some intermediate processing, it is once more separated into
different paths, each leading to a more specialized area. In all there are over 30
regions performing such tasks as determining colour, shape, solidity, size and
motion of whatever hits the eye. It is currently believed that motion is processed
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 81
117
117
in two areas, one of which, the middle temporal gyrus, often referred to as MT or
V5, is involved in processing an object’s speed and direction, while the other, the
medial superior temporal area or MST, is specialized for detecting its dynamical
properties, like rotation and tilting.
It seems likely that all connections between these higher visual processing
areas are reciprocal. What’s more, information may re-enter a given area after it
has been processed in any number of other areas (Martin, 2002). It is difficult to
see how the different attributes of a scene that are analysed in specialized areas
would otherwise be integrated into a visual image, a feature of the visual system
often referred to as the ‘binding problem’, a ‘problem’ since its workings are still
largely unknown (Robertson, 2003). With respect to motion perception the bind-
ing (or maybe we should say ‘unbinding’) problem, comes down to determining
what is moving and what is stationary, and distinguishing between the motion of
different parts of a single object and the motion of different objects, something
choreographers like to play with (Figure 2). Already we catch a glimpse of how
aspects of the visual system can be employed artistically. By keeping the back-
ground and some attributes of an object constant, a choreographer can create a
setting which, in terms of brain processing, leaves more resources available for
processing one salient feature — movement, for instance.
Not all visual information passes from the thalamus to the primary visual cor-
tex. One pathway leads straight to MT, one of the areas specialized in visual
motion processing. This may be why some people with damage to the primary
visual cortex are nevertheless able to perceive movement, a condition called
blindsight (e.g., Schoenfeld et al., 2002). Another pathway projects from the
thalamus to the amygdala, a brain structure associated with emotional behaviour,
in particular in relation to danger (LeDoux, 1996). The direct pathway from the
thalamus to the amygdala ensures that the brain can already begin to respond
before an object has been identified, for instance by modulating the processing in
the visual cortex, with which it is reciprocally connected, or by initiating
82 I.G. HAGENDOORN
Figure 2.
Which limb belongs to
which body? Jiri
Kylian, As i f nev er
been (1992). Dancers:
Nancy Euver ink and
Patrick Delcroix.
Photo:
© Dirk Buwalda.
118
118
withdrawal. Even though theatres tend to be relatively safe environments, by
emulating the perceptual characteristics of potentially dangerous events, the
brain will respond in much the same way as in the case of a real threat, as every
movie director knows. A possible explanation for this is that from an evolution-
ary point of view it is better to retreat ten times too often than once too few. A
related explanation is that in these circumstances the emotional system overrules
cognition. We are carried away even though we know ‘it’s only a movie’. The
opposite, by the way, may also occur: we may cognitively value something we
don’t enjoy, for instance an avant-garde dance performance with little or no
movement.
Interestingly, the amygdala also appears to play a role in attributing emotions to
movements. When watching animation movies featuring simple geometric figures
like circles, squares and triangles, people tend to attribute emotions such as joy,
anger and frustration to the figures based on the nature of their movements (Heider
& Simmel, 1944). Patients with damage to the amygdala, however, fail to assign
emotions to the movements of such figures (Heberlein et al., 1998). It would there-
fore be interesting to investigate how these patients respond to dance, especially if
the dancers were to wear single-coloured costumes and masks or hoods covering
both head and face so as to create a purely graphic display.
III: Motion Anticipation and Smooth Pursuit
Having hit the retina it takes between 50 and 100 milliseconds before the infor-
mation carried by a light particle reaches the visual cortex. Some simple arithme-
tic shows that a car driving at 100 km/h will have covered an additional 2 to 3
metres by the time the light activates the appropriate regions in the brain. To
make up for this delay it has been conjectured that the brain somehow forms a
prediction of the path of a moving object (Nijhawan, 1994). According to this
view the brain forms an internal simulation of the trajectory covered so far, on
the basis of which the object’s movement is extrapolated into the future. Such an
extrapolation would ensure that its perceived position coincides with its actual
position.
Perceptual anticipation and prediction are likely to involve both low- and
high-level brain processes. It has been proposed that a rudimentary prediction in
the form of a forward shift of the image occurs as early as the retina (Nijhawan,
1997; Berry et al., 1999; for a review, see Nijhawan, 2002). At the level of object
representations, learned characteristics of the behaviour of target objects (e.g.,
the bouncing of a ball) bias the processing of its speed and direction of motion
(the second bounce is lower than the first). Getting to the roots of this phenome-
non, an intriguing study, which compared the performance of astronauts catch-
ing a ball on earth and under zero gravity, recently suggested that the brain uses
an internal model of gravity-induced acceleration when predicting the trajectory
of a falling ball (McIntyre et al., 2001).
Various other experimental findings support the hypothesis that motion per-
ception is predictive. It has been found that, if a moving target suddenly
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 83
119
119
disappears, its last position is remembered as being slightly ahead of its actual
final position, a phenomenon technically known as representational momentum,
which suggests that the percept is shifted forward (Freyd & Finke, 1984). What’s
more, even still images of an object in motion, such as that of a falling glass,
exhibit ‘representational momentum’, in that they convey information about the
motion implied in the picture (Freyd, 1983). This, of course, is nothing new to
dance audiences. While many dance photos could also have been posed, some
‘capture’ the dance (see Figure 3, as well as Figures 6 and 9). They extend the
movement frozen by the camera forward in time. Interestingly it has been dem-
onstrated that watching still photographs with implied motion yields increased
neural activity in MT/MST, the part of the visual cortex involved in the process-
ing of visual motion (Senior et al., 2000; Kourtzi & Kanwisher, 2000).
Perhaps some of the most compelling evidence for the predictive nature of
motion perception has come from the study of fast ball sports like tennis, base-
ball and cricket. In baseball, outfielders run to where they expect to be able to
catch a fly ball, which implies that the brain forms a prediction of the ball’s tra-
jectory (McBeath et al., 1996; McLeod et al., 2001). Land and McLeod (2000)
recorded the eye movements of batsmen in cricket as they prepared to hit an
approaching ball. They found that the batsmen’s eyes monitored the ball shortly
after its release, then made a predictive saccade to where they expected it to hit
the ground, waited for it to bounce and then tracked its trajectory for 100–200 ms
afterward. As the authors argue ‘information provided by these fixations may
allow precise prediction of the ball’s timing and placement’. If this is true for
moving balls, why wouldn’t it also be the case for human movements such as
dance?
The study by Land and McLeod (2000) demonstrates that the central nervous
system employs two types of eye movements to track a moving target: saccades
84 I.G. HAGENDOORN
Figure 3. Implied motion. A good dance photo ‘captures’ the dance. The movement appears to
continue in the direction of motion. Indeed, photos such as these yield increased activity in
MT/MST, the part of the visual cortex involved in the processing of visual motion (Senior et al.,
2000; Kourtzi & Kanwisher, 2000). William Forsythe, As a Garden in this Setting (1992). Dancer:
Regina van Berkel (l), Tony Rizzi (r). Photo: © Dominik Mentzos.
120
120
and smooth pursuit. Saccades direct the fovea, the high-acuity region at the cen-
tre of the retina, towards a target in discrete jumps. Smooth pursuit eye move-
ments stabilize the image of the object on the retina by smoothly rotating the eyes
in congruence with the motion of the target. Sometimes, however, moving the
eyes alone is not enough and the head, and if necessary other parts of the body,
are integrated into the smooth pursuit movement. In tennis, the heads of the
umpire and the public can often be seen to move from side to side as they track
the ball. What’s more, when tracking an object with binoculars or a camera, the
eyes are fixed and the body adjusts to keep the object within view. I would there-
fore like to suggest that the mechanisms for tracking a moving target — eyes,
head or attention — form a continuum and as such share a common neural net-
work, which predicts the trajectory of the moving target.
While many studies have investigated the properties of neurons in visual
motion area MT and the neural mechanisms of smooth pursuit eye movements
and saccades, surprisingly few have analysed the functional neuroanatomy of
visual motion anticipation. Ando (2002) recently reported the results of a func-
tional neuroimaging experiment using fMRI, in which participants had to predict
the motion trajectory of a virtual three-dimensional object. The data revealed
activity in the intraparietal sulcus, the inferior and superior parietal lobules (BA
40 and BA 7), the dorsal premotor cortex (BA 6), the inferior frontal gyrus (BA
44, Broca’s area) and the lateral cerebellum. These findings are in congruence
with recent experiments by Schubotz and Von Cramon (2001; 2002a; 2002b).
They found increased activity in the right ventrolateral premotor cortex (BA 6)
and the right intraparietal sulcus, as subjects predicted the last in a series of 12
sequentially displayed circles of different size, which created the illusion of reg-
ularly pulsing motion (Schubotz & Von Cramon, 2002b). What is perhaps most
interesting about these results is the activation of premotor regions, as in all
experiments the task was purely perceptual.
Catching a ball, hitting a moving object or tracking a target with a camera, all
require a form of sensorimotor integration. Visual information about the object’s
motion trajectory has to be instantaneously transformed into movement of the
body, whether the eye, finger, hand or arm. The results by Ando (2002) and
Schubotz and Von Cramon (2002b) indicate that even in the absence of a motor
task, attending to and predicting the trajectory of a moving target activates
premotor regions. On a speculative note Schubotz and Von Cramon suggest that
when we try to predict how a target will move, the motor system generates a ‘blue-
print’ of the observed motion that allows potential sensorimotor integration. In the
absence of any motor requirement, this blueprint appears to be not a by-product of
motor planning, but rather the basis for target motion prediction (Schubotz and Von
Cramon, 2002b).
What’s more, since the lateral premotor cortex is also activated when predicting
a colour or pattern transition or a change in auditory pitch, Schubotz and Von
Cramon (2002a) suggest that the lateral premotor cortex is involved in the pre-
diction of any kind of sequential event, with human movement as a special case.
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 85
121
121
Additional evidence for the involvement of motor areas in perceptual anticipa-
tion comes from the study of handwriting movements. It has been shown that
when writing an ‘l’ followed by another ‘l’, the first ‘l’ is written down faster
than when it is followed by an ‘n’ (Orliaguet et al., 1997). Further experiments
demonstrated that, when watching a dot as it traces the curve of an ‘l’, observers
are able to use information about its kinematics to predict the upcoming letter, ‘l’
or ‘n’ (Kandel et al., 2000a; 2000b). Chaminade et al. (2001) used positron emis-
sion tomography (PET) to measure brain activity as observers predicted the con-
tinuation of the motion trajectory of a dot produced by mechanical, pointing and
writing movements. All three conditions were associated with a common neural
network comprising the orbitofrontal and right frontal cortex. However, unlike
Schubotz and Von Cramon (2002b), activation of the premotor cortex and the
right intraparietal sulcus was found only when the trajectory had been produced
by a pointing movement. The authors also report increased activity in the supe-
rior parietal lobule and Broca’s area (BA 44) during the anticipation of writing
movements, two of the areas that were also activated in the fMRI study by Ando
(2002). Some caution in interpreting the findings by Chaminade et al. (2001) is
warranted though, since the baseline condition required subjects to indicate
whether they expected the dot to move up or down. This, however, is itself a pre-
diction task and can therefore not be properly used to test for perceptual anticipa-
tion. In summary, while the results are not unequivocal they do partially
reinforce each other, suggesting that (pre-)motor areas are involved in predicting
visual motion, which obviously would include the movements of a dancer or
group of dancers.
IV: Apparent Motion
It frequently happens that a moving object is temporarily occluded from view, an
arm which briefly moves behind the body, a car entering a tunnel or a dancer who
disappears behind one of the other dancers. In principle almost anything can hap-
pen while we are unable to see the object, but in practice most objects continue
along their track and we are able to accurately predict where and when it will
reappear.1Indeed, we tend to be surprised if the object does not reappear or at a
different location. Conversely, once the object has reappeared the brain is able to
infer the motion trajectory between the points where it vanished and resurfaced.
The latter is a specific example of what is known as ‘apparent motion’, since we
did not really see the object move, we merely saw it dis- and reappear.
The term apparent motion is commonly used to refer to the illusory perception
of motion from the rapid sequential display of static images as in a film or ‘mo-
tion picture’. Consider a prototypical film consisting of two frames, one with two
vertically aligned dots on the left, the other with two vertically aligned dots on
the right. If the two frames are rapidly interchanged, the dots are perceived as
86 I.G. HAGENDOORN
[1] A choreographer aware of this phenomenon could manipulate dancers, movements and stage props
such that part of the audience sees what is occluded from the rest of the audience as in William
Forsythe’s Enemy in the Figure (1987), which features a wavy-shaped wooden panel positioned diag-
onally on the middle of the stage.
122
122
jumping from left to right and back. However, this is only one of several logical
possibilities, the upper and lower dots could in principle have swapped positions.
This classic example illustrates that given a point A and B, the visual system
selects the shortest path to connect both points, that is, to describe the movement
needed to go from A to B (e.g., Dawson & Pylyshyn, 1988). However, as shown
by Shiffrar and Freyd (1990; 1993), if A and B are body positions, perception fol-
lows an anatomically feasible path, even though a physically impossible route
may be shorter (Figure 4).
A recent PET study in which subjects were briefly presented with two body
positions and subsequently had to choose between a direct impossible and an
indirect possible connecting movement, yielded selective activation of regions
in the primary motor and superior parietal cortex, but only if the connecting
movement was biomechanically possible (Stevens et al., 2000). On the basis of
these findings the authors suggest that
we might expect an absence of motor executive activations during the visual per-
ception of actions that an observer interprets as beyond his/her motor capabilities,
e.g., a technically challenging ballet movement. In such cases, the impossibility of
completing the action is determined within the context of the observer’s own motor
experience rather than in terms of the general movement limitations of the human
body [emphasis mine].
This hypothesis could be tested empirically by first showing two positions
unlikely to be the beginning and end of a continuous movement and subse-
quently the movement that connects them.
Stevens et al. (2000) suggest that, when connecting two body positions, the
brain engages in a form of motor imagery to create the motion percept. Motor
imagery refers to the mental performance of a movement, or, more formally ‘a
dynamic state during which the representation of a given motor act is internally
rehearsed within working memory without any overt motor output’ (Decety &
Grèzes, 1999). It is the experience of seeing and feeling oneself executing a
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 87
Figure 4. Apparent motion of the human body. When alternately shown two body positions the
brain chooses a biomechanically feasible path to connect them. The example used in most experi-
mental studies shows a hand behind and in front of a knee, the idea being that the hand cannot move
through the leg. If the hypothesis is correct in the above example the brain will simulate a move-
ment of the head underneath the arm, a movement most people are unlikely to have previously per-
formed. Dancer: Ester Natzijl. Photo: © Ivar Hagendoorn.
123
123
movement, a kinaesthetic feeling of speed, effort and changing body configura-
tion. As Jeannerod (1994) explains ‘visual images are experienced by the self in
the same way as a spectator who watches a scene, motor images are experienced
from within as the result of a first-person process where the self feels like an
actor rather than a spectator’. Thus, according to this view, if within a movement
sequence part of the body is temporarily occluded, the brain will either extrapo-
late the movement from the last visible position or interpolate between the posi-
tions before and after the occlusion, by covertly performing the movement itself.
V: Biological Motion
Most of the experimental studies discussed so far dealt with the movement of
abstract stimuli like dots or virtual balls. If we want to learn more about dance we
will have to consider movements of the (whole) human body. As it turns out
there is a large body of evidence which shows that humans have a special ability
for recognizing what has become known as biological motion, the visual motion
patterns of humans and animals. In the early 1970s Gunnar Johansson developed
a now classic technique whereby an actor was filmed as he moved through a
darkened room with small light bulbs attached to the head and key joints such as
the shoulders, elbows, wrists, hips, knees and ankles. When the actor was sitting
in a chair people watching the film reported seeing nothing but a random collec-
tion of lights. But as soon as the actor started moving they instantly identified the
pattern as that of a moving person. In a second experiment a dancing couple was
filmed under the same conditions and again observers had no difficulty in identi-
fying the moving pattern of lights as that of a dancing couple (Johansson, 1973).
This technique has since been replicated in a variety of experimental settings.2
The lights, or their more contemporary motion capture equivalents, were placed
on other parts of the body, actors were instructed to perform a range of activities,
from hammering to greeting to climbing a staircase, and the display as a whole
has been turned upside-down or masked with a cloud of random noise (for a
review, see Pinto & Shiffrar, 1999). One experiment demonstrated that people
are able to recognize acquaintances and even themselves from a point-light dis-
play of their movements. In fact in this particular experiment the probability of
correct self-recognition was even higher than all other probabilities (Beardworth
& Bukner, 1981). The latter is particularly striking since apart from dancers,
most people don’t often see themselves moving. Not only does this rule out the
possibility that recognition was due to perceptual learning, as might be suspected
if the phenomenon only showed up in the perception of other persons, it also sug-
gests once more a possible link between motor and perceptual processes.
Another experiment showed that people are able to extract an emotional state
from motion characteristics alone (Dittrich et al., 1996). For this experiment the
88 I.G. HAGENDOORN
[2] Given the number of experimental studies, it may surprise that only recently, some 25 years after the
original experiments, the potential of this technique has been explored in a ballet (video projected on
a scrim covering the full stage height as part of the set design), Merce Cunningham, Biped (1999), in
collaboration with multimedia artists Paul Kaiser and Shelley Eshkar.
124
124
researchers asked two dancers to portray fear, anger, grief, joy, surprise and dis-
gust. The movements were recorded as such and using a point-light display. Par-
ticipants in the study then had to judge which emotion was being portrayed.
Interestingly they got 88% right from the full display and 63% from the
point-light display.
In recent years various neuroimaging studies have examined the brain regions
associated with the recognition of biological motion from point-light displays
(Bonda et al., 1996; Grossman et al., 2000; 2002; Vaina et al., 2001; Grezès et
al., 2001; Servos et al., 2002). A common area found in all studies is a region on
the (posterior) superior temporal sulcus (STS). In addition, perhaps not unex-
pectedly, Grossman et al. (2000), Vaina et al. (2001) and Grezès et al. (2001)
report activity in MT/MST. Other areas that were found to be significantly acti-
vated were the cerebellum (Grossman et al., 2000; Vaina et al., 2001), the lin-
gual gyrus (Servos et al., 2002), the ventral premotor cortex (Grezès et al., 2001)
and the amygdala (Bonda et al., 1996). The activation in STS is consistent with
neurophysiological studies in monkeys (Oram & Perrett, 1994). Using single
cell recordings it was shown that some neurons in the superior temporal cortex
are selectively activated by arm movements and the direction of walking. The
posterior STS therefore appears to be a key area for the recognition of biological
motion.
Neurological case studies provide further evidence for this hypothesis.
Howard et al. (1996) and Vaina et al. (1990) report patients who, because of a
lesion in the visual motion processing areas MT/MST, are practically motion
blind, but are still able to perceive biological motion. This led Beintema and
Lappe (2002) to re-examine the stimuli used in point-light studies. They
observed that the standard point-light displays exhibit what they call ‘local
image motion’: each individual dot, for instance the dot tied to the right elbow or
the left ankle, traces a stationary trajectory through space and thus not only con-
tains motion information but also information about its location on the (implied)
body. They therefore developed an alternative technique in which the location of
the light points changes from frame to frame and position and motion informa-
tion are dissociated (e.g., the point that was on the elbow in the first frame moves
somewhere between elbow and shoulder in the second frame, then changes again
in the third frame, etc.). Even though Beintema and Lappe’s ‘sequential position
walker’ as they call it, is somewhat more difficult to recognize, observers’ per-
formance is comparable with standard point-light displays. Based on these find-
ings they suggest that the perception of biological motion relies on the sequential
analysis of body postures.
Point-light displays were originally intended to show how little information is
needed for the human brain to recognize human motion. There is a danger that
explaining this phenomenon, which is essentially a laboratory condition,
becomes an end in itself. Real-life motion stimuli are often imperfect, due to dif-
ferential lighting conditions and partial occlusion, but as the study by Beintema
and Lappe (2002) shows, standard point-light displays may be ‘perfect in their
imperfection’. It may surprise that only few neuroimaging studies have been
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 89
125
125
performed using real human movements as stimuli, rather than point-light dis-
plays. Decety et al. (1997) and Grèzes et al. (1998) conducted a PET study using
video filmed pantomimes of opening a bottle, hammering a nail, sewing a button
or turning the pages of a book as stimuli, as well as movements from American
Sign Language. Participants were instructed to memorize the movements so that
they could either imitate or recognize them after the scanning session or to just
watch the movements. Since the subjects were unfamiliar with American Sign
Language the authors were able to make a distinction between the observation of
meaningful and meaningless movements. In the absence of a goal, observation of
both meaningful and meaningless movements activated the occipital–temporal
junction, which corresponds to MT/V5, the superior occipital gyrus, the middle
temporal gyrus and the inferior parietal lobe. Meaningful movements also acti-
vated the dorsal precentral gyrus (BA 6), the inferior frontal gyrus (BA 44/45)
and the fusiform gyrus, whereas meaningless movements resulted in stronger
activation in the inferior parietal lobe, the superior parietal lobule and the cere-
bellum. If the goal was to imitate, both meaningful and meaningless movements
led to activation in the cerebellum and the occipital–parietal (‘dorsal’) pathway
extending to the premotor cortex, while meaningful movements additionally
activated the supplementary motor area (SMA) and orbitofrontal cortex.
The differences in activated areas between observing point-light figures and
the experiments by Decety et al. (1997) and Grèzes et al. (1998) may be due to the
fact that the former represented full body motion, whereas the latter were confined
to movements of the hand and arm. It would therefore be interesting to replicate
the study by Decety et al. (1997) with the higher spatial resolution that can be
obtained with today’s fMRI and using full body movements rather than move-
ments of the arm and hand alone. In combination with a point-light version of the
same movements, it could then be tested whether real human movements and their
point-light representation do indeed rely on the same neural mechanisms.
Of course it would go too far to straightforwardly extrapolate any of these
findings to the perception of dance. The motion stimuli in the study by Decety et
al. (1997) were very short, lasting for as little as 4 seconds and were restricted to
movements of the hand and arm, with only the upper limbs and trunk being
shown on a computer display. What is perhaps most interesting about these two
studies is that differences between the observation of meaningful and meaning-
less movements on the one hand and cognitive strategy on the other can actually
be detected at the level of neural processing. Translated to dance one may there-
fore speculate about a difference in neural processing already at the level of per-
ception between watching the abstract ballets of for instance George Balanchine
and the expressionist more gestural based dance theatre of Pina Bausch.
VI: Mirror Neurons
Above I have briefly outlined some of the brain regions involved in (biological)
motion perception and perceptual anticipation. Although the evidence is as yet
inconclusive, it appears that motor regions contribute to action perception. What
90 I.G. HAGENDOORN
126
126
is missing is a possible mechanism or a common computational framework link-
ing these scattered reports.
Some years ago neuroscientists discovered a population of neurons in the
premotor cortex of a monkey that discharge both when the monkey performs a
movement and when it observes the same action performed by someone else
(Rizzolatti et al., 1996; Gallese et al., 1996). These so-called ‘mirror neurons’
could therefore provide a neurophysiological bridge between perception and
action. At first sight mirror neurons appear to respond in much the same way as
the neurons in the superior temporal cortex that respond to the sight of a moving
hand, face or body (Oram & Perrett, 1994). What is striking about mirror neu-
rons, though, is that they also fire when the monkey performs a movement.
Therefore they cannot be exclusively visual. A recent study showed that mirror
neurons in the premotor cortex also respond to auditory stimuli, making them not
just bi- but multimodal (Kohler et al., 2002).
At present there is much speculation about the role of mirror neurons in per-
ception and behaviour. Rizzolatti et al. (1996) have suggested a role in the under-
standing of motor events. The brain has an implicit knowledge of the immediate
consequences of its own actions, that is, of the movements it generates, both in
terms of its changing relation to the external world, as in terms of a change in
body state and body configuration. This knowledge is the result of an association
between the representation of a movement and its consequences, in other words,
the movement has a meaning (e.g., ‘grasp’) and this meaning is represented by a
specific cortical activation pattern. Mirror neurons show that this movement
knowledge can be attributed to actions made by others. When an external stimulus
evokes a neural activity similar to that which, when internally generated, represents
a certain action, the meaning of the observed action is recognized because of the
similarity between the two representations, the one internally generated during
action and that evoked by the stimulus (Rizzolatti et al., 1996).
This account, if correct, might explain why people understand not only mime, but
also non-imitative movements in dance, which have a more abstract ‘meaning’.
The back of the hand is more vulnerable than the palm, so leaning on the back
rather than the palm of the hand may exude a sense of ‘vulnerability’. It may also
explain why a dancer balancing in a virtuoso position may inspire awe and be
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 91
Figure 5.
Is he actually? To understand what
is going on the brain virtually per-
forms the movement and body
configuration. Jirí Kylián,
Click–Pause–Silence (2001).
Dancers: Patrick Marin and Stefan
Zeromski.
Photo: © Joris-Jan Bos.
127
127
literally breathtaking: we hold our breath as we internally simulate the movement
(see Figures 5, 6 and 8).
Jeannerod (1994; 1997) has suggested a role for mirror neurons in the learning
of new motor skills. There is now a growing body of literature linking mirror
neurons to imitation learning (Meltzoff & Prinz, 2002; Hurley & Chater, 2004),
which of course is the basis of not only dance education but also much choreog-
raphy. Jeannerod (1994) gives the example of a pupil learning a motor skill such
as playing a musical instrument, but we could also think of a choreographer dem-
onstrating a new movement sequence to a dancer. Although the dancer remains
immobile during the demonstration, he or she must somehow internalise the
movement, that is, he or she must form an image of the movement sequence as it
unfolds. The choreographer in turn will compare the movements of the dancer
with what he himself had in mind and in the words of Jeannerod ‘[will] experi-
ence a strong feeling of what should be done and how’. As Jeannerod continues,
Similar feelings may be experienced by sports addicts watching a football game on
television. They mentally perform the appropriate action to catch the ball (and
indeed, they express frustration when the ball has been missed by the player). The
vividness of the imagined action can induce in the watchers changes in heart and
respiration rates related to the degree of their mental effort (Jeannerod, 1994; 1997).
Although mirror neurons have only been directly demonstrated in monkeys,
there is accumulating evidence that similar cells or a similar mirror system exist
in humans. As one experiment showed, the observation of human movement
facilitates the same muscle groups and motor circuits as when the movements are
executed (Fadiga et al., 1995). Neuroimaging studies also suggest the existence
of a human mirror system. Iacoboni et al. (1999) performed an fMRI experiment
in which participants were instructed to imitate a finger movement, lift a finger
in response to a spatial cue or to just watch either the finger movement or the spa-
tial cue. Activation in the left inferior frontal cortex (BA 44, Broca’s area), the
right anterior parietal cortex and the right parietal operculum was significantly
higher during imitation than in the other tasks. Of particular interest is the activa-
tion in Broca’s area, since this area has been proposed to be the human
homologue of area F5 in the monkey premotor cortex (Rizzolatti & Arbib,
1998), where mirror neurons were first discovered. To directly test the involve-
ment of Broca’s area in imitation Heiser et al. (2003) used transcranial magnetic
stimulation (TMS) to temporarily disrupt processing in the left inferior frontal
cortex as subjects imitated a finger movement or alternatively, performed a fin-
ger movement in response to a visual cue. It was found that while simple key
presses were unaffected, imitation was significantly impaired during the applica-
tion of TMS, thus providing evidence for the involvement of Broca’s area, or at
least the region comprising Broca’s area, in imitation. However, Manthey et al.
(2003) recently reported that the ventrolateral part of the premotor cortex (BA 6)
and not Broca’s area (BA 44) was predominantly activated when subjects
observed 36 short movies of simple goal-directed actions. This is in congruence
with findings by Decety et al. (1997) for the observation of meaningful move-
ments, but at odds with the observation-only condition in, for instance, Iacoboni
92 I.G. HAGENDOORN
128
128
et al. (1999). If Broca’s area is associated with imitation one would expect it to
be activated during all instances of action observation, as indeed in the observa-
tion only task in an experiment similar to Iacoboni et al. (1999) by Carr et al.
(2003), which used facial expressions instead of finger movements. It may there-
fore be too early to draw firm conclusions about the precise role of Broca’s area
in the perception of action or about the location of a brain region subserving the
function of matching action observation and execution.
Subsequent experiments by Iacoboni et al. (2001) also revealed increased
activity in the superior temporal sulcus (STS). While this is consistent with find-
ings that report activation during the perception of point-light displays of human
motion (see references above), interestingly and somewhat unexpectedly activa-
tion was greater during the imitation task. This could of course be due to
increased attention, but Iacoboni (2003) offers an intriguing alternative explana-
tion, suggesting that
the increased STS activity may be due to efferent copies of motor commands origi-
nating from fronto-parietal mirror areas. These efferent copies would allow a pre-
diction of the sensory consequences of the planned imitative action that would be
compared with the description of the observed action provided by STS.
Although Iacoboni et al. (2001) provide some evidence in favour of the latter
hypothesis it may be difficult to show that this is indeed what is happening.
Gallese (2002) also interprets mirror activity in premotor regions in terms of
efference copies of motor commands, but places premotor mirror neurons at the
receiving rather than the sending end, suggesting that they perform a simulation
of the planned movement, allowing for the prediction of its sensory conse-
quences. Interestingly this hypothesis is in line with findings reported above,
that suggest a role for the lateral premotor cortex (BA 6/44) in predicting percep-
tual events (Schubotz & Von Cramon, 2002a; b). This raises the question
whether the same neural mechanism underlies either perceptual anticipation and
imitation or the computations of the lateral premotor cortex.
VII: A Common Representational Framework
Neural processing delays interfere not only with the sensory system, but also
with motor control. As various computational studies have shown, sensorimotor
loops are too slow to allow feedback control of fast coordinated movements
(e.g., Kawato, 1987). To overcome this problem it has been proposed within the
motor control literature that the brain uses what are known as internal models to
calculate a feed-forward motor command from the desired motion trajectory (for
a review, see Kawato, 1999). An internal model in general is a system that mim-
ics the behaviour of a natural process. Internal models come in two types. An
inverse model provides the motor commands necessary to perform a movement.
Aforward model captures the forward or causal relationship between the input
to a system and its output, by predicting the next state of the system, given its
current state. Thus the forward model predicts how the pointer will move if the
mouse is moved. The inverse model estimates how the mouse should be moved
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 93
129
129
so that the pointer moves in the desired direction. In the context of motor control,
forward models could compensate for delays in sensory feedback, anticipate and
cancel out the sensory effects of self-produced movements or transform the
errors between the desired and actual outcome of a movement into the corre-
sponding errors in the motor command (Wolpert, 1997; Wolpert et al., 1995). It
seems reasonable to suppose that, when tracking a rapidly moving target with a
cursor, as controlled by a mouse or joystick — think of a computer game — the
brain also forms a forward model of its motion trajectory. From here it is only a
small step to the hypothesis that the brain forms a forward model of a target’s
motion trajectory regardless of whether it tracks the target with a mouse, the
eyes, a camera or attention, and whether the target is a dot on a screen, a ball, a
limb or a dancer. Indeed the term forward model is just a technical notion for the
predictive function referred to in previous sections.
There is accumulating evidence that the brain does indeed employ internal
models. Above we already encountered one instance in the form of catching a
ball on earth and in outer space (McIntyre et al., 2001). Of relevance to the pres-
ent context is a recent study by Mehta and Schaal (2002). Comparing the output
of various computational models with actual performance of a visuomotor task
requiring subjects to balance a pole on a finger, a task which depends crucially
on visual feedback, they conclude that the data are best described by a forward
model at the sensory processing stage. Given the nature of this task, it seems rea-
sonable to extend these results to the visual tracking of a moving target, whether
with the eyes or head.
In recent years various authors have proposed a role for internal models, more
in particular forward models, in the perception of action and the understanding
of behaviour (Blakemore & Decety, 2001; Jeannerod, 2001; Gallagher &
Jeannerod, 2002; Gallese, 2002; Rizzolatti et al., 2001; Iacoboni, 2003; Wolpert
et al., 2001; 2003). In its most intriguing and also most speculative form, it is
hypothesized that, when watching human movement, the predictions of a for-
ward model are compared, not with sensory feedback from the own body, but
with the next movement of the person performing the movement (Wolpert et al.,
2003). This extends the computational logic of internal models far into the realm
of understanding intentions (Blakemore & Decety, 2001) and social cognition
(Gallese, 2002; Iacoboni, 2003; Wolpert et al., 2003).
To illustrate this hypothesis it may be instructive to go back to the example of
a choreographer demonstrating a movement sequence to a dancer and interpret it
in the context of forward and inverse models. When the choreographer performs
the m o v ement, al o n g w ith the mot o r c ommands th a t i nnervat e t h e
musculoskeletal system, efference copies of the motor commands are fed into a
corresponding forward model, which simulates the sensory consequences of the
movement. These predictions can be compared with visual and proprioceptive
feedback and used to update the movement in real-time or to improve its future
performance. The possible role of forward models in perception becomes most
apparent when we consider what happens the moment the choreographer
watches the dancer imitate the movement sequence. The relevant inverse models
94 I.G. HAGENDOORN
130
130
are now run ‘off-line’, without acting on the musculoskeletal system, and their
output is sent directly as input to the corresponding forward models. The output
of the forward models is now compared with the movements of the dancer and in
case of an error give rise to the ‘feeling of what should be done and how’ referred
to by Jeannerod (1994; 1997).
It should be emphasized that, while attractive for reasons of its theoretical
appeal, this framework is speculative. For instance, the present scenario does not
specify how and where visual information is transformed into a format that can
be used as input to a forward model. Iacoboni (2003) speculates that, when imi-
tating a movement, the STS forms an inverse model of the movement to be imi-
tated. It then sends a visual description of the movement to the mirror areas in the
parietal and premotor cortex, which generate the motor commands necessary to
perform the movement, a copy of which is send back to the STS as input to a for-
ward model, which predicts the sensory consequences of the planned movement.
However, there is no experimental evidence that the computations of the STS do
indeed constitute an internal model, whether inverse or forward model or both.
What’s more, whereas Gallese (2002) suggests that mirror neurons in premotor
cortex act as a forward model, an interpretation recently put forth by Schubotz
and Von Cramon for ventral lateral premotor cortex as a whole3, Blakemore and
Decety (2001) propose the cerebellum as a possible repository of internal mod-
els. This need not be contradictory though, since internal models may be located
in all brain regions having synaptic plasticity, as pointed out by Kawato (1999).
VIII: Two Hypotheses
I would now like to propose two hypotheses. First of all I would like to speculate
that, when watching dance, the brain is submerged in motor imagery. If this
hypothesis is correct when watching dance the observer is in a sense virtually
dancing along. Above I have reviewed some lines of evidence in support of a role
fo r m o t o r a r ea s i n th e p e r c e p ti o n o f h u m a n mo v e m e n t . Us i ng
magnetoencephalography (MEG) it has recently been demonstrated that, when
listening to piano music, pianists exhibit involuntary motor activity in the unilat-
eral primary motor cortex (Haueisen & Knösche, 2001). Since this requires audi-
tory information to be mapped onto motor areas it is not improbable to assume
that watching dance also activates motor areas.
It is also plausible that watching dance involves a form of motor imagery. To
state the obvious, with the exception of some avant-garde performances that
question the assumptions of other performances, dance has a high movement
density. Movements are at once fast and slow and often intertwine without any
clear beginning or end. Even within a solo the next movement may already have
started before a motion percept of the previous movement has been formed. It
also frequently happens that some limbs are temporarily occluded from view or
that a dancer briefly disappears behind another dancer. As argued above the
brain will complement the movement by interpolating between the positions it
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 95
[3] Poster presented at the CNS meeting in New York, March 2003.
131
131
did perceive, which in case of body positions means choosing a biomechanically
feasible path. Stevens et al. (2001) have provided tentative evidence that this
task involves motor areas. In dance the brain has to work overtime in this respect,
as it is confronted not only with a flood of movement, but also with movements
not part of the brain’s own movement repertoire. And just as actual movement
when exercised to excess produces a state of arousal, so may virtual movement.
It would be interesting to directly test the hypothesis put forth here, for
instance by recording the brain activity of someone watching a short dance
sequence. Since this is likely to activate a variety of brain areas the results may
be difficult to interpret. However, in principle the stimuli would not be very dif-
ferent from those used in biological motion studies or the experiments of Decety
and Grèzes (1997) or Manthey et al. (2003).
In the example adapted from Jeannerod (1994; 1997) it was said that a chore-
ographer watching a dancer imitate a movement ‘experiences a strong feeling of
what should be done and how’, yet it was not explained how and where this feel-
ing arises. Jeannerod also remarked that spectators who see a player miss a ball
will experience a sense of frustration. This however depends which side they are
on. The same event, hitting a target, whether in soccer, basket ball, golf or com-
bat, can be a reward for one person and a punishment for another. Rewards and
punishments are the technical notions for the positive respectively negative
value ascribed to a stimulus. Rewards, like all sensory stimuli, can be both antici-
pated and unanticipated. At the level of neural processing single cell recordings
from monkeys indicate that dopamine neurons respond to the delivery of an
unexpected reward. What’s more, if the monkey learned that a certain stimulus
always preceded delivery of the reward, dopamine neurons would respond to the
predictive stimulus rather than the reward itself. The output of dopamine neurons
therefore appears to encode reward expectation. Another way of saying this is
that dopamine neurons code for an error in the prediction of reward, the discrep-
ancy between the occurrence of reward and the predicted occurrence of reward
(Schultz, 2000).
Dopamine neurons are not the only neurons that respond to prediction errors.
Neurons in the orbitofrontal cortex, a brain region that has been implicated in the
processing of emotion (Rolls, 1999), have also been found to be activated when
stimuli deviate from their expected value. Nobre et al. (1999) asked subjects to
respond as quickly as possible to visual targets appearing at peripheral locations
in the visual field. Immediately preceding the target a visual cue was presented
that either correctly or incorrectly predicted the location of the upcoming target.
This experiment is particularly interesting since the network of activation in the
latter condition comprised regions associated with sensory prediction (the lateral
premotor cortex, BA 6), attention (the posterior parietal cortex) and emotion (the
orbitofrontal cortex). Janata and co-workers (2002) recently reported significant
activity in the orbitofrontal cortex as eight musically experienced listeners per-
formed a perceptual discrimination task requiring them to respond whenever
they heard a note played by a flute instead of a clarinet, or when a note violated
local tonality. Similar results had previously been reported in a task whereby
96 I.G. HAGENDOORN
132
132
participants were asked to respond to the degree of dissonance or consonance in
the chords accompanying a melody (Blood et al., 1999). Goel and Dolan (2001)
finally found the orbitofrontal cortex (BA 10/11) to be associated with the appre-
ciation of humour in the form of semantic and phonological jokes of the kind
‘Why did the golfer wear two sets of pants?… He got a hole in one,’ which are
based on the juxtaposition of expectation and its resolution.
The function of the orbitofrontal cortex may therefore be to evaluate the
reward value of environmental stimuli and to attach an emotional ‘tag’ to a pre-
diction error. In support of this hypothesis in a fascinating fMRI study Anderson
et al. (2003) recently showed that, with respect to olfaction, stimulus intensity
was associated with activation in the amygdala, regardless of whether the odour
was pleasant or unpleasant. Activity in the right medial orbitofrontal cortex by
contrast correlated with pleasantness, but not intensity, while a region in the left
lateral orbitofrontal cortex responded to both unpleasantness and intensity. Even
though these results cannot be straightforwardly extended to other than olfactory
stimuli, the fact that the same event, a goal in sports, as noted above, almost
instantly brings about two opposite responses, suggests that a similar dissocia-
tion may underlie visual processing.
I would now like to propose the following scenario. When watching dance, to
keep track of the movement, which may require a visuomotor transformation in
order to move the eyes or head, the brain makes an internal prediction of its
motion trajectory and dynamics. As argued in the previous section this task is
described by the computational logic of forward models. A deviation from the
expected path results in a prediction error. If the same movement sequence were
to be repeated, the prediction error would be used to adjust the next prediction so
that the visual representation of the motion trajectory and dynamics would be
learned and the prediction error would converge to zero.4Since in dance the fol-
lowing movements are usually different, the brain will put a premium on increas-
ing the likelihood of getting the next movement right. It does so by focussing
attention on processing the motion stimuli. Interestingly dopamine neurons have
also been proposed to play a role in the regulation of attention (Nieoullon, 2002)
and could control a ‘nonselective form of attention or arousal, which is depend-
ent on uncertainty and designed to aid the learning of predictive stimuli and
actions’ (Fiorillo et al., 2003). Furthermore, if within a sequence of movements
the predicted path of a given section corresponds to the actual path, the
orbitofrontal cortex awards a positive tag, if it deviates it awards a negative tag.
The almost instant sigh an audience gives out if in a ballet a dancer suddenly
falls, a radical deviation from the expected continuation of the movement, may
be regarded as the physical manifestation of such a negative tag.
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 97
[4] The activity of dopamine neurons has been shown to resemble a class of reinforcement learning algo-
rithms known as temporal difference models, in which prediction errors are used to adjust the model’s
parameters (Schultz et al., 1997). What is most interesting in the present context is that the standard
temporal difference model can be extended to incorporate internal models, whereby dopamine neu-
rons code for the difference between the actual reward and the reward predicted by a forward model
(Suri, 2001). This could, with the emphasis on could, bring the observed activity of dopamine neurons
in line with the computational logic of forward models.
133
133
Thus in dance there is a double route to pleasure, one operates through the
increased allocation of attention and by promoting a general state of arousal if a
movement deviates from its predicted path, the other by rewarding the correct
prediction of the motion trajectory. It follows that without the interplay of correct
and incorrect predictions the brain may as it were ‘lose interest’5: if the move-
ments are too predictable attention wanes and we feel bored; if they are too
erratic and unpredictable there is no positive reinforcement, which ultimately
leads the brain to focus attention on something else. Anecdotal evidence for this
hypothesis can be gathered both from audience responses and dance reviews.
Now dopamine neurons respond to errors in the prediction of reward, so the
question is, where is the reward in watching dance? And how does the above sce-
nario lead to the conscious experience of pleasure? It could be that the mecha-
nisms that evolved to facilitate avoiding and catching moving objects are also
activated in the absence of immediate threats or rewards. When pursuing a goal
the final reward may only occur after considerable effort and so the brain may
have evolved a motivational mechanism, which signals to the person whether he
is on or off track, analogous to the childhood pastime whereby the distance from
a hidden object is indicated by ‘cold’, ‘warm’ and ‘hot’. There may also be an
evolutionary advantage in correctly predicting the motion trajectory of a moving
target (think of trying to hit it or having to avoid it). Even though this, of course,
is only speculative, it means that correctly predicting the unfolding of a move-
ment is by its very nature rewarding: after all, how could doing something wrong
be rewarding?6
Everyday experience tells us that there is an asymmetry between positive and
negative tags: we are not thrilled every time our perceptual expectations are met,
even though we tend to be surprised if they aren’t. On the other hand, we can
ascribe a goal to almost any movement, for example which of two cars will reach
the traffic lights first, and rejoice if our prediction is correct. Thus it appears that
the emotional tag, when juxtaposed to a goal, gives rise to a feeling of pleasure or
frustration, which is consistent with the distinction between emotions and feel-
ings put forth by Damasio (1994; 2001). An emotion according to Damasio is ‘a
patterned collection of chemical and neural responses that is produced by the
brain when it detects the presence of an emotionally competent stimulus’
whereas feelings are ‘the mental representation of the physiological changes that
characterize emotions’ in juxtaposition to the mental images that caused them
(Damasio, 1994; 2001). This may also explain why we can be puzzled by our
own emotions, for instance, when for the first time in our life, we attend a dance
performance. The act of going to a theatre to watch someone perform a series of
movements may be enough to establish the sort of goal that is a necessary but not
sufficient condition for deriving pleasure from watching movement. Indeed, as I
myself have witnessed, the same people who will cheer if a dancer performs on
98 I.G. HAGENDOORN
[5] Properly speaking it is the person who loses interest.
[6] I would like to thank Ricarda Schubotz for drawing my attention to this possible explanation.
134
134
stage, will ignore her if she performs on the street in front of the theatre, an artis-
tic and behavioural experiment I one day hope to formalize.
The idea that the elicitation, undermining and fulfilment of expectations
underlie our affective response to dynamic stimuli is well established in music
theory (e.g., Meyer, 1956; Tramo, 2001) and has also been proposed as a possi-
ble explanation for humour (Deacon, 1997; Goel & Dolan, 2001). Interestingly a
recent fMRI study investigating the neural correlates of listening to Western
tonal music showed a similar network of activation as has been hypothesized
here, comprising the precentral gyrus (BA 6), inferior frontal gyrus (BA 44), the
cerebellum, all three of which may be associated with perceptual prediction, and
the orbitofrontal cortex (BA 11), in addition to regions associated with auditory
processing (Janata et al., 2002). A PET study whereby neural activity was mea-
sured as subjects listened to pre-selected music which elicited an intense emo-
tional response, often referred to as ‘chills’ or ‘shivers-down-the-spine’, yielded
increased activity in the ventral striatum and midbrain, both sources of
dopaminergic signals, the orbitofrontal cortex, the medial supplementary motor
area (BA 6), the insula and the cerebellum (Blood & Zatorre, 2001). These stud-
ies therefore not only add to the hypothesis put forth here, they also suggest that
one reason music and dance go together so well is that both stimuli share the
same neural mechanisms.
With respect to humour, Marc Jeannerod recently gave an interesting interpreta-
tion in terms of internal models of why we laugh when we see a clown pretending
to make a huge effort to lift a seemingly heavy object, fall on his back. ‘We laugh’,
according to Jeannerod, ‘because we have created in ourselves an expectation by
simulating the effort of the clown, and we see something that is very different from
the expectation. The effect we see is at discrepancy with respect to our internal
model, and this is the source of comedy’ (Gallagher & Jeannerod, 2002).
IX: The Beautiful and the Sublime
The second hypothesis has a remarkable resemblance with the aesthetic theory of
the German philosopher Immanuel Kant (1724–1804). In his analysis of aes-
thetic judgment Kant draws a distinction between the beautiful and the sublime.
Beauty, according to Kant, is the feeling we experience and subsequently endow
upon the object of our experience, when we discover a harmonious order, whether
in art or in nature, that appeals to the mind’s own drive towards creating order. The
sublime also refers to a feeling, or perhaps better, a state of mind, but differs quite
markedly from the beautiful in that it is characterized by disorder and internal con-
flict. Faced with an immense object, a skyscraper or a Boeing 747, or a powerful
phenomenon like a hurricane, the faculty of imagination, in Kantian terms, is over-
whelmed, unable to form an adequate representation of the phenomenon at hand.
This in itself would not be much of a problem if it were not for the faculty of rea-
son, which demands that every object be captured in its totality. It is at this moment
of conflict that, again in Kantian terms, the subject realizes that even though it can-
not represent this grand object, it can conceive of it as such. That is, the subject
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 99
135
135
becomes aware of the ‘presence’ of something that exceeds ‘representation’. The
sublime is the feeling which accompanies the resolution of this conflict. In a more
contemporary terminology we could say that the sublime refers to a moment of
intense awareness, following an initial moment of disorientation, during which
attention peaks and the self is filled with awe.
It is this feeling or state of mind that can be said to characterize the thrill of
watching dance. It is triggered by the failure of the brain to correctly predict the
unfolding of a movement sequence and maintained by its effort, through
increased attention, to keep up with the movement.7
Following the same logic we can also account for beauty, which may be
defined as the feeling that arises when the movement trajectory as simulated by
the brain, and the actual perceived movement as it unfolds in front of our eyes,
coincide. In the event of human motion, this feeling is intensified if the perceived
movement is performed seemingly without effort (Figure 6).
It follows that slow and fluid movements are more likely to be considered
beautiful than fast, jerky movements. This prediction appears to agree
100 I.G. HAGENDOORN
Figure 6. Before the brain has figured out what it sees, the dancers have already moved into a dif-
ferent body configuration. What remains is the memory of something seemingly impossible, here
captured on a photo. Notice also the symmetry and extension of lines in the body. This is what, in
combination with the fluidity and effortlessness with which the movements are performed, creates
a sense of beauty, the virtuosity of the positions and movements fills us with awe and keeps our
attention span at peak level. At the back of the stage a mirror doubled the image of the dancers. Jirí
Kylián, Click–Pause–Silence (2001). Dancers: Stefan Zeromski and Elke Schepers.
Photo: © Joris-Jan Bos.
[7] Surprisingly it is not only neural processing delays that pose a challenge to the perception of (fast)
moving objects, the human brain also takes a long time to respond to slow visual target motion
(Kawakami et al., 2002), possibly because it requires increased attention.
136
136
remarkably well with audience responses and dance reviews, although of course
familiarity with a particular style or movement aesthetic influences the ability to
predict the unfolding of movements, as can be concluded from the above. Here
is, for instance, what one dance critic wrote when Firstext, a ballet by William
Forsythe, was first performed in London at the Royal Ballet in 1995.
[the dancers] have come up with astonishing ways of coordinating their limbs and
timing their interconnections — but they are no match for Sylvie Guillem. She has
the reactions of a racing driver, the hyper-agility of a computer-generated figure:
hardly a pretty sight, but we’re not here to watch pretty (Parry, 1995; my emphasis).
Within a proper conceptual framework the feelings experienced when watch-
ing dance can thus be called ‘aesthetic’. It should be emphasized though that the
distinction between the beautiful and the sublime made here is purely concep-
tual, and that actual experience is a mixture of a variety of feelings. The beautiful
and the sublime are best seen as two moments of an otherwise indeterminate feel-
ing or experience. Just how indeterminate and conflicting aesthetic experience
can be is exemplified by the many words we use to describe our feelings, or
indeed our struggle to find the words that carry exactly the right subtleties. This
does not attest to the inadequacy of language, but rather to the complexity of the
feelings involved.
XI: From Perception to Principles of Aesthetic Experience
Movement proper is only one aspect of dance and for a complete picture we will
also have to look into other features of the visual system than motion processing.
In their seminal paper, The science of art: A neurological theory of aesthetic
experience, Ramachandran and Hirstein (1999) introduced eight, as they claim
universal, laws of aesthetic experience. In short these laws are: enhancement of
features that deviate from average; grouping of related features; isolation of a
particular visual clue; contrasting of segregated features; a dislike of unnatural
perspectives; perceptual problem solving, which refers to the pleasure the brain
takes in deciphering ambiguous scenes; metaphor; and symmetry. The elicita-
tion and resolution of expectations can be considered a principle specific to
dance, music and cinema. Ramachandran and Hirstein continue where Rudolf
Arnheim, in Art and Visual Perception. A Psychology of the Creative Eye, first
published in 1954, had stopped, providing it with the neurobiological founda-
tions made possible by recent advances in cognitive neuroscience.
The main law or aesthetic principle, according to Ramachandran and Hirstein,
is what they call a peak-shift effect. By accentuating traits that are otherwise con-
sidered to be distinctive, perception can be intensified. What are characteristic
features of women? Breasts, hips and waists. And thus, from Indian art to car-
toons, manga and computer games such as Tomb Raider, we find women with
large breasts, tight waists and pronounced hips. The visual system immediately
recognizes these features as belonging to a woman and, because of the
exaggeration, gives off a quicker and stronger than usual response. One prob-
lem with the peak-shift effect as defined by Ramachandran and Hirstein is that
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 101
137
137
it may be difficult to apply to landscapes and still lives or the work of artists like
Rogier van der Weyden. Another problem is that, as Wittgenstein (1953) showed
in his Philosophical Investigations, it is at least doubtful whether there is such a
thing as an essence or an essential feature. Rather than saying that an artist
emphasizes essential features, I would therefore like to suggest that he empha-
sizes some features. Whether they are essential or not is irrelevant, they become
essential in his work. Art is leaving out and emphasizing what remains, hoping
that what is left is essential and says something definite, albeit fleetingly. Fol-
lowing this line of thought, dance can be seen as a reverse-engineered peak-shift
effect in human movement perception.
The second principle suggested by Ramachandran and Hirstein concerns the
grouping of related features. The brain constantly searches for patterns, struc-
tures and other regularities in the environment. Since patterns tend to signal an
object or an underlying regularity that the brain can use to its advantage, it pays
to be apt at recognizing patterns, even if from time to time it means being lured
into seeing things that aren’t there. It follows that by adding structure and pattern
an artist can please the viewer or audience. The frame acceleration in the movie
Koyaanisqatsi (1983), for instance, revealed the implicit patterns in traffic. In
dance synchronizing the movements of limbs belonging to different bodies
teases the brain into seeing a kind of moving hyperbody. Having patterns evolve
in time creates tension and suspense, both synonymous with anticipation. Conse-
quently it may not be necessary to perfectly organize everything in a ballet, since
the brain of the observer will do part of the job all by itself. However, as in La
Bayadère or Balanchine’s Symphony in Three Movements, razor-sharp and per-
fectly synchronized lines create a peak-shift effect in pattern perception, at
which Ramachandran and Hirstein (1999) and Ramachandran (2001) also hint.8
As to the other principles introduced by Ramachandran and Hirstein, symme-
try of course is a principle that is very popular, both in classical ballet and
102 I.G. HAGENDOORN
Figure 7.
Grouping and contrast. Count
with me: group vs. individual, sit-
ting vs. standing, front vs. back
stage, hat vs. bare head. The scene
and its subsequent unfolding are
perfectly composed. Pina Bausch,
Viktor (1986).
Photo: © Jochen Viehoff.
[8] I disagree with Ramachandran (2001) that introducing a postural peak-shift in a point-light display
would create a stronger response than real movement and be more pleasing to watch. As argued
above, point-light displays show how little information is needed to recognize biological motion.
Enhancing the outcome of this process of subtraction is not necessarily a postural peak-shift. Distort-
ing the point-light display may even make the movements more difficult to observe.
138
138
modern dance, whereas contrast can be achieved by the opposition of a group
and an individual, male and female, left and right, front and back, global move-
ments involving the whole body and local movements confined to one limb, etc.
It should be noted that none of these principles specify how they are to be
applied. Apart from that, the application of one principle may coincidentally
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 103
Figure 8.
Symmetry and perceptual
problem solving. How do
they do it? Notice also how
arms and leg s al ig n an d
extend into space. Jirí
Kylián, Dreamtime (1983).
Dancers: Gaby Baars, Elke
Schepers and Johan Inger.
Photo: © Dirk Buwalda.
Figure 9.
Grouping, symmetry, con-
trast. Hans van Manen,
Große Fuge (1971). Danc-
ers: N etherla nd s Dance
Theatre 2.
Photo: © Joris-Jan Bos.
Figure 10.
Grouping in combination
with isolation of a particu-
lar visual clue. The photog-
rapher clicked at the right
moment, but the fact that he
co u l d m a k e t he ph o t o
means that the moment was
already implicated by the
choreographer. William
Forsythe, Workwithinwork
(1998). Dancers: Ballett
Frankfurt.
Photo: © Dieter Schwer.
139
139
implicate another, which is why even a formal analysis like this remains ambigu-
ous as to what produces which effect to what extent (Figures 7–10).
XII: Conclusion. The Nature of Dance and Choreography?
The feelings we experience when watching dance are the product of a myriad of
sensory, cognitive and emotional brain processes. As such they are not accidental
but depend on the properties of the brain processes involved in the analysis of sen-
sory stimuli and on the interaction of expectations, associations and personal pref-
erences as laid down in the brain. The present analysis has been limited to only one
aspect, that of sensory processing, which itself has been largely narrowed down to
one feature. Nothing was said about the functional organization of visual motion
processing areas MT and MST in relation to dance, which many will no doubt see
as a grave omission. However, I would be happy if as a result of the present article
others will be encouraged to correct my errors and omissions.
In so far as sensory processing is concerned there is no difference between an
audience watching the finished work and a choreographer watching a work in
progress. A choreographer will continue adjusting a piece until every aspect has
been fine-tuned to its desired perceptual and emotional effect. The feelings expe-
rienced by the audience are therefore in part prefigured by the choreographer.
From this it follows that the feelings embedded in a choreography can be
regarded as a function of the properties of the brain mechanisms that give rise to
these feelings. I would therefore like to suggest that what is composed in a chore-
ography is at once the material — movement — and the sensation it entails, first
and foremost a sensation of movement, but extending to other feelings, events
and contingencies.9Choreography could thus be defined as the two-way merger
of movement and sensation, whereby movement passes into sensation and vice
versa. When creating or rehearsing a ballet both choreographer and dancers are
as much spectator as they are author and composer. It may be said that what are
exercised during rehearsals are movements, but they are exercised to learn the
moves as much as to craft the sensations they entail. The art of dancing is as
much ‘moving’ as it is knowing the effect a movement sequence has on the
104 I.G. HAGENDOORN
[9] The philosophically inclined reader may see in this hypothesis the resonance of what French philoso-
phers Gilles Deleuze (1925–1995) and Félix Guattari (1930–1992) in What is Philosophy? argue is
the aim of art, ‘by means of the material, to wrest the percept from perceptions of objects and the states
of a perceiving subject, to wrest the affect from affections as the transition from one state to another:
to extract a bloc of sensations, a pure being of sensations.’ (Deleuze & Guattari, 1994). Their notion of
percept is somewhat idiosyncratic and differs from its common usage in cognitive neuroscience. For
Deleuze and Guattari, percepts are no longer perceptions, any more than affects are feelings or affec-
tions. ‘They are independent of a state of those who experience them’ (Deleuze & Guattari, 1994),
they have an existence of and in themselves. What Deleuze and Guattari are trying to say is that the
artist has enveloped into the work of art the perceptual and emotional effect it has on the observer.
Why are we moved, thrilled or delighted by a ballet, a novel or a movie? We are because what is
caught and preserved in a work of art is not only an act or a situation, but also the emotional affect it
entails. The interested reader is also referred to Les Muses (1994) by the French philosopher Jean-Luc
Nancy who argues that a work of art opens up a realm of perceptual qualities beyond the traditional
definition of the senses. For instance music is more than ‘sound’ and encompasses such qualities as
melody, rhythm, timbre, tonality, consonance, dissonance and chroma.
140
140
observer. It is knowing where to put an accent, which phrase to emphasize, when
to accelerate or when to release. Most of this knowledge is implicit and dancers
like to refer to it as ‘the body’s knowledge’, but this is just metaphor, it all resides
in the brain. It encompasses the principles of perception and motor control and
what is summed up by experience, the product of years of training, itself shaped
by tradition, that is, the choices and aesthetic preferences of past generations.
In this article I have attempted to make explicit some of the implicit principles
that govern the perception of dance and thereby its creation. Perception, of
course, is only the springboard leading up to the judgement involved in creating
a work of dance. Nonetheless a better understanding of the brain mechanisms
involved in perceiving dance may help dancers and choreographers in fine-tun-
ing their material to its perceptual effect, after all, this is what they already do,
implicitly.
What is true for dance may also hold for cognitive neuroscience. As cognitive
neuroscientists probe deeper into the neural mechanisms for recognizing human
movement, they devise ever more complex and subtler motion stimuli. Effec-
tively this means choreographing appropriate movement sequences. A meeting
of cognitive neuroscience and choreography could therefore tell us more about
both perception and dance.
In my own work I try to translate what I have here referred to as ‘principles of
perception’ into techniques dancers can use when they are improvising
(Hagendoorn, 2003). One such technique is based on the principles of a mecha-
nism not discussed here: attention. Movements can be distinguished into global
movements involving the whole body or an arm or leg and local movements
involving a hand, shoulder or finger. Now whatever is global or local depends to
a large extent on the preceding and subsequent movements. By switching
between global and local movements a dancer can bring structure into her danc-
ing and ‘expand’ or ‘contract’ the attention of the audience into space or onto a
focal point. Indeed one could say that this is what choreographers do: they com-
pose movements such that they draw and maintain attention. Several other of my
improvisation techniques address the tendency of the brain to extrapolate a
movement. Knowing this tendency a dancer can consciously play with it by
either continuing a movement into its ‘natural’ or ‘most likely’ direction or by
what one of my dancers once called ‘going against the logic of the movement’. A
simple example is to undo a body configuration not by withdrawing or re-mov-
ing the limb that gave rise to it, but by moving another limb. If this sounds com-
plicated just stick your right arm between your legs and instead of removing your
arm move your right leg around your arm.10 Practice these techniques and sur-
prise your friends and colleagues at the next party you attend.
The real test for any theory is whether it explains existing and predicts new
experimental findings. In the present case these ‘experimental findings’ take the
shape of ballets and dance performances. The problem with ‘explaining’ why
something is ‘good’, or is generally considered as such, even when concentrating
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 105
[10] Examples can be found on my website, http://www.ivarhagendoorn.com.
141
141
on perceptual effects, is that there may be several competing and equally valid
explanations. This follows from the fact that the aesthetic principles mentioned
above can only be isolated in theory, in practice some or all combine and rein-
force each other, as the illustrations show. An interesting aspect of the present
account is that it also explains why some things don’t work (e.g., Figures 11 and
12).
But, as I once heard a voice say in The Vile Parody of Address, a ballet by Wil-
liam Forsythe, ‘Despite what I keep saying it doesn’t have to be this way’.
Acknowledgements
I would like to thank William Forsythe, Michael Arbib and Ricarda Schubotz for
many thought-provoking discussions about dance, the brain and aesthetic expe-
rience, and Sarah-Jayne Blakemore and the anonymous referees for their com-
ments on an earlier version of this article. I would also like to thank the dancers
of the Frankfurt Ballet, both past and present, for their friendship and hospitality
as well as the dancers I myself have had the pleasure of working with, especially
Andrea Mitschke, Raphaëlle Delaunay and Ester Natzijl.
References
Anderson, A.K., Christoff, K., Stappen, I., Panitz, D., Ghahremani, D.G., Glover, G. et al. (2003), ‘Dis-
sociated neural representations of intensity and valence in human olfaction’, Nature Neuroscience 6,
pp. 196–202.
Ando, H. (2002), ‘Human Brain Regions Involved in Visual Motion Prediction’, NeuroImage S20054.
106 I.G. HAGENDOORN
Figures 11 and 12. Find the dancer(s). Great photos, but on stage costumes of dancers and back-
ground merge, as they do in these stills, making it difficult to discern the movement. This, of
course, was the original concept, but it worked much better than intended. Ed Wubbe, Perfect skin.
(1993). Dancers: Charlotte Baines, Mariëlle de Jong and Rinus Sprong (l). Mariëlle de Jong and
Rinus Sprong (r). Photo: © Hans Gerritsen.
Figure 11 Figure 12
142
142
Arnheim, R. (1974), Art and Visual Perception. A Psychology of the Creative Eye. The new version
(Berkeley, CA: University of California Press).
Beardworth, T., Bukner, T. (1981), ‘The ability to recognize oneself from a video recording of one’s
movement without seeing one’s body’, Bulletin of the Psychonomic Society 18, pp. 19–22.
Beintema, J.A., Lappe, M. (2002), ‘Perception of biological motion without local image motion’, Pro-
ceedings of the National Academy of Science USA 99, pp. pp. 5661–3.
Berry, M.J., Brivanlou, I.H., Jordan, Th.A., Meister, M. (1999), ‘Anticipation of moving stimuli by the
retina’, Nature 398, pp. 334–38.
Berthoz, A. (2000), The brain’s sense of movement (Cambridge, MA: Harvard University Press).
Blakemore, S.-J., Decety, J. (2001), ‘From the perception of action to the understanding of intention’,
Nature Reviews Neuroscience 2, pp. 561–7.
Blood, A.J., Zatorre, R.J., Bermudez, P., Evans, A.C. (1999), ‘Emotional responses to pleasant and
unpleasant music correlate with activity in paralimbic brain regions’, Nature Neuroscience 2,
pp. 382–7.
Blood, A., Zatorre, R.J. (2001), ‘Intensely pleasurable responses to music correlate with activity in brain
regions implicated in reward and emotion’, Proceedings of the National Academy of Sciences 98,
pp. 11818–23.
Bonda, E., Petrides, M., Ostry, D., Evans, A. (1996), ‘Specific involvement of human parietal systems
and the amygdala in the perception of biological motion’, The Journal of Neuroscience 16,
pp. 3737–44.
Carr, L., Iacoboni, M., Dubeau, M.-C., Maziotta, J.C., Lenzi, G.L. (2003), ‘Neural mechanisms for
empathy in humans: A relay from neural systems for imitation to limbic areas’, Proceedings of the
National Academy of Sciences USA, early edition.
Chaminade, Th., Meary, D. Orliaguet, J-P., Decety, J. (2001), ‘Is perceptual anticipation motor simula-
tion? A PET study’, NeuroReport 12, pp. 3669–74.
Damasio, A. (1994), Descartes’ Error. Emotion, Reason and the Human Brain (New York: Avon).
Damasio, A. (2001), ‘Fundamental feelings’, Nature 413, p. 781.
Dawson, M.R.W., Pylyshyn, Z.W. (1988), ‘Natural constraints on apparent motion’, in: Pylyshyn, Z.W.
(Ed.), Computational processes in human vision (Norwood, NJ: Ablex), pp. 99–120.
Deacon, T. (1997), The symbolic species. The co-evolution of language and the human brain (London:
Penguin).
Decety, J., Grezes, J., Costes, N., Perani, D., Jeannerod, M., Procyk, E. et al. (1997), ‘Brain activity dur-
ing observation of actions. Infl uence of act ion content and subject’s strategy’, Brain 120,
pp. 1763–77.
Decety, J., Grèzes, J. (1999), ‘Neural mechanisms subserving the perception of human actions’, Trends
in Cognitive Sciences 3, pp. 172–8.
De Duve, Th. (1996), Kant after Duchamp (Cambridge MA: MIT Press).
Deleuze, G., Guattari, F. (1994), What is Philosophy? [transl. H. Tomlinson and G. Burchell] (New
York: Columbia University Press).
Dittrich, W.H., Troscianko T., Lea, S.E., Morgan, D. (1996), ‘Perception of emotion from dynamic
point-light displays represented in dance’, Perception 25, pp. 727–38.
Engel, A.K., Fries, P., Singer, W. (2001), ‘Dynamic predictions: oscillations and synchrony in top–down
processing’, Nature Reviews Neuroscience 2, pp. 704–16.
Fadiga, L., Fogassi, L., Pavesi, G., Rizzolatti, G. (1995), ‘Motor facilitation during action observation: a
magnetic stimulation study’, Journal of Neurophysiology 73, pp. 2608–11.
Fiorillo, C.D., Tobler, P.N., Schultz, W. (2003), ‘Discrete coding of reward probability and uncertainty
by dopamine neurons’, Science 299, pp. 1898–902.
Freyd, J. (1983), ‘The mental representation of movement when static stimuli are viewed’, Perception
and Psychophysics 33, pp. 575–81.
Gallagher, S., Jeannerod, M. (2002), ‘From action to interaction’, Journal of Consciousness Studies 9,
pp.*******
Gallese, V. (2002), ‘The Shared Manifold Hypothesis’, Journal of Consciousness Studies 8, pp. 33–50.
Gallese, V., Fadiga, L., Fogassi, L., Rizzolatti, G. (1996), ‘Action recognition in the premotor cortex’,
Brain 119, pp. 593–609.
Goel V., Dolan R.J. (2001), ‘The functional anatomy of humor: segregating cognitive and affective com-
ponents’, Nature Neuroscience 4, pp. 237–8.
Grèzes, J., Costes, N., Decety, J. (1998), ‘Top–down effect of strategy on the perception of human bio-
logical motion: a PET investigation’, Cognitive Neuropsychology 15, pp. 553–82.
Grèzes, J., Fonlupt, P., Bertenthal, B., Delon-Martin, Ch., Segebarth, Ch., Decety, J. (2001), ‘Does per-
ception of motion rely on specific brain regions?’, NeuroImage 13, pp. 775–85.
Grossman, E.D., Donelly, M., Price, R., Pickens, D., Morgan, V., Neighbor, G. et al. (2000), ‘Brain areas
involved in perception of biological motion’, Journal of Cognitive Neuroscience 12, pp. 711–20.
Grossman, E.D., Blake, R. (2002), ‘Brain areas active during visual perception of biological motion’,
Neuron 35, pp. 1167–75.
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 107
143
143
Hagendoorn, I.G. (2003), ‘Cognitive dance improvisation. How study of the motor system can inspire
dance (and vice versa)’, Leonardo 36, pp. 221–7.
Haueisen, J., Knosche, T.R. (2001), ‘Involuntary motor activity in pianists evoked by music perception’,
Journal of Cognitive Neuroscience 13, pp. 786–92.
Heberlein, A.S., Adolphs, R., Tranel, D., Kemmerer, D., Anderson, S., Damasio, A. (1998), ‘Impaired
attribution of social meanings to abstract dynamic visual patterns following damage to the amygdala’,
Society for Neuroscience Abstracts 24, p. 1176.
Heider, F., Simmel, M. (1944), ‘An experimental study of apparent behavior’, American Journal of Psy-
chology 57, pp. 243–9.
Heiser, M. Iacoboni, M, Maeda, F., Marcus, J., Mazziotta, J.C. (2003), ‘The essential role of Broca’s
area in imitation’, European Journal of Neuroscience 2003 17, pp. 1123–8.
Howard, R.J., Brammer, M., Wright, I., Woodruff, P.W., Bullmore, E.T., Zeki, S. (1996), ‘A direct dem-
onstration of functional specialization within motion-related visual and auditory cortex of the human
brain’, Current Biology 6, pp. 1015–19.
Huettel, S.A., Mack, P.B., McCarthy, G. (2002), ‘Perceiving patterns in random series: dynamic pro-
cessing of sequence in prefrontal cortex’, Nature Neuroscience 5(5), pp. 485–90.
Hurley, S., Chater, N. (2004), Perspectives on Imitation: From Cognitive Neuroscience to Social Sci-
ence (Cambridge, MA: MIT Press).
Iacoboni, M., (2003), ‘Understanding others: Imitation, language, empathy’, in Perspectives on Imitation:
From Cognitive Neuroscience to Social Science, ed. Hurley, S., Chater, N. (Cambridge, MA: MIT Press).
Iacoboni, M., Woods, R.P., Brass, M., Bekkering, H., Mazziotta, J.C., Rizzolatti, G. (1999), ‘Cortical
mechanisms of human imitation’, Science,286, pp. 2526–8.
Iacoboni, M., Koski, L.M., Brass, M., Bekkering, H., Woods, R.P., Dubeau, M-C. et al. (2001),
‘Reafferent copies of imitated actions in the right superior temporal cortex’, Proceedings of the
National Academy of Sciences USA,98 (24), pp. 13995–9.
Ione, A. (2001), ‘Innovation in art and science. Reply to Semir Zeki’, Trendsin Cognitive Sciences 5, p. 140.
Janata, P., Birk, J.L., Van Horn, J.D., Leman, M., Tillmann, B., Bharucha, J.J. (2002), ‘The cortical
topography of tonal structure underlying Western Music’, Science 298, pp. 2167–70.
Jeannerod, M. (1994), ‘The representing brain: Neural correlates of motor intention and imagery’,
Behavioral and Brain Sciences 17, pp. 187–245.
Jeannerod, M. (1997), The Cognitive Neuroscience of Action (Cambridge, MA: Blackwell).
Jeannerod, M. (2001), ‘Neural simulation of action: A unifying mechanism for motor cognition’,
NeuroImage 14, pp. S103–9.
Johansson, G. (1973), ‘Visual perception of biological motion and a model for its analysis’, Perception
and Psychophysics 14, pp. 202–11.
Kandel, S., Orliaguet J.P., Viviani, P. (2000), ‘Perceptual anticipation in handwriting: the role of implicit
motor competence’, Perception and Psychophysics 62, pp. 706–16.
Kant, I. (1987), Critique of Judgment [transl. W.S. Pluhar] (Indianapolis: Hackett).
Kawakami, O., Kaneoke, Y., Maruyama, K., Kakigi, R., Okada, T., Sadato, N. et al. (2002), ‘Visual
detection of motion speed in humans: spatiotemporal analysis by fMRI and MEG’, Human Brain
Mapping 16, pp. 104–18.
Kawato, M. (1999), ‘Internal models for motor control and trajectory planning’, Current Opinion in
Neurobiology 9, pp. 718–27.
Kawato, M., Furukawa, K., Suzuki, R. (1987), ‘A hierarchical neural-network model for control and
learning of voluntary movement’, Biological Cybernetics 57, pp. 169–85.
Kohler E., Keysers C., Umilta M.A., Fogassi L., Gallese V., Rizzolatti G., (2002) ‘Hearing sounds,
understanding actions: action representation in mirror neurons’, Science 297, pp. 846–8.
Kourtzi, Z., Kanwisher, N. (2000), ‘Activation in human MT/MST by static images with implied
motion’, Journal of Cognitive Neuroscience 12, pp. 48–55.
Land, M.F., McLeod, P. (2000), ‘From eye movements to actions: how batsmen hit the ball’, Nature
Neuroscience 3, pp. 1340–5.
Latto, R., Brain, D., Kelly, B. (2000), ‘An oblique effect in aesthetics: Homage to Mondrian’, Perception
29, pp. 981–7.
LeDoux, J. (1996), The emotional brain. The mysterious underpinnings of emotional life (New York,
Simon & Schuster).
Manthey, S., Schubotz, R.I., von Cramon, D.Y. (2003), ‘Premotor cortex in observing erroneous action:
an fMRI study’, Cognitive Brain Research 15, pp. 296–307.
Martin, K.A.C., (2002), ‘Microcircuits in visual cortex’, Current Opinion in Neurobiology 12,
pp. 418–25.
McBeath, M.K., Shaffer, D.M., Kaiser, M.K. (1995), ‘How baseball outfielders determine where to run
to catch fly balls’, Science 268, pp. 569–73.
McIntyre, J., Zago, M., Berthoz, A., Lacquaniti, F. (2001), ‘Does the brain model Newton’s laws?’,
Nature Neuroscience 4, pp. 693–4.
108 I.G. HAGENDOORN
144
144
McLeod, P., Reed, N., Dienes, Z. (2001), ‘What we do not know about how people run to catch a ball’,
Journal of Experimental Psychology: Human Perception and Performance 27, pp. 1347–55.
Mehta, B., Schaal, S. (2002), ‘Forward models in visuomotor control’, Journal of Neurophysiology 88,
pp. 942–53.
Meltzoff, A.N., Prinz, W. (2002), The imitative mind: Development, evolution and brain bases. (Cam-
bridge: Cambridge University Press).
Meyer, L.B. (1956), Emotion and meaning in music (Chicago: University of Chicago Press).
Nancy, J.-L. (1994), Les muses (Paris: Galilée).
Nieoullon, A. (2002), ‘Dopamine and the regulation of cognition and attention’, Progress in
Neurobiology 67, pp. 53–83.
Nijhawan, R. (1994), ‘Motion extrapolation in catching’, Nature 386, pp. 256–7.
Nijhawan, R. (1997), ‘Visual decomposition of color through motion extrapolation’, Nature 386,
pp. 66–9.
Nijhawan, R. (2002), ‘Neural delays, visual motion and the flash-lag effect’, Trends in Cognitive Sci-
ences 6, pp. 387–93.
Nobre, A.C., Coull, J.T., Frith, C.D., Mesulam, M.M. (1999), ‘Orbitofrontal cortex is activated during
breaches of expectation in tasks of visual attention’, Nature Neuroscience 2, pp. 11–12.
Oram, M.W., Perrett, D.I. (1994), ‘Responses of anterior superior temporal polysensory (STPa) neurons
to “biological motion” stimuli’, Journal of Cognitive Neuroscience 6, pp. 99–116.
Orliaguet, J.P., Kandel, S., Boe, L.J. (1997), ‘Visual perception of motor anticipation in cursive hand-
writing: influence of spatial and movement information on the prediction of forthcoming letters’, Per-
ception 26, pp. 905–12.
Parry, J. (1995), ‘Out on a limb’, The Observer, 30 April 1995.
Pinto, J., Shiffrar, M. (1999), ‘Subconfigurations of the human form in the perception of biological
motion displays’, Acta Psychologica 102, pp. 293–318.
Ramachandran, V.S. (2001), ‘Sharpening up “The Science of Art”. An interview with Anthony Free-
man’, Journal of Consciousness Studies 8, pp. 9–29.
Ramachandran, V.S., Hirstein, W. (1999), ‘The science of art: A neurological theory of aesthetic experi-
ence’, Journal of Consciousness Studies 6, pp. 15–51.
Rizzolatti, G., Arbib, M.A. (1998), ‘Language within our grasp’, Trends in Neurosciences 21, pp. 188–94.
Rizzolatti, G., Fadiga, L., Gallese, V., Fogassi, L. (1996), ‘Premotor cortex and the recognition of motor
actions’, Cognitive Brain Research 3, pp. 131–41.
Rizzolatti, G., Fogassi, L., Gallese, V. (2001), ‘Neurophysiological mechanisms underlying the under-
standing and imitation of action’, Nature Reviews Neuroscience 2, pp. 661–70.
Robertson, L.C. (2003), ‘Binding, spatial attention and perceptual awareness’, Nature Reviews Neuro-
science 4, pp. 93–102.
Rolls, E.T. (1999), The Brain and Emotion (Oxford: Oxford University Press).
Schoenfeld, M.A., Noesselt, T., Poggel, D., Tempelmann, C., Hopf, J.M., Woldorff, M.G. et al. (2002),
‘Analysis of pathways mediating preserved vision after striate cortex lesions’, Annals of Neurology
52, pp. 814–24.
Schubotz, R.I., von Cramon, D.Y. (2001), ‘Functional organization of the lateral premotor cortex: fMRI
reveals different regions activated by anticipation of object properties, locatio and speed’, Cognitive
Brain Research 11, pp. 97–112.
Schubotz, R.I., von Cramon, D.Y. (2002a), ‘Predicting perceptual events activates corresponding motor
schemes in lateral premotor cortex: an fMRI study’, NeuroImage 15, pp 787–96.
Schubotz, R.I., von Cramon, D.Y. (2002b), ‘A blueprint for target motion: fMRI reveals perceived
sequential complexity to modulate premotor cortex’, NeuroImage 16, pp. 920–35.
Schultz, W. (2000), ‘Multiple reward signals in the brain’, Nature Reviews Neuroscience 1, pp. 199–207.
Schultz, W., Dayan, P., Read Montague, P. (1997), ‘A neural substrate of prediction and reward’, Sci-
ence 275, pp. 1593–9.
Senior, C., Barnes, J., Giampietro, V., Simmons, A., Bullmore, E.T., Brammer, M. et al. (2000), ‘The
functional neuroanatomy of implicit-motion perception or representational momentum’, Current
Biology 10, pp. 16–22.
Servos, Ph., Osu, R., Santi, A., Kawato, M. (2002), ‘The neural substrates of biological motion percep-
tion: an fMRI study’, Cerebral Cortex 12, pp. 772–82.
Shiffrar, M., Freyd, J.J. (1990), ‘Apparent motion of the human body’, Psychological Science 1,
pp. 257–64.
Shiffrar, M., Freyd, J.J. (1993), ‘Timing and apparent motion path choice with human body photo-
graphs’, Psychological Science 4, pp. 379–84.
Suri, R.E. (2001), ‘Anticipatory responses of dopamine neurons and cortical neurons reproduced by
internal model’, Experimental Brain Research 140, pp. 234–40.
Stevens, J.A., Fonlupt, P., Shiffrar, M., Decety, J. (2000), ‘New aspects of motion perception: selective
neural coding of apparent human movements’, NeuroReport 11, pp. 109–15.
Tramo, M.J. (2001), ‘Music of the hemispheres’, Science 291, pp. 54–6.
NATURE AND PERCEPTION OF DANCE & CHOREOGRAPHY 109
145
145
Vaina, L.M., Lemay, M., Bienfang, D.C., Choi, A.Y., Nakayama, K. (1990), ‘Intact “biological motion”
and “structure from motion” perception in a patient with impaired motion mechanisms: a case study’,
Vision Neuroscience 5, pp. 353–69.
Vaina, L.M., Solomon, J., Chowdhury, S., Sinha, P., Belliveau, J.W. (2001), ‘Functional neuroanatomy
of biological motion perception in humans’, Proceedings of the National Academy of Sciences 98,
pp.11656–61.
Wittgenstein, L. (1953/19993), Philosophical Investigations [transl. G.E.M. Anscombe], (New York:
Prentice Hall).
Wolpert, D.M. (1997), ‘Computational approaches to motor control’, Trends in Cognitive Sciences 1,
pp. 209–16.
Wolpert, D.M., Flanagan, J.R. (2001), ‘Motor prediction’, Current Biology 11, pp. R729–32.
Wolpert, D.M., Ghahramani, Z., Jordan, M.I. (1995), ‘An internal model for sensorimotor integration’,
Science 269, pp. 1880–2.
Wolpert, D.M., Ghahramani, Z., Flanagan, J.R. (2001), ‘Perspectives and problems in motor learning’,
Trends in Cognitive Sciences 5, pp. 487–94.
Wolpert, D.M., Doya, K., Kawato, M. (2003), ‘A unifying computational framework for motor control
and social interaction’, Philosophical Transactions of the Royal Society London B 358, pp. 593–602.
Zeki, S. (2001a), ‘Artistic Creativity and the Brain’, Science 293, pp. 51–2.
Zeki, S. (2001b), ‘Closet reductionists’, Trends in Cognitive Sciences 5, pp. 45–6.
110 I.G. HAGENDOORN
146
146
Visualization as Interpretive Practice: The Case of Detective
Fiction
Disseratation Presentation, delivered 28 February 2003
Andrea K. Laue
http://www.people.virginia.edu/~akl3s/diss/dissertation_presentation.html
My talk today has two parts - an introduction to work I'm doing for a
p
articular section of my dissertation followed by an introduction to the
guiding principles, to the argument which frames my project. I start with a
discussion of narratology and detective fiction, pointing to theoretical and
methodological gaps in our current studies of narrative before outlining work
I'm doing to fill this gap. Then I shift to the more general discussion of
visualization, or, more specifically, of external aids to cognition, ending with
an argument for increased use of deformative modes of interpretation which
take advantage of visual cognition. I'll be using terms which will probably be
new to many people in the audience, and even those familiar with the terms
may not agree with my definitions of them.
[SHOW VISUAL] This is a diagram of plot which follows the conventions of
Thomas Pavel's narrative grammar.
Pavel's diagrams draw on what David Herman has labelled "classical
narratology," or theories of narrative based largely on structuralist linguistics.
Seeking systematic descriptions of narrative form, classical narratologists
developed grammars of narrative based on fundamental units and on rules for
combining those units. Basically the units were events and the combinatory
rules were either based in causal or temporal logic. The implicit argument was
that the structure of narrative was a structure of events: narrative reduced to
story. Now concepts of "event" evolved into functions, moves, kernels and
catalysts and different logics of combination were considered, but the focus on
the extractable fabula remained. Visualizations produced by scholars working
within this structuralist framework display their primary interest in the fabula
-- these diagrams look almost skeletal, paltry frameworks for supporting all
that isn't plot.
Beginning in the late 1970's scholars began incorporating more recent work in
linguistics -- paricularly in semantics and in discourse analysis -- with
p
revious theories of narrative. This work, labelled by Herman as "post-
classical" narratology, investigates narrative as a mode of telling, as a tool for
Pa
g
e 1 of 8AKL: Dissertation Presentation
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\AKL Dissertation Presentation.htm
147
147
structuring human experience. Narrative asks interpreters to structure a world
-- not just events in sequence -- and the work of the narratologist is the study
of how contextually oriented strategies for reading effect that structuring. In
his 1999 introduction to post-classical narratologies, David Herman argues
that the question central to current narratology is: "how do narrative designs
both shape and get shaped by the process of interpretation?" And
narratologists should study "narrative understandings" rather than seek
skeletal forms.
Working within the context of post-classical narratology, cognitive
narratologists inform our understanding of narrative with models of
interpretation developed by cognitive scientists -- researchers interested in the
p
rocesses of human intelligence and often in the computational modelling of
those processes -- and cognitive psychologists -- researchers interested in
things such as attention, perception and memory and their effects on
information processing. Central to this endeavor is the argument that reading
always involves interpretation. From the moment we begin perceiving a text
we are engaging in an act of interpretation-- there is no pure act of "reading"
before interpretation. A reading -- in an effort to reform our discourse, reading
is used only as a noun -- is an expressible retelling of the act of interpretation:
it's a retrospective retelling of the telling. Although a critical framework is
often more evident in the reading, it's always a part of the interpretation as
well. It can be more or less conscious during interpretation, but it's never
absent. Cognitive narratologists are interested in the relation between the
material artifacts we call texts and the interpretive practices enacted as we
realize texts. Texts become series of cues which prompt activities in the
reader, and it is these activities which give form to narrative. Thus the objects
of study for narratologists are the cues which prompt these structurings, and
the goal of this study is a characterization of the constraints at work in these
cues. Classical narratologists advanced causality as the logic immanent to plot
and therefore to narrative; cognitive narratologists propose a preference-
b
ased
logic by which textual cues are identified and patterned into story worlds.
These theories echo the work of Stanley Fish even if he is not always cited in
cognition-based studies of literature. Two decades ago Fish argued that we
misuse the word "read," that we pretend that there exists a text that can be
p
erceived prior to interpretation and that reading involves some unstructured
experience of that text. It is his project to eliminate this usage of the word
Pa
g
e 2 of 8AKL: Dissertation Presentation
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\AKL Dissertation Presentation.htm
148
148
"read" and to convince professional scholars that we are always already
interpreting. In this sense a text is only what we, as readers, make of it.
Composing a work with a particular interpretive community in mind, an
author codes a text in such a way that it seems likely a reader, a member of the
target community, will realize these codes when she interprets the text, when
she performs the narrative. We think of these interpretive communities as
trafficking largely in theories of literature, devices common to difference
genres, graphical conventions employed by, say, a line of poetry. At least to
my knowledge, cognitive narratologists have yet to connect these habits of
interpretation with their theories of textual cues. The cues identified are most
often grammatical -- changes in pronouns which prompt deictic shifts, for
example -- rather than formal or conventional -- say clues in detective fiction,
or certain spatial arrangements of words which prompt poetical readings. So I
p
ropose to study literary devices and convetions as textual cues, and to
explore what I call "rules for reading," or the preference-based logics
employed during interpretation.
And I plan to study these cues and these rules using visual deformations of
textual artifacts. I generate visualizations of narrative structuring, of the
p
rocess of interpretation commonly referred to as reading. I'm beginning with
detective fiction, a structuring activity we see as largely conventional and
associate with a literal implementation of plot as chain of causally-related
events. Although ostensibly about something -- a crime -- it is literally about
the (re)construction of something -- the acts preceeding and constituting the
crime. In its temporal inversion between the fabula and the sjuzhet, its
incorporation of character-bound narrators and narratees, and its literalization
of causal logic through the use of clues, detective fiction seems to perform
narrative. Narratologists using a variety of approaches argue for the centrality
of detective fiction in our studies of narrative. And many theorists of detective
fiction argue that clues are the critical component of the genre both on a
formal and on a socio-historical level. And, fittingly for my project, Ginzburg
argues that Conan Doyle's stories are about interpretation itself, about a
cognitive model which allows retrospective predictions, which enact a method
of interpretation that is prognostication. Umberto Eco goes so far as to argue
that detective fiction is a machine, "a machine that functions basically on a set
of precise units governed by rigorous combinational rules." His units are pairs
of contrary values or characters, not clues, but Eco too generates a
visualization of Fleming's detective stories based on their incorporation of
Pa
g
e 3 of 8AKL: Dissertation Presentation
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\AKL Dissertation Presentation.htm
149
149
these pairs.
Eco describes his intentions thus: "it is my plan to devise a descriptive table o
f
the narrative structure in the works of Ian Fleming while evaluating for each
structural element the probable incidence upon the reader's sensitivity" (96).
Two key phrases here: "in the works" and "probable incidence." My project
differs from Eco's in its conception of narrative structure -- is it in the text or
in the interaction between text and interpreter -- and in its method -- I'm
asking where clues actually function as cues. Instead of taking the linguistics-
based approach to defining cues, I'd like to look at how literary devices and
conventions function as cues. That is, how authors encode artifacts so as to
encourage particular structuring activities during performance, during
interpreting, during reading.
Back briefly classical narratology and its skeletal visualizations. Such fabula-
based approaches present narrative as syntax, reducing it to a linear, forward-
moving experience. While trying to escape the confines of Saussurean
linguistics, cognitive narratologists often cite the fallacious analogy between
the sentence and the narrative, the premise that one can essentially
deconstruct, can diagram, a narrative as one would diagram a sentence. In fact
we might feel very comfortable with such diagrams of narrative just as we feel
comfortable with diagrams of sentences, not because they offer insight into
our experience of language but because they offer visual tools for exposing
where asyntactical structures diverge from the norm. But what more recent
narrative theorists speculate is that narrative structure lies in the reader as
much as it lies within any text, and the text might be understood as a machine
which produces certain reactions in the reader. What I'm interested in are the
interactions of the reader and the machine, what I've called the experience of
reading but what others have called the phenomenal text.
So, my questions are:
zHow do interpreters map literary conventions onto textual artifacts, and
how does that mapping effect narrative structuring?
Or, more specific to my first study:
zWhen do interpreters render/impart clues on a textual artifact, and how do
these clues structure detective fiction.
Pa
g
e 4 of 8AKL: Dissertation Presentation
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\AKL Dissertation Presentation.htm
150
150
It is interpretation or the phenomenal text that I am trying to visualize here.
Fish argues that interpretation of texts is a temporal phenomenon; Brooks and
others argue that narrative is quintessentially temporal. I agree. Although I
don't know that we have to think of time as strictly linear; and I don't think we
have to reduce textual artifacts to a sequence of words on a page. That
sequence is one of the axes of interpretation, but only one. I sometimes ask
students to graph in a rather traditional sense a timeline of the fabula and a
timeline of the sjuzhet which represents their structured narrative, and then I
ask them to connect these two lines. The lattice, the netting that connects the
two seems a closer approximation of narrative than either of the lines alone.
That space between seems to be the real location of what we call narrative,
and "time" in that space lingers and loops. To express this a different way . . .
linguists often speak of the plane of discourse and the plane of reference. The
p
lane of discourse includes those words connected in sequence on pages
grouped in chapters and bound in a book--the textual artifact. The plane of
reference is the "world" one imagines as one performs the artifact. The space
between in what I'm looking at.
[SHOW/DESCRIBE MY VISUALIZATION]
I propose an empirical method of investigation, a method that explores actual
readings by real readers. I will give readers a copy of Doyle's "The Red-
Headed League" and ask them to mark points in the text at which they identify
something as a clue. They will use two colors to mark the text, one to indicate
something that seems to be a clue when first encountered and another to
indicate something that seems to be a clue only after reading further in the
text. So readers mark the text the instant they identify some bit of information
as a clue to the mystery, and they'll also have the option of flipping back and
finding a detail that seems significant only after reaching some later point in
the story.
N
ow I want to step back a bit and show how this fits into my project as a
whole. Visualizing the structuring of narrative is only one part of my project;
the larger project involves a more general argument for the role of external
aids which draw on visual modes of cognition. Now that's a mouthfull. Why
don't I just say: I argue that visualizations have a place in literary research. I
chose the more precise if clumsy diction of "external aids which draw on
visual cognition" in recognition of the fact that we already use visualization in
literary study. Critical articles and monographs are themselves visualizations,
Pa
g
e 5 of 8AKL: Dissertation Presentation
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\AKL Dissertation Presentation.htm
151
151
external aids to cognition based on conventions so familiar that they seem
almost natural. The page employs visual structures intended to invoke certain
p
ractices of reading, and we as readers process these cues even if we do so
largely unconsciously.
So I'm suggesting that we produce visualizations which challenge our
p
ractices of reading. In their 1999 article "Deformance and Interpretation,"
Jerome McGann and Lisa Samuels argue for a mode of interpretation termed
deformance, a mode which emphasizes performative rather than
hermeneutical methods of interpretation. McGann and Samuels deform
several poems using operations such as "print poem backwards" or "print only
the nouns," all the while arguing that this creation of pattern is no different
than what all interpreters do except in the extent to which the operations are
made explicit and employed self-consciously. Critical exegesis in our familiar
scholarly journals is no less deformative than McGann and Samuels project:
the former involves patterning with the goal of producing a reading which is
subsequently published in a visual form which is itself a deformance of the
original artifact.
In his disseration entitled _Algorithmic Criticism_, Stephen Ramsay proposes
to build (digital) deformance machines, tools that would facilitate such ludic
criticism. By virtue of the fact that computer languages are largely unfamiliar,
we still think of programming a computer in a way that we no longer think
about programming a critical article: we routinely call functions and initiate
loops when interpreting literature, we just call the activities by different
names. Computers do offer exciting potentials for deformance, however,
p
articularly for defamiliarization through re-presentation in unfamiliar visual
forms.
To inform this re-presentation, I draw on research in the fields of information
science, computer science and logic. These scholars argue that we have for a
long time relied heavily on external aids to cognition -- think about how
difficult it is to multiply two three-digit numbers if you have to do it in your
head . . . add paper and pencil and the tast becomes far easier -- but computers
offer potential for significant improvement on our current aids. Coming from
a background in Human-Computer Interaction, Ben Shneiderman argues that
computers offer offer interactivity, increased speed and increased
computational capabilities. In short, digital technologies allow for complex
manipulation of very large amounts of data in a networked space. Logicians
Pa
g
e 6 of 8AKL: Dissertation Presentation
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\AKL Dissertation Presentation.htm
152
152
Jon Barwise and John Etchemendy are developing Hyperproof, a computer
p
rogram fo
r
teaching elementary logic which draws on both linguistic and
diagrammatical representations. Barwise argues that diagrammatic reasoning
exposes the dual nature of any act of reasoning -- reasoning as an act of model
building and an act of deduction. This reasoning involves two activities: 1)
operating on a diagram site according to a set of operational rules 2) reading
off information from a diagram site according to semantic conventions. Users
of Hyperproof manipulate a diagrammatic re-presentation of linguistic
information with the intention of generating some new information, some new
conclusion.
N
ow I have no intention of reducing literary criticism to reasoning . . . quite
the contrary, in fact. But I do think we might learn from the tools of the
computer scientist and the logicians. Imagine a machine that allows you to
visualize complex textual artifacts -- that's what I'm approaching with the
images I showed you today. And imagine a computer program that allows you
to manipulate those visualizations -- that is, in part, what the Ivanhoe Game
p
romises.
I propose that we, members of English departments, explore external aids that
target, challenge and exploit our visual intelligence, our visual imaginations.
Why might we do such a thing? As a preliminary list, I present six affordances
of visualizations:
1. pattern recognition (visual intelligence seems particularly apt at this)
2. extend working memory
3. additional semantics = additional set of meanings and relations
4. translation of axis of experience = spatial rather than temporal
5. "closure under constraints" (barwise) - diagrams generate lots of
information that would need to be inferred if using sentential logic; lots
of the work is done for you
6. "cycles of inspection" (glasgow, et. al, xxiii) -- inferences may be added
and removed from the diagram. visual manipulation -- we can peel away
the onionskin with the option of restoring it later
The key in all of this is the argument that external aids to cognition must
p
rovoke thought, must help us ask interesting questions as much as they help
us answer existing ones. We at IATH and in SpecLab often toss around the
idea of an "aesthetic provocation," by which we mean a visual representation
Pa
g
e 7 of 8AKL: Dissertation Presentation
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\AKL Dissertation Presentation.htm
153
153
of linguistic data that is both aesthetically pleasing -- in the company of some
we go so far as to say that it must be seductive -- and intellectually
challenging or productive. We seek cognitive aids that make critical
arguments and promote creative imaginings.
Created on ... March 11, 2003
Pa
g
e 8 of 8AKL: Dissertation Presentation
3/16/2006file://C:\Documents and Settin
g
s\Michael\Deskto
p
\AKL Dissertation Presentation.htm
154
154
AN INTRODUCTION BY BENJAMIN BRATTON
"Translation passes through continua of transformation, not abstract ideas of
identity and similarity."
--W. Benjamin.
"And, finally, whether it has essential limits or not, the entire field covered by
the cybernetic program will be the field of writing."
--J. Derrida.
"Myths of Electronic Living," begins what we hope will be a different kind of
conversation about contemporary life and politics, a forum composed of many
different voices that in the past have not often been heard along side one
another. This first "transmission" of _SPEED_ includes articles both by
members of the "academy" and "industry"; people too often separated, not by
interest or conviction, but by impersonal structures of various kinds. Part of
what we hope to accomplish in this, and in future issues, is to circumvent
these structures toward the discovery of new objectives, languages and
understandings that might produce better and more public ways of thinking
about technology, media and society. This first collection grew out of one
such coming together.
Last August, at the annual SIGGRAPH meetings in Anaheim, CA, _SPEED_
sponsored two panels entitled "Technology, Representation, Politics," made
up of people from the academy and the business community, professionals
interested in theorizing questions of technology. The contrast between our
p
anel participants' marginal (within the context of the SIGGRAPH setting),
b
ut earnest attempt to demand a dialogic relationship with technology, and the
beyond-overwhelming razzle-dazzle of the nation's largest computer graphics
industry hoe-down, was, to say the least, severe. Nevertheless, the sometimes
Pa
g
e 1 of 13Introduction: Code and M
y
th
3/16/2006htt
p
://
p
rox
y
.arts.uci.edu/~nideffer/
_
SPEED
_
/1.1/bratton1.html
155
155
confusing but surprisingly successful communicative relationship that we
were able to construct amongst ourselves demonstrated an avenue of hope. It
is in the spirit of each participant's attempt to make sense of the various "local
tongues" that s/he had to work with in Anaheim, that this first transmission is
dedicated.
This process of learning to hear across our technically afforded languages, and
the producing of better (if temporary) "hybrid languages" is a skill that we can
only learn from each other. It is my hope then, that the informal, introductory
essay below is written in such a way that certain arguments central to the
different approaches we wish to bring together will be accessible to a
(relatively) wide audience. The reader therefore, may find many things
already known in a re-stated form, but, s/he will also hopefully find some
ideas (even if they weren't intentionally included) that work to establish useful
if temporary "links" between present and potential concerns. It is meant as an
initial gesture, not a call for ground-rules, in what we hope will be a fruitful
conversation between those who are paid to think and those who are paid to
construct the tools with which we think. Since we all do both, we anticipate an
exciting exchange.
We would also like to thank the participants at the SIGGRAPH panels whose
work does not appear in this collection: Steve Kurzman of U. C. Santa Cruz,
Department of Anthropology; Alan Barnum-Scrivener of Advanced Digital
Systems; Garth Gillespie of U. C. Santa Barbara, Department of German; and
David Frerichs of Future Vision Technologies, Inc.
MYTHS OF ELECTRONIC LIVING
This issue is about "myth"--myth as a language with which we make sense of
life in the midst of electronically mediated social reality, myths as stories that
we tell ourselves and each other about what that reality *is*, and myths as
stories that we are *told* about where that reality is going and what we should
be doing in that course. We hope to put forward the term as a mode of critical
p
olitical discourse, one that will make the inherent reflexivity of "electronic
living" a useful political priority. In effort to do this, a re-orienting of what the
term "myth" means is necessary in order to ask it to do what we suggest. First,
instead of being "inherited stories" whose authority depends upon their age,
these myths are present and future-oriented operations. Second, a particular
contingent understanding of language as being now "infinitely rearrangeable,"
Pa
g
e 2 of 13Introduction: Code and M
y
th
3/16/2006htt
p
://
p
rox
y
.arts.uci.edu/~nideffer/
_
SPEED
_
/1.1/bratton1.html
156
156
dependent only upon its internal structures and not an external symbolic order
that would precede it, is necessary to make myth, as Roland Barthes wrote
almost forty years ago, "... a type of speech." (1957, 109)
How then does "myth" operate for and through electronic speech, especially
the kind of speech we all make back through the electronic information that
surrounds each of us and yet does not seem to want our replies? The question
of language, how and what it "represents," and how and what it can be forced
to "represent', is one that works through and across each of the articles in this
current issue of _SPEED_. From strikingly different "angles," and with an
*approximate* variety of "native tongues," the authors work the question of
"myth" and what it can do (and already does) in our different "electronic
lives." But as suggested above, what lies behind this question is a particular
notion of language's structure, a "digital" structure which has for better or
worse asked us to translate all experience into its particular form. This
structure has been commonly called the "code." A number of theorists of
electronic subjectivity, Jean Baudrillard and Arthur Kroker perhaps most
severely, have approached the "code" as an apocalyptic monolith; not so much
as a social structure but, rather, as the end what we could take the "social" to
b
e. It is seen as instituting a terminal seriality of meaning by disembedding the
sign, not only from its referent, but also (perhaps therefore) instating any and
all conceivable action into a synchronic and implosive mechanism of sign
circulation. *Our* purpose is to address something quite different. The "code"
works in different ways at different times and in different spaces for different
p
eople; it is not, therefore, external to the particular social circumstances of
those who must organize their lives through its limits and affordances. The
"code" is rather, in its peculiar "structure," an imagined linguistic space whose
p
roperties are always a matter of the location from where one is able to enter
its demands. By pushing its traces through the widest variety of experiences,
back onto itself and out into other locations, we hope to, in whatever small
way, indicate a suspension of the "code" as a "total" and paradigmatic logic,
and point towards its democratic possibilities as an avenue of practical, if
unpredictable, politics.
CODE
First, however, in order to find our way back to that address of "something
different," we need a more specific explanation of what the "code" *is*. As
already said, as "code," language seems to become infinitely re-arrangeable.
Pa
g
e 3 of 13Introduction: Code and M
y
th
3/16/2006htt
p
://
p
rox
y
.arts.uci.edu/~nideffer/
_
SPEED
_
/1.1/bratton1.html
157
157
Computer "code," for example, allows "any" speech as sequential
combinations of zeroes and ones. Keep in mind that the relationship between
the social "code" and machine language is a crucial conjunction. In both cases,
by re-organizing the basic blocks, or "grams," anything can be produced, since
virtually anything and everything is composed of these serial, digital units.
This is not only the basis of computer languages, but now also according to
some notions, of the process of thinking itself. The character, Bryce, from the
_
Max Headroom_ TV show seems to have a handle on this; he says that "the
human mind is nothing but a binary computer ....you know, lots of on/off
switches."
This basic machinic "understanding" of cognitive processes has, however,
taken hold in more arenas than science-fiction television. In many Psychology
departments for instance, mental health is losing ground to artificial
intelligence. Here the "code" pushes in and offers a new model of the mind as
machine. A mathematical theory of information, the amplification of
"information" as source and as solution, over and beyond the flimsiness of the
dream and the story, has made possible the marriage between neural-network
cognitive psychology and the theoretical wings of computer science. The
consideration of the computer as a mind is congratulated by the consideration
of the mind as a computer. What brings the two together is the common
language of the code, the binary "digital." Computers speak a language of
these infinite binaries, zeros and ones in particular but re- assemblable
p
atterns of repetition and variation, as do, in this popular manner of speaking,
human social machines.
As the uncanny abilities of some machines to give the "feeling" of human
sentience and the empathetic interaction of intelligence is seen as ultimately
reducible to the seemingly unromantic flow of zeros and ones, what becomes
clear to researchers in artificial intelligence is that thought itself, or what can
count as thought, is an equation, an effect of flows, intensities and networks o
f
numbers and number theory. Outside of the laboratory, however, what
becomes difficult is the totalizing character of such a linguistic structure.
Though, it is then now possible to "say" anything, there is (according to
Kroker's principle) nothing much left to say, or no new ways to say anything,
since all utterances would be simply variations of this closed gaming. This
p
roblem of "replication" has been central to the theory and philosophy of
language long before the invention of the computer, but that should be
Pa
g
e 4 of 13Introduction: Code and M
y
th
3/16/2006htt
p
://
p
rox
y
.arts.uci.edu/~nideffer/
_
SPEED
_
/1.1/bratton1.html
158
158
expected, as in many ways the computer is a synthetic moment of a longer and
broader trajectory of technical translation and production of "universal
languages." The reader should be cautioned at this point that our concern is
not with the "scientific" validity of neural network modeling, we assume that
within the range of its own questioning, it has plenty. Our concern is with
what is now popularly understood as the "paradigmatic" operations of the
code: the notion of a "code," whether semiotic as for Baudrillard, machinic as
for the computer scientist, biological as for the geneticist, constructable as for
the neural-network psychologist, or repeatable and beautiful as for the
enthusiast of fractal equations, which has, in many ways emerged as the
dominant logic of our period. This grand pronouncement is, of course, part of
that emergence, and as such should be read as a question, as part of what we
wish to complicate and what we invite the reader to help us complicate, and
not as the deadly end-game envisioned by Baudrillard.
The "code," broadly conceived as "universal language," has been a haunting
vision within the specifically Western story of media for quite some time. It is
p
art of its genealogical evolution. Marshall McLuhan's spiritualization of
globalization was based in an ecumenical imagination of transcending
difference through the device and toward a universal orality of culture,
community and presence. The reception of his thought benefited from the
existence of digital language and the location of that language in the external,
yet extensional, device, whether the computer, the car or the phone line. His
p
redecessors were focused on language itself as the device through which to
work both the science and spirituality of media. Attempts have been made
throughout modernity to construct a "global" language capable of indicating
any idea or truth to and by anyone through the artificial device. A sketched
history includes Francis Bacon's bilateral alphabet, Pascal and Leibniz'
calculating alphabets and machines, Charles Babbage's "difference engine,"
Claude Shannon and Norbert Wiener's mathematical theories of
communication, as well as later productions of computer languages as the
basis of working artificial intelligence and the neural-network cognitive
models of the mind. This history of total languages parallels the historical
development of mathematics, machines, ciphers and philosophies of "mind,"
and as I have drawn it here, continues to perform an ordering role in the
techniques of scientific knowledge.
In attention to the limited space afforded by an introductory essay, we will
Pa
g
e 5 of 13Introduction: Code and M
y
th
3/16/2006htt
p
://
p
rox
y
.arts.uci.edu/~nideffer/
_
SPEED
_
/1.1/bratton1.html
159
159
consider only two of the above moments in the social and mythological
history of "global language" and allow them to contextualize the others as
well as the trajectory of such projects in general: (1) Jorge Luis Borges"
story/essay "Analytical Language of John Wilkins," and (2) the social
p
hilosophy of "code" as the basis of artificial intelligence and "interpreting
machines." This, hopefully, will indicate the character of our framing the
works included in this first transmission as also "interpreting machines" of a
sort.
According to Borges' short story, Wilkins, taking his cue from Descartes'
p
roposition for a "general language that could organize and contain all human
thought" began to undertake that task and conceive such a device. Again
quoting Borges directly: "Wilkins divided the universe into forty categories or
classes, which were then subdivisible into differences, subdivisible in turn into
species. To each class he assigned a monosyllable of two letters; to each
difference, a consonant; to each species, a vowel. For example, de means
element; deb, the first of the elements, fire; deba, a portion of the element of
fire, a flame." (This system, as Borges notes, is similar to Leibniz' numerical
alphabet wherein zero is written as 0, one 1, two 10, three 11, four 100, five
101, six 110, seven 111, eight 1000... such a device is, of course, the basis for
digital computer language.) Since such a language is based on the "universal"
p
rinciples of mathematics and infinite serial sequentially, it would then be
assumed that Wilkins had indeed produced, as Descartes intoned, a language
that could express any and all concepts and things, by and to anyone,
regardless of their native tongue. Problems arise, however, in the schematics
by which hierarchies of the "order of things" are arranged, and subsequently
quantified in letters. Such apparently arbitrary classifications and subdivisions
are presented; the ninth category: metals as "imperfect," "artificial,"
"recremental" and "natural." Ultimately, these, and all categories, mimic the
"Chinese Encyclopedia" also here discussed, in which animals are divided
into: "(a) those that belong to the Emperor, (b) embalmed ones, (c) those that
are trained, (d) suckling pigs..."--*stop me if you've heard this one before*.
The ultimate inability to close the machinings of language has not, however,
deterred anyone from continuing the attempt to construct devices of "pure
knowledge." (see Borges, ibid)
The computer chips that are fast becoming ubiquitous in our daily lives,
whether or not we recognize the chip in our coffee makers to be a computer,
Pa
g
e 6 of 13Introduction: Code and M
y
th
3/16/2006htt
p
://
p
rox
y
.arts.uci.edu/~nideffer/
_
SPEED
_
/1.1/bratton1.html
160
160
are built upon the mathematical alphabets briefly discussed above:
combinations of zeros and ones in specific sequences, ordered by higher
"linguistic" functions, and producing the effect of a picture, a word, a voice--
"knowledge" of any and all sorts, or so it may seem. Here, a brief sketch of
the procedure's modern history is useful. The attempt to construct artificial
linguistic devices that could say and think all things was made somewhat
more modest by attempts to construct physical devices that could "think"
specific, limited ("mathematical") things. Leibniz, Pascal and Babbage
(among others, see Goldstone) built cumbersome but elegant "calculators" that
could (taken as a group) perform basic mathematical functions by
manipulating zeros and ones in accordance to a language of their
correspondence to complex integers. Wiener and Shannon, as a project for the
U.S. military during World War II, constructed firing tables and mechanisms
that would allow surface-to-air missiles to "track" and fire upon moving
targets in the sky. The steam-rolling complexification of such machines soon
resulted in the development of vacuum tube based "supercomputers," such as
the Eniac. These machines could process (what seemed at the time to be)
massive amounts of information, in the form of zeros and ones, in order to
p
rovide ("translated") information more quickly than previously possible.
Especially after the advent of the silicon chip, the sub- discipline of artificial
intelligence began to attract more and more interest; here, again, the vision of
a "linguistic device" that could "think" all things, was resumed.
The excitement over constructing artificial thinking devices based on
conceived parameters of human cognition, memory and judgment, resulted in
a peculiar discursive switch: as models of the computer became increasingly
based upon models of the brain, models of the brain became increasingly
based upon models of the computer. The neural-network or Connectionist
discourses of cognitive psychology assume similar models of thought--
decentralized, information-routing, pattern-building and electro-stimulus
based processing--as do those in Computer Science who attempt to build
"learning networks," artificial systems that can learn to route information and
solve unexpected problems without immediate human assistance. Present day
p
hone call routing networks, for example, are based on paradigmatic
knowledges gleaned from Connectionist cognitive psychology. These
developments hold tremendous interest not only for the study of social
knowledge, but also for the practical, reflexive production of social speech.
They indicate a shift toward (or successful radicalization of) the quantification
Pa
g
e 7 of 13Introduction: Code and M
y
th
3/16/2006htt
p
://
p
rox
y
.arts.uci.edu/~nideffer/
_
SPEED
_
/1.1/bratton1.html
161
161
of the social self, a self now conceived as a "node" through which social
information might pass, and thereby force the spiritual character of citizenship
in the "global village" (sic) to re-focus its sites.
MYTH
As the model of the mind, however fashioned, is conversant with the model of
the machine, the model of the self is fashioned in a manner conversant with
the model of that mind. This is the crucial insight afforded by our truncated
readings of Baudrillard and Kroker. But as suggested above, we wish to
activate the term "myth" in order to map not simply routes of escape from
these models, but to indicate better mutations of those codes, to orient the
seriousness of electronic subjectivity toward its own terms. A general,
synthetic model of how the works included in _SPEED_ 1.0 might do this is
p
robably not what is required. Mutation is always specific in character, even i
f
"global" in purpose and potential ramification. As such, this introduction will
couch each piece in its own terms and thereby, with a little luck, the reader
will be able to formulate a glimpse, even if by negative definition, of what
each affords toward these ends.
After stating this, allow me to contradict myself somewhat by suggesting, if
not a model for "myth," at least some moments of overlap between the works
below. Myth, as it is formulated here, is understood as the stories that we tell
ourselves and each other about what the social reality of electronic life "is," as
well as the capacity of those stories to subsequently circulate themselves in
the terrain from which future stories are formulated. That is, in a circumstance
in which nothing can be granted unconditional belief, as nothing can be
confidently assumed to have veracity, the procedure of establishing meaning
in contemporary context is definitely "risky business." Instead of performing
the concluding actions of past stories, thereby extending the security of those
stories to order and give meaning to present circumstances, each of us is
forced to improvise --re-code-- meaning from incomplete clues.
Contradictory information from equally "valid" sources comprises the
landscape from which we formulate the stories that allow us to continue living
in the social world. These "fragments" make claims for us and ask us to make
claims for them. Two possible reactions to this situation might be opposed for
our purposes. One is to assume a real veracity of a particular combination of
these seemingly infinite fragments (like the scholars in Borges' library of
Pa
g
e 8 of 13Introduction: Code and M
y
th
3/16/2006htt
p
://
p
rox
y
.arts.uci.edu/~nideffer/
_
SPEED
_
/1.1/bratton1.html
162
162
Babel who searched in vain amongst an infinity of texts for the single "true"
combination of graphs and letters); that is, to confront the incompleteness of
the social sign as one would confront a broken device--the imperative is then
to put it back together in the form already known and understood, to make it
do what it once did. Another possible approach is to accept the challenge of
the incomplete sign as a source of constructive (not re- constructive) power
and knowledge (and not necessarily only power/knowledge.) To do so is to
actively re-arrange found texts in order to produce new meanings and to
imagine and to speak new social organization through them. In this sense,
there is no such thing as a pure cultural producer and no such thing as a pure
cultural receiver. To produce is, especially in the moment of the "code," a
matter of creative re-arrangement and active dissemination of potential maps,
combinations, based on that re-arrangement. Focusing on the fact that the
"myths" I create from incomplete codes might in some way find their way into
the myths you subsequently disperse, might seem a needlessly "Idealist" (in
the sense of concepts leading the change of material circumstances) way to
imagine a political electronic language, but it remains true that the guarantees
afforded by clear distinctions between what is substantial and what is un-
substantial are no longer available, except as potential contingencies of
electronic negotiation.
This first transmission begins with conversations with two contemporary
"fiction" writers: (1) "Technically Speaking" with Mark Leyner and, (2)
"Apparatus and Memory" with Kathy Acker. Both authors have, as we have
arranged it here, taken the position of cultural producer, myth-maker, to be
always an of interpreter of fragments. Leyner's novels, fast-forward romps
through the debris of the consumer collective consciousness, force to the
surface the sublime delinquency of what hopes to "succeed" the "public
sphere." His literary characters can be read in a manner similar to the way one
reads the competitors in a video game; the ability to "inhabit" their pre-
determined intentions is directly proportional to their two-dimensionality-- it's
very easy. Somewhere between Trivial Pursuit and Pat Sajak's rewriting of
_
Mein Kampf_ --if the ingredients in "Twinkies" were to serve as the basis of
a sequel to _Easy Rider_-- Leyner dissects the imagined and lived
metaphysical contradictions and epiphanies of the channel-surfing
consciousness.
Kathy Acker's procedure of constructing her texts can be (and often is) seen as
Pa
g
e 9 of 13Introduction: Code and M
y
th
3/16/2006htt
p
://
p
rox
y
.arts.uci.edu/~nideffer/
_
SPEED
_
/1.1/bratton1.html
163
163
a distillation, of sorts, of life lived without guaranteed social signs. For those
less familiar with her work, Acker's novels, short-stories and plays are
constructed primarily from bits and pieces of already written works. Acker
"re-writes" works that have "written her," in order to make them say what she
needs to say back at them and through them. A passage from Conrad might be
made to condemn the condescending and disciplining gaze of the colonial cop.
A passage from Burroughs might be made to implicate the reader in (his) own
desire for the "climax of the text." Some lines from Rimbaud might be made
into cyberpunk science-fiction. Everything is made into something else, made
to say what Acker hears in it; and primarily what they are made into is a
"throwing" of Acker's desires into the text, at the reader and therefore out into
the world where they might come back to her on better terms. Even though
her texts are constructed from precedent bits and pieces of literature, Acker's
"voice" is so loud in her re-workings that it sometimes deafens the reader. One
cannot "read" her works in the distanciated manner of an eavesdropper, they
are too demanding. They demand, not empathy, but a response. The procedure
of reading her works is quite like the procedure by which they are created: the
text sits before the reader, and like few others, Acker's work requires an
interjection of the reader's desires into the text by "listening" to the text,
thereby allowing it to do what the reader needs from it. In this sense, her's is
p
erhaps archetypal of what "myth" can now be. It is a re-arrangement of a
p
recedent reservoir of incomplete signs, one that injects that reservoir with
subjective desires, and as disseminated, demands its host to re-inject the text
with its own reactions and re-formulations.
Mark Jenkins, in his piece entitled: "The "Pleasures & Terrors" of Identity:
Language & Subjectivity in Kathy Acker's _Empire of the Senseless_,"
explores the nuts and bolts of a particular myth, Acker's re-created, post-
p
atriarchal Paris. He explores the parameters of Acker's specific invocations
of the horizons of transgression, at the level of the body and at the level of
language, as a political juncture or axis around which the literature of life can
be thought to function. The circumstance of contemporary political writing is
one in which "the stable identity of mythic self-hood becomes a wavering
mirage, causing the subject the immense pain and confusion of desire
infinitely deferred. But amidst this infinite mortgaging of self, Acker argues
that the very ex-centricity of our regime of dislocation and absolute difference
p
resents new opportunities for the construction of previously impossible
narratives of new desires and, therefore, new kinds of subjectivities."
Pa
g
e 10 of 13Introduction: Code and M
y
th
3/16/2006htt
p
://
p
rox
y
.arts.uci.edu/~nideffer/
_
SPEED
_
/1.1/bratton1.html
164
164
In "Final Amputation: Pathogenic Ontology in Cyberspace," Mark Pesce
hopes to complicate the conceptual distinction between what has
unfortunately been bifurcated as the "meat world" and the "virtual" worlds of
lived and potential cyberspace. Even as a conduit of electronic information,
the fleshly hardware through which we make any meaningful connection has
its thresholds, its breaking points. It not only can get drunk on its own code, it
can be made ill by inappropriate enterings into the inertial spaces of
electronically- immersed sociality. His is neither a single-track warning
against the "unnatural" dangers of VR, nor a repetition of some inherent
difference between a reality "out here" and a quadriplegic falling "in there." It
is rather, a complication of the overlaps. Pesce's warning assumes the
concretely physical character of the electronic image. The lighted image
induces specific physiological changes in its "host," indeed, this is how we
even see it. With this re-organized vision of cybernetic connection, he initiates
a conceptual groundwork for the ambulances that will undoubtedly patrol the
"information super-highway" some time in the not-too-distant future.
Robert Nideffer's piece: "Imag(in)ed Gulfs," examines the role of myth in the
reception, by both intellectuals and "consumers," of the Persian Gulf War. He
suggests that fundamental mis-negotiations of "myth" made critical
understandings of "postmodern" death and destruction, not only limited, but in
many ways impossible. Framing much of the critical work on the Persian Gulf
War was what might be seen as a panic over a loss of "reality" in the midst of
the incredible technologization of its operation. At one extreme is
Baudrillard's famous statements that the war "will not," "is not," and "did not"
happen, but was rather circulated un-tethered in the "unreal" spaces of
irreferential media. At the other end are complaints from many on the
oppositional Left who, disappointed in the ineffectuality of traditional means
of protest, demanded a recovery of the "real" spaces of war's violence.
N
ideffer suggests that this "myth," the verifiable and desirable differentiation
between "natural" war and "artificial" war (a notion reproduced in critical
commentary from both the Left and the Right), served to obsfuscate the
always cybernetic character of social-conflict-at-a-distance, and thereby left
oppositional voices without the explanatory tools necessary for undoing the
effectiveness of the nationalist, racist, nostalgic and spectacular "myths" that
ordered the war's participation for the world audience.
In "Digital Fall Guys," Will Kreth examines the various potential
Pa
g
e 11 of 13Introduction: Code and M
y
th
3/16/2006htt
p
://
p
rox
y
.arts.uci.edu/~nideffer/
_
SPEED
_
/1.1/bratton1.html
165
165
ramifications of being lead by a particular myth, the "information super-
highway," and what that will mean for the future development of tomorrow's
common sense of what "one" can do with what should be very public
technologies. Present throughout the text is a warning concerning the direct
application of older social and legislative precedents to the emerging arenas o
f
media law. Without anything approaching consensual or even hegemonic
understandings of where "we" are going with new mechanisms of
communication, we are left *entirely* at the level of metaphor, presently a
metaphorical "highway." The awkward, "futurist" tones of these "debates"
have not invited an outpouring of serious, critical commentary. Kreth,
however, demonstrates the potential danger of remaining disengaged from
these issues. It will be fatal if we underestimate the intentions of corporate and
governmental interests for control over and access to new media, simply
because others are overestimating what is frequently seen as the media's
inherently democratic capacities. The "myths" that are leading the cornerstone
legal decisions currently being debated in congress and the courts will become
tomorrow's unchallenged precedent as to how electronic sociality will
function. Kreth's work is an intervention in that procedure, one that hopes to
orient thought toward the elaboration of better "leading myths" by tracing the
dystopian and utopian imaginations from different blocs of legal realities and
p
ossibilities.
We hope you enjoy...
REFERENCES
Babbage, Charles. _The Analytical Engine and Mechanical Notation_. New
York: New York University Press. 1989.
Bacon, Francis. _The Advancement of Learning_. Ed. William Aldis Wright.
4th ed. Oxford: Clarendon Press. 1891.
Barthes, Roland. "Myth Today." _Mythologies_. Trans. Annette Lavers.
London: Paladin. (1972): 109-159.
Baudrillard, Jean. "Or, the End of the Social." _In the Shadow of the Silent
Majorities--Or the End of the Social: and Other Essays_. Trans. Paul Foss,
Paul Patton and John Johnston. New York: Semiotext(e). (1983): 65-94.
Pa
g
e 12 of 13Introduction: Code and M
y
th
3/16/2006htt
p
://
p
rox
y
.arts.uci.edu/~nideffer/
_
SPEED
_
/1.1/bratton1.html
166
166
Benjamin, Walter. _Illuminations_. Ed. Hannah Arendt; Trans. Harry Zohn.
N
ew York: Schocken Books. 1968.
Borges, Jorge Luis. "The Library of Babel." _Ficciones_. Ed. Anthony
Kerrigan. New York: Grove Press. (1962): 79-88.
Borges. "The Analytical Language of John Wilkins." _Other Inquisitions_.
Trans. Ruth L. C. Simms. Austin, (1964): 101-105.
Derrida, Jacques. _Of Grammatology_. Trans. Gayatri Chakravorty Spivak.
Baltimore: Johns Hopkins university Press. 1976.
Dreyfus, Hubert L. _What Computers Can't Do: The Limits of Artificial
Intelligence_. Rev. ed. New York: Harper & Row. 1979.
Goldstine, Herman H. _The Computer: From Pascal to von Neumann_. New
Jersey: Princeton University Press. 1972.
Kroker, Arthur and David Cook. "Baudrillard's Marx's." _The Postmodern
Scene: Excremental Culture and Hyper-Aesthetics_. New York: St. Martin's
Press. (1986): 170-188.
Leibniz, Gottfried Wilhelm. _The Monadology and Other Philosophical
Writings_. Trans. Robert Latta. Oxford: Clarendon Press. 1898.
Shannon, Claude and Warren Weaver. _The Mathematical Theory Of
Communication_. Urbana: University of Illinois Press. 1962, [c1949].
Wiener, Norbert. _Cybernetics; Or, Control and Communication in the
Animal and the Machine_ . Cambridge: Technology Press [c1948].
_
SPEED_1.1: TOC
Pa
g
e 13 of 13Introduction: Code and M
y
th
3/16/2006htt
p
://
p
rox
y
.arts.uci.edu/~nideffer/
_
SPEED
_
/1.1/bratton1.html
167
167
