Immersive Painting.

Abstract

This paper presents a human machine interface, which helps elderly people learn how to become aware of their physical state
and how to influence it. One of the biggest requirements for such a system is to provide an intuitive interface which does
not overexert an elderly person, while also being easily accepted. Here, the connection of art and computer science offers
the ideal outlet for such an interface. In our work, we show an user interface that is pleasant, expressive and does not look
like the traditional computer interactions. We use classical biosignals, such as skin temperature or skin conductance, to
get the necessary information of the physical state of the user. This information is presented to the user as individual artwork,
which is created from the measured biosignals and the position of a cursor. Of course, the traditional scientific graph output
is also made available. Another aspect of the system is that its design, allows its smooth integration into a normal home
environment, or art studio. All necessary components are off-the-shelf, commercially available products to reduce costs and
to allow a quick setup time.

Full text

Immersive Painting
Stefan Soutschek1, Florian Hoenig2, Andreas Maier2, Stefan Steidl2, Michael
Stuermer2, Hellmut Erzigkeit1, Joachim Hornegger2, and Johannes Kornhuber1
1Department of Psychiatry and Psychotherapy, University of Erlangen-Nuremberg,
Erlangen, Germany
2Chair of Pattern Recognition, Department of Computer Science, University of
Erlangen-Nuremberg, Erlangen, Germany
Abstract. This paper presents a human machine interface, which helps
elderly people learn how to become aware of their physical state and
how to influence it. One of the biggest requirements for such a system
is to provide an intuitive interface which does not overexert an elderly
person, while also being easily accepted. Here, the connection of art
and computer science offers the ideal outlet for such an interface. In
our work, we show an user interface that is pleasant, expressive and
does not look like the traditional computer interactions. We use classical
biosignals, such as skin temperature or skin conductance, to get the
necessary information of the physical state of the user. This information
is presented to the user as individual artwork, which is created from
the measured biosignals and the position of a cursor. Of course, the
traditional scientific graph output is also made available. Another aspect
of the system is that its design, allows its smooth integration into a
normal home environment, or art studio. All necessary components are
off-the-shelf, commercially available products to reduce costs and to allow
a quick setup time.
Key words: Computer Science, Art, Ambient Assisted Living, Elderly
People, Human Machine Interface, Biosignals
1 Introduction
Within the next decades, the continuous improvement of medical care, the in-
creasing standard of living in the world will result in an increase of the antici-
pated average age [1]. In Germany, for example, the population of people older
than 80 years is the fastest growing segment of population. This segment is ex-
pected to grow from currently four million people to approximately ten million
people in 2050 [2]. This implies a huge challenge to society, government and in-
dustry but also for each individual person. It is necessary to address the different
kinds of limitations due to for example illnesses, injuries or natural deteriora-
tion, that occur or arise with aging. Sustainable solutions need to be created
that support elderly people in their everyday activities and to provide assistance
in case of emergency. Interdisciplinary research is required for the development
of solutions that help cope with the upcoming changes in our society.

2 Stefan Soutschek et al.
Especially for systems developed for the focus group of elderly people it is
important that such developments are attractive, easy to use and address individ-
ual needs. The majority of elderly persons the advantages of a new acquisitions
need to be clear. Price, training period and personal benefit are only a few of
the important factors, which influence their purchase decision.
In our work, we investigate a new approach, where we combine art, psy-
chology and computer science to develop interfaces for elderly people. We utilize
artwork as access to modern communication technology and additionally we want
to use it as a means for promoting the collection of clinical relevant data. The
aim is to direct the attention, especially of elderly people, to clinically relevant
topics, to impart knowledge and to offer screening and diagnostic instruments.
In this paper, we show how the combination of art and computer science can
help elderly people gain an intuitive and playful access to modern technologies,
without the need of complicated instruction manuals. We show that one can
develop systems that make it possible to learn how to interpret and manipulate
ones physical state while having fun.
2 State of the art
An example setup that combines artwork and computer science, to arouse inter-
est and to gain the attention of a user was developed by Shugrina et al. [3]. Here
the properties of an image, e.g. coloring or style, are changed according to the
classified emotion of the user. Features are continuously extracted from a face,
which is recorded by a camera. This example shows that the utilization of adap-
tive art, independent of the actual media, has big advantages. Art as intuitive
and attractive interface for the visualization of the acquired data significantly
increases usability and joy. Our approach, in comparison, is based on the ideas
of classical biofeedback applications, such as described in [4]. In principle, such
applications, much like our system, aim at teaching the user how to interpret
and manipulate his body signs. There is however a significant difference between
traditional biofeedback and our setup. While classical biofeedback approaches
perform a predefined action that is influenced by the change of the biosignals,
e.g. show a movie of a flower (see Fig. 1) which starts to bloom [5], our system
involves the user’s creativity, to create unique artwork everytime it is used. The
basic idea, to combine artwork with the interpretation of biosignals has also been
introduced in the field of biofeedback, e.g. by [9]. But once again there, the user
is just able to modify an existing artwork and not to create his own, individual
artwork. One advantage of our approach is that it gives a larger freedom of ex-
pression to the user and thus keeps the fun factor elevated. One can use it for
entertainment or artistic purposes instead of only a means of necessary biosignal
data collection. Furthermore, while previous systems could not be integrated in
a home environment, our session uses a regular TV screen as a display unit,
which is available in nearly every home nowadays.

Fit4Life 3
Fig. 1. The state of the flower depends on biosignals (images from [5])
3 Feature extraction from biosignals
As input for the painting device, we need to know, whether a user is currently
relaxed or tensed up. To obtain this information, we first of all measure the
skin temperature (ST) and the skin conductivity (SC) with the nexus-10 from
Mind Media [5]. These signals were chosen from the available signals of the
nexus-10, because they are easy to record, compared to other signals, e.g. the
electrocardiogram (ECG), but still robust enough to allow for an analysis of
the physical state of the user. For the ECG, we would need to place additional
electrodes on the body, whereas the recording of ST and SC can be realized by
small sensors, shown in Fig. 2, which are attached to the hand. The measured
raw values are sent to the computer via Bluetooth, which avoids unnecessary
cables and allows the user a higher range of motions.
The big advantage of choosing sensors that are easy to use is, that our sys-
tem can be used without a complicated preparation, which in turn would lead
to a loss of acceptance. There are already systems under development, where
the biosignals can be collected via a glove [6], a vest [7] or a T-shirt [8] in a
comfortable way. Such devices can also be integrated in our system in the future
so as to obtain additional signals, like the ECG, or the respiratory frequency
in a comfortable way. Our system is flexible enough to allow the possibility to
process additional signals besides skin temperature and skin conductivity. As
soon as these devices become commercially available and can be comfortably
and intuitively used by elderly persons, it will be investigated, if these devices
can be integrated in future versions of our painting device.
To ensure that the measured signals have the consistent influence on the
painting process, whether the user tends to relax or tends to tense-up, the nexus-
10 is calibrated and the received raw values are normalized. The normalized
signal values s(n) of each sensor nare multiplied by a sign factor f(n), which
guaranties that an increase of the values implies a tendency for tension and a
decrease of the values indicates relaxation. Afterward they are summed up to a
total sum S, which is the sum over all corrected signals received from the number
of available sensors N.

4 Stefan Soutschek et al.
(a) Nexus-10 (b) Temperature sensor (c) Skin conductance sen-
sor
Fig. 2. Sensors for acquiring skin temperature and skin conductivity (images from [5])
Fig. 3. Workflow of the painting device
S=
N
X
n=1
f(n)·s(n) (1)
The sum Sin Eq. 1 is calculated for each time frame t. In the next step
the sum of the current frame is compared to the average Asum of the last five
frames. If the current sum is larger than this average, this indicates that the
user has become more tense. If the sum of the current frame is smaller than
the average, then this is an indication that the user is more relaxed. With this
information one part of the necessary information that is needed as input for the
painting device is available. The workflow of the device is shown in figure 3.

Fit4Life 5
Fig. 4. Individual artwork created with the painter
4 The painting device
To finally setup the Painter, the position of a paint brush and its color need
to be defined. To realize the paint brush, we currently use a Wii MoteTM[10],
which is used as pointing device. The user just needs to point to the direction of
the desired spot with this device to define the position of the paint brush. For
people, who are used to a computer, it is also possible to use a standard mouse
device to define the current position of the brush.
The color of the paint brush is defined according to the sensed psychological
state as described in Section 3. The information of the state of the user, relaxed
or tensed up, is used to shift the current color between two colors for each frame.
To avoid the case where a single wrong measurement in one frame could result
in a large change of the color, the color only changes gradually from frame to
frame. In the images shown in Fig. 4, the colors are set to vary between blue and
red. As long as the user relaxes, the color of the paint brush continuously shifts
from red to blue over time. If the user is tensed, the color changes from blue to
red. The user is always able to interrupt the painting, just by pushing a button
on the Wii Mote or on the computer mouse. This enables the user to precisely
position the brush before the point is colored. When the user is pointing to a
position, which is already colored, the mean of the old and the current color is
set. If the painting is finished, the user can stop the painting, the image is saved
and one can then either start with the next one or quit the software.
5 Conclusion and future work
With our approach we show one possible way to combine artwork and computer
science to build attractive and intuitive applications that motivate usage and
direct the interest to clinical relevant data at the same time. People, who use this
application, playfully learn, how to interpret and manipulate their psychological
state. Additionally, we are able to integrate our system into a home environment.
This is important, as future work will focus on two main applications.

6 Stefan Soutschek et al.
We want to use the different biosignals from the available sensors to enhance
the existing painter. These different signals could be utilized to control the input
to the artwork. For example, one signal could be used to choose the color, another
signal could be used to choose the type of the paint brush. Additionally, the time
the brush is at the same spot in the image could determine the thickness of the
brush. We also plan to have artists use our prototype and guide us in the next
version of our immersive painting method.
Furthermore, we want to provide a continuous training, which can be utilized
as a screening instrument. It will be investigated, if our system can be used
as platform to build a screening instrument to support the early diagnostic
of dementia. Therefore, a set of pictures of known persons or events of the
past will be shown to an elderly person on the TV screen. As reference, also
pictures with neural content will be in this selection. For each picture, known or
neutral, an artwork from our painting device is created, but in contrast to the
described painting device, the artwork will be created in the background. After
each session these artworks can be used to compare the session with previous
ones. Although, the spatial resolution of the artwork and the delay in the change
of the biosignals will probably not be sufficient to give a detailed prediction,
which detail in the picture had the most influence, we expect the user to show
different reactions, between known and unknown pictures. With the additional
information, which picture shows known persons or events to the user, we will
examine, if a system can be set up, which automatically detects, whether a
picture is known or unknown to the elderly person with the help of the created
artwork. The aim of this enhancement is to gain continuous information if the
memory of an elderly person acts according to the age.
6 Acknowledgement
This research project is supported by the BFS1Bavarian Research Association
“Zukunftsorientierte Produkte und Dienstleistungen f¨ur die demographischen
Herausforderungen - FitForAge”2.
References
1. Commision of the European Communities, Green Paper - Confronting demographic
change: a new solidarity between the generations, COM(2005), 94 final, 2005
2. M. Eisenmenger, et al., Bev¨olkerung Deutschlands bis 2050. 11. koordinierte
Bev¨olkerungsvorausberechnung. Wiesbaden: Statistisches Bundesamt, 2006.
3. M. Shugrina, M. Betke and J. Collomosse, Empathic painting: interactive stylization
through observed emotional state, NPAR ’06: Proceedings of the 4th international
symposium on Non-photorealistic animation and rendering, 87–96, 2006, Annecy,
France
1Bayerische Forschungsstiftung
2www.fit4age.org

Fit4Life 7
4. S.Horowitz, Biofeedback Applications: A Survey of Clinical Research, Alternative
and Complementary Therapies, 12(6), 275-281, 2006
5. http://www.mindmedia.nl/english/index.php (06.07 2009)
6. C. Peter, E. Ebert and H. Beikirch, A Wearable Multi-sensor System for Mobile
Acquisition of Emotion-Related Physiological Data, Affective Computing and Intel-
ligent Interaction, 691-698+, Springer-Verlag, Berlin-Heidelberg, 2005
7. R. Paradiso, G. Loriga and N. Taccini, A wearable health care system based
on knitted integrated sensors, IEEE Transactions on Information Technology in
Biomedicine, 9(3), 337-344, 2005,
8. T. Pola and J. Vanhala, Textile electrodes in ECG measurement, ISSNIP -07 Third
International Conference of Intelligent Sensors, Sensor Networks an Information
Processing, Melbourne, Australia, 2007
9. T. Tirel and S. Hahne and J.Garancs and N. Muller, BIOS - Bidirectional In-
put/Output System, VISION - image and perception, Budapest, Hungary, 2002
10. http://www.nintendo.com (06.07 2009)