Guidance and support for healthy food preparation in an augmented kitchen

Abstract

An important barrier to healthful eating is a lack of cooking competence. To assist people who are motivated to increase their cooking competence, we envision a context-aware kitchen that offers a recipe retrieval and recommendation system. Preceding the system design we conducted an at-home observation study to identify a meaningful set of utensils that can provide insights into the cooking process. Accelerometers embedded into these utensils provide relevant context information for the recipe recommender system to select the meals that fit the user's meal preferences and cooking competence.

Full text

1
Guidance and Support for Healthy Food Preparation
in an Augmented Kitchen
Juergen Wagner, Gijs Geleijnse, Aart van Halteren
Philips Research Europe
High Tech Campus 34
Eindhoven, the Netherlands
firstname.lastname@philips.com
ABSTRACT
An important barrier to healthful eating is a lack of cooking
competence. To assist people who are motivated to increase
their cooking competence, we envision a context-aware
kitchen that offers a recipe retrieval and recommendation
system. Preceding the system design we conducted an at-
home observation study to identify a meaningful set of
utensils that can provide insights into the cooking process.
Accelerometers embedded into these utensils provide
relevant context information for the recipe recommender
system to select the meals that fit the user’s meal
preferences and cooking competence.
Author Keywords
Context awareness, recipes, recommender systems,
accelerometers.
ACM Classification Keywords
H5.m. Information interfaces and presentation (e.g., HCI):
Miscellaneous.
INTRODUCTION
People are in general aware of the importance of a healthy
and balanced eating pattern. However, they have difficulties
to translate this knowledge into a healthy diet. Although
food choice is affected by a wide range of elements [1], the
lack in cooking competence and confidence is a common
barrier towards a healthful main meal [2]. People with
lower levels of cooking competence tend to consume more
convenience meals [3], which generally are considered less
healthy.
We describe a context-aware recipe retrieval and
recommender system that assists the user to select and
prepare healthful meals. The main aim of the system is to
gradually increase cooking competence and confidence, and
consequently motivate its users to prepare healthier food.
We target users who have a basic motivation to improve
both their nutrition and cooking skills but need assistance to
actually plan and prepare new, more healthful meals.
Studies have shown that tailored advice is more effective
than general nutritional information [4]. By offering
concrete meal suggestions that fit the user’s taste, habits
and current cooking skills, actionable advice [5] is given.
To understand the user’s needs and to provide personalized
support, the system tracks the user’s cooking activities with
sensors in kitchen utensils. The information collected by the
infrastructure enables for an assessment of the cooking
competence. Combined with past meal selections, this
knowledge is used to recommend healthy recipes that may
increase the user’s cooking competence. To develop a
meaningful context-aware system, an at-home observation
study was conducted. This study has led to a number of
insights that allow us to assess the cooking competence of a
user-based utensil usage. Having designed a combination of
activity detection sensors, we explore its potential usage to
formulate personalized meal recommendations. Finally,
directions for future work are presented.
THE INFLUENCE OF COOKING COMPETENCE
Cooking competence does not only affect the cooking
process, it has a strong influence a person’s diet. People
with a higher cooking competence are more aware of the
healthiness of their diet and tend to prepare and incorporate
more often healthy ingredients [2, 6].
Cooking competence consists of a variety of different
aspects. Based on literature we identify seven different
properties that describe someone’s cooking competence [2,
6-8].
Use of utensils and appliances describes how utensils and
appliances are physically and mechanically manipulated
during meal preparation.
Being able to do multiple activities at the same time is often
required for more complex meals. Therefore multitasking
is also an indicator of someone’s cooking competence.
To react appropriately in different situations (e.g. stressful
situations) good monitoring and adaptation skills are
required. This property is also important when unexpected
events take place during the preparation of a meal.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,
or republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee.
CHI 2009, April 4–9, 2009, Boston, MA, USA.
Copyright 2009 ACM 978-1-60558-246-7/08/04…$5.00

2
Planning skills, are involved when acquiring ingredients
and when preparing the meal.
The better someone can reproduce recipes with similar
qualities but without following instructions, the higher is
the cooking competence.
A good perceptual skill, allows to predict the effects when
combination different ingredients. It also describes how
well the user knows the effects on an ingredient when
heated.
Nutrition knowledge is needed to prepare a meal with a
high dietary quality.
Users that want to improve their cooking competence
should address one of more of these properties. The
following float chart (Figure 1) shows a schematic of the
system that supports cooking competence.
Figure 1 Float chart that describes a context-aware cooking
guidance and support system..
To develop a sensor infrastructure that can provide the
proper amount of context information for meal preparation
we conducted an observational study to assess the main
activities in the kitchen.
TOWARDS A MEANINGFUL SENSOR INFRASTRUCTUE
Observational Study
In this study we invited nine participants representing
(M=5, F=4)) three different household categories (single
household, couple and family with a child). These user
groups have been identified because we expect differences
in their cooking style, strategy, competence and confidence.
The participants are asked to prepare a meal while being
recorded by multiple cameras. To receive as authentic
behavior as possible we conducted this study in the
participant’s kitchens, instead of our research facilities. As
the participants are in a familiar environment with access to
their own kitchen utensils, this procedure allows them to
stay as close as possible to follow their normal cooking
routines. The participants could freely choose which meal
they are confident to prepare as long it is not a convenience
meal. Because we aim to record authentic cooking
behaviour we always used an unobtrusive camera setup
(Figure 2).
Figure 2 Participants kitchen with an unobtrusive camera
setup
Annotation Schema
After recording the videos an annotation schema was
designed to classify utensils used and activities conducted
in the kitchen. To define the schema two questions need to
be clarified. The first one is to define a level of activities,
between being present in the kitchen and motion primitives
that we want to annotate. Our approach uses two levels.
Level 1 describes when on objects is being moved and level
2 contains more descriptive activities (as they can be found
in a cook book). The second question is when does an
activity start and end? Here we take advantage of the two
levels of activities. As soon a utensil is being moved, it is
moving. This can frame more descriptive activities (e.g.
cutting). Figure 3 visualizes this concept.
The selection of these activities has been an iterative
process during the annotation. New activities had been
defined and similar ones grouped together. The utensils are
these which have been used to perform the activities.
Figure 3 Schematic Image of different levels of activities and
their annotation
The final annotation schema consists of 21 activities and 34
utensils/objects. The activities contain a selection of
actively performed activities (cutting, stirring or grating) as
well as passive activities (e.g. boiling, frying). With this
schema a complex cooking process can be simplified and
described in a structured manner. For each moment in the
cooking process, the current, possibly parallel activities as
well as the utensils in use can be described.
The annotation schema can be used to get insights in the
performed activities, as well as in the utensils used. Figure
Cooking Competence &
Preference Profile
Personalized Support
Recipe Recommender
Recipe DB
Collaborative
filtering
Content based
filtering
........
Level 2 (cutting)
Level 1 (moving)
Being present in the kitchen
Motio n primitives (knife up, down ...)
Level of Activity
Time

3
4 gives insights into the most common (both frequency and
duration) activities and used utensils.
a) This diagram shows the 12 most
common activities based on their
occurrence in relation to the total
number of recorded activities.
b) This diagram presents the 12
activities that have been performed
for the longest time in relation to
the total preparation time.
c) This diagram displays the 12
most common utensils based on
how often they have been used
relation to the overall number of
recorded utensils used.
d) This diagram presents the 12
utensils that have used for the
longest time in relation to the total
preparation time.
Figure 4 Most common (both occurrence and duration)
activities and utensils/objects.
Cooking Behavior Insights
Although the activity adding (to add an ingredient from one
container to another) is normally a very short activity it is
the main meal preparation activity. Comparing these results
with the used utensils/objects, it is noticeable that
ingredients have been manipulated almost twice as often as
others (both occurrence and duration). Besides adding,
cutting and stirring seem to be the most frequent activities.
These two actions represent about one third of all activities
when preparing a meal. These two do not only occur
frequently, also their duration covers 30% of the overall
cooking process. Also these two activities have been
performed by every participant. This finding is also
reflected in the use of the utensils. Knifes and spoons as
well as chopping boards and pans or pots are used most
often. This confirms the intuitive notion that meal
preparation can be described as the manipulation and
adding of ingredients from one location to another.
An activity that occurs very frequent when preparing a meal
is opening and closing of kitchen furniture. These activities
have been recorded at almost any stage of the meal
preparation while ingredients are added or utensils are used.
Opening and closing occur frequently, but generally very
brief actions. Nevertheless they seem to be an important
task.
Further, our observations indicate that there are different
meal preparation strategies depending on personal
preferences, competences, cooking space, and cultural
background. E.g. some people first gather all ingredients for
their meal, separate and miniaturize them and as last step
treat them with heath. Others follow a just-in-time
strategies, where ingredients are gathered and removed
based on the situation. Also a kitchen is not exclusively
used to cook. While preparing a meal many other activities
(e.g. drinking or cleaning) can occur that can be sensed but
do not affect the meal preparation.
Sensor infrastructure
The sensor infrastructure consists of common kitchen
utensils with small wireless accelerometers (inside a
waterproof case) embedded to their handles (as shown in
Figure 5). The total sensor infrastructure consists of more
than 20 utensils. With the wireless accelerometer embedded
into a redesigned handle we can unobtrusive track meal
preparation activities. The tracked sensor data can be used
to identify several properties that describe cooking
competence. The annotation schema developed for the
observational study is used to characterize someone’s
cooking performance.
Figure 5 Example kitchen utensil with an embedded wireless
accelerometer
PERSONALIZED MEAL PREPARATION SUPPORT
Cooking Competence Profile
The annotation schema of recording during the
observational study has shown that there is a potential to
assess someone cooking competence. In our system the
sensor infrastructure replaces the human sensor to assess
attributes that define cooking competence. The data tracked
by the wireless accelerometer can be used to identify some
of these attributes. The most obvious is to characterize how
well someone is using utensils and appliances. By
measuring single activities, the overall cooking process can
be monitored and evaluated as well. The developed
annotation schema is used to simplify and structure
someone’s cooking. This allows us to measure
multitasking, the reproduction of recipes and to a certain
extent monitoring and adaptation skills. These
competences can be described as active ones. They require
the user to perform a task which can be measured. The
other attributes describing cooking competence will be
0% 5% 10% 15% 20% 25% 30%
cut open
drinking
peeling
throwing away
washing
organizing
adjusting
closing
opening
stirring
cutting
adding
Count of Activities
0% 10% 20% 30% 40%
opening
organizing
peeling
grating
baking
washing
adding
stirring
frying
cutting
boiling
Duration of Activities
0% 5% 10% 15% 20%
Cup
Casserole
Pot
Sink
Chopping Board
Bowl
Plate
Knife
Food-Container
Spoon
Wok
Ingredient
Count of Utensils/Objects
0% 5% 10% 15% 20% 25%
Pot
Plate
Bowl
Sink
Food-Container
Chopping Board
Wok
Oven
Knife
Spoon
Stove
Ingredient
Duration of Utensils/Objects
Redesigned
handle Blade
Wireless accelerometer embedded
into the handle

4
handled in a passive form, by taking the responsibility away
from the user.
Personalized Recipe Recommender
We envision a personalized recipe recommender system,
which provides the user with meal suggestions that match
both his food and eating preferences as well as his current
cooking competence. To motivate the user to increase his
cooking skills, the recipes presented should be palatable
and attractive to prepare. Therefore, patterns in available
preparation time (e.g. Mondays are busy days with limited
time for cooking) and food choices (e.g. fish on Fridays),
should be taken into account when developing an effective
recipe recommender system. Moreover, as consumers
demand variety in their meal choices, recipes suggested for
two consecutive days should not be too similar.
To effectively provide the user with meal suggestions, we
opt for a content-based approach. One the one hand, we
want to reason with relevant parameters (cooking time,
ingredients, similarity to other meals) that affect the
acceptability of the meal for the occasion. On the other
hand, the aim is to match meals with the user’s cooking
competence. In [9], a similarity-based method is discussed
to identify recipes that fit a user’s eating pattern. The
algorithm to extract the number of grams of vegetables [10]
can be used to identify healthful recipes in a collection.
End-user interviews have shown that the perceived cooking
complexity of a recipe is strongly determined by the
number of ingredients and the number of preparation steps.
The latter can be computed using a part-of-speech tagger by
extracting the number of imperative verbs. When analyzing
a large collection of recipes, less common ingredients and
preparation instructions (e.g. ‘blanch the green beans’) can
be identified. Further, user-centered research is required to
identify the best match between the assessed cooking
competence and the parameters that indicate a recipe’s
cooking complexity.
Having selected a set of meals that fit the user’s needs and
skills, this subset can be further tailored to the user. When a
large and active community is present, a collaborative
filtering approach can be used to select the meal that best
fits the user’s profile. Alternatively, a conversational
recommender system may be used to identify specific
demands for a meal.
Personalized Support
Personalized guidance and support is based on the users
cooking competences, recipe history and selected recipes.
While preparing a meal the system provides the right
amount of support depending on the strengths and
weaknesses of the cook. If the cook is very competent in an
action no support is provided. However, if the user is
preparing an unfamiliar meal, containing actions identified
as weaknesses, more support and additional information
will be provided.
CONCLUSION AND DISCUSSION
This paper describes a context-aware recipe retrieval
system that guides and supports people to increase their
cooking competence. This approach follows Vygotsky’s
theory, where situated guidance provides scaffolding that
helps people to gradually improve their competences [11].
Instead of giving the user instructions what a healthy diet
contains, we offer the user a context-aware meal
preparation tool that empowers them to change their
cooking behavior.
ACKNOWLEDGMENTS
This project is part of Balance@Home, a 48 month research
project, partly funded by the support for training and career
development for researchers (Marie Curie) of the European
7th Framework Program.
REFERENCES
1. Furst, T., et al., Food Choice: A Conceptual Model of
the Process. Appetite, 1996. 26(3): p. 247-266.
2. Soliah, L., J. Walter, and D. Antosh, Quantifying the
impact of food preparation skills among college women.
College Student Journal, 2006. 40(4): p. 729-739.
3. Levy, J. and G. Auld, Cooking classes outperform
cooking demonstrations for college sophomores. Journal
of Nutrition Education and Behavior, 2004. 36(4): p.
197-203.
4. Brug, J., et al., Performing your original search, The
Impact of a Computer-Tailored Nutrition Intervention.
Preventive Medicine 1996. 25(3): p. 236 – 242.
5. Borra, S., et al., Developing Actionable Dietary
Guidance Messages: Dietary Fat as a Case Study.
Journal of the American Dietetic Association, 2001.
101(6): p. 678-684
6. Larson, N.I., et al., Food Preparation by Young Adults
Is Associated with Better Diet Quality. Journal of the
American Dietetic Association, 2006. 106(12): p. 2001-
2007.
7. Branca, F., H. Nikogosian, and T. Lobstein, The
challenge of obesity in the WHO European Region and
the strategies for response. 2007, Copenhagen: World
Health Organization.
8. Sidenvall, B., M. Nydahl, and C. Fjellström, Managing
food shopping and cooking: the experiences of older
Swedish women. Ageing and Society, 2001. 21(2): p.
151-168.
9. Pinxteren, Y.v., G. Geleijnse, and P. Kamsteeg.
Deriving a Recipe Similarity Measure for
Recommending Healthful Meals. in IUI 2011. 2011:
ACM.
10. Geleijnse, G., et al. Extracting Vegetable Information
from Recipes to Facilitate Health-Aware Choices. in
PERSUASIVE 2010. 2010. Copenhagen, Denmark.
11. Vygotsky, L.S., Mind in society: The development of
higher psychological processes. Vol. xi. 1978,
Cambridge: MA: Harvard University Press. 159.