Performance Comparison of Manual and Touch Interface using Video-based Behavior Analysis

Abstract

The objective of this study is to quantitatively incorporate user observation into usability evaluation of mobile interfaces using monitoring techniques in first- and third-person points of view. In this study, an experiment was conducted to monitor and record users' behavior using Ergoneers Dikablis, a gaze tracking device. The experiment was done with 2 mobile phones each with a button keypad interface and a touchscreen interface for comparative analysis. The subjects included 20 people who have similar experiences and proficiency in using mobile devices. Data from video recordings were coded with Noldus Observer XT to find usage patterns and to gather quantitative data for analysis in terms of effectiveness, efficiency and satisfaction. Results showed that the button keypad interface was generally better than the touchcreen interface. The movements of the fingers and gaze were much simpler when performing given tasks on the button keypad interface. While previous studies have mostly evaluated usability with performance measures by only looking at task results, this study can be expected to contribute by suggesting a method in which the behavioral patterns of interaction is evaluated.

Full text

Journal of the Ergonomics Society of Korea
Vol. 29, No. 4 pp.655-659, August 2010 DOI:10.5143/JESK.2010.29.4.655
Performance Comparison of Manual and Touch Interface
using Video-based Behavior Analysis
Chaiwoo Lee, Sangwoo Bahn, Ga Won Kim, Myung Hwan Yun
Department of Industrial Engineering, Seoul National University, Seoul, 151-742
ABSTRACT
The objective of this study is to quantitatively incorporate user observation into usability evaluation of mobile interfaces
using monitoring techniques in first- and third-person points of view. In this study, an experiment was conducted to monitor
and record users' behavior using Ergoneers Dikablis, a gaze tracking device. The experiment was done with 2 mobile phones
each with a button keypad interface and a touchscreen interface for comparative analysis. The subjects included 20 people
who have similar experiences and proficiency in using mobile devices. Data from video recordings were coded with Noldus
Observer XT to find usage patterns and to gather quantitative data for analysis in terms of effectiveness, efficiency and
satisfaction. Results showed that the button keypad interface was generally better than the touchcreen interface. The
movements of the fingers and gaze were much simpler when performing given tasks on the button keypad interface. While
previous studies have mostly evaluated usability with performance measures by only looking at task results, this study can
be expected to contribute by suggesting a method in which the behavioral patterns of interaction is evaluated.
Keywords: User-observation, Usability, Mobile interface, Gaze-tracking
1. Introduction
User observation is a type of usability evaluation
technique that looks into the process of interaction to
identify behavioral characteristics, potential problems, and
usage patterns (Oh and Lee, 2004; Lee et al., 2006). Since
users' latent needs are not usually expressed clearly, user
observation is integral in usability evaluation (Jang, 2008).
For a more comprehensive evaluation of usability, this
study employs user observation and behavior monitoring
into usability evaluation of mobile interfaces.
Previous studies have mostly used third-person user
observation only, where the interaction is recorded from
outside the user. Park and Kim (2008) recorded their
usability testing of touchscreen mobile phones to define
errors. Kjelskov and Stage (2004) used video recording and
think-aloud method together. Lee et al. (2006) collected
data about mobile user interaction by video recording as
well, and identified common navigation paths and critical
incidents based to the results.
Usability evaluation based on first-person observation,
where recording is done by the users, is relatively difficult.
For effective first-person observation, gaze analysis has
gained attention from previous studies since a user's spatial
focus of attention has shown to be represented by the user's
eye movements (Schroeder, 1998; Nakamachi et al., 2007;
Sawahata et al., 2008). Goldberg and Kotval (1999)
developed metrics for usability evaluation of computer
interfaces based on information about gaze fixation and
scanpath complexity.
Previous studies were limited in that they incorporated
user observation at a level where only simple video
recording was done for qualitative analyses. Also, gaze
analysis has not been fully used despite its importance. This
study proposes a method for usability evaluation of mobile
Corresponding Author: Myung Hwan Yun
Address: 39-312, College of Engineering, Seoul National Univ., Daehak-dong, Kwanak-gu, Seoul, 151-744. Tel: 02-880-1403, E-mail: mhy@snu.ac.kr

656ChaiwooLee·SangwooBahn·GaWon Kim·MyungHwanYun JESK
interfaces with utilization of user observation techniques
from both first- and third-person points of view. Eye gaze
and finger movements were monitored closely and data were
gathered by coding video recordings. Also, for quantitative
evaluation, metrics were developed for factors of usability.
2. User observation tools
There are a variety of techniques for usability evaluation,
including in-depth interview, think-aloud protocol, scenario
building, cognitive walkthrough, heuristic evaluation, and
surveys (González et al., 2008). However, in conventional
methods such as usability testing and surveys, users often
do not speak well or express fully. Thus, user observation
is necessary for eliciting the detailed information needed.
Oh and Lee (2004) categorized user observation
techniques into 2 types: first-person and third-person
observations. Behavior is self-recorded in first-person
observation while it is recorded from an outside position in
third-person observation. Since first-person observation
gives understanding in a user's point of view while third-
person observation shows related environmental factors,
both views need to be considered.
In this study, both first- and third-person observations
were employed by using Ergoneers Dikablis, a gaze tracking
device. To process the video recordings and to gather
quantitative data, Noldus Observer XT, a video coding
software, was used.
3. Development of metrics
Attributes and factors of usability have been defined in
various ways by existing standards and models. However,
most of them include common attributes for defining
usability: effectiveness, efficiency and satisfaction. Defini-
tions for the three factors are described in Table 1.
Studies have focused on developing methodologies and
measurements for evaluating usability in terms of effec-
tiveness, efficiency and satisfaction. Quantitative metrics
are necessary in order to analyze data more objectively.
This study developed metrics for each factor, while also
adopting relevant measures. The metrics for effectiveness
and efficiency as shown in Tables 2 and 3, respectively,
were developed to produce results based on data from
monitoring finger and eye movements. For satisfaction,
relevant subjective criteria were organized for questionnaire
Tabl e 3. Metrics for evaluation of efficiency
Metric Description
Task completion time Total time to task completion
Deviation from optimal Number of extra actions taken
Number of fixations Total number of fixations
Scanpath length Total length of scanpath
Convex hull area Area of convex hull circumscribin
g
scanpath
Tabl e 4. Criteria for evaluation of satisfaction
Criterion Description
Preference The degree in which users prefer over others
Ease of use The degree in which users feel convenient
Perception of
interaction Rating of perceived process of interaction
Sense of control User's sense of control towards interactivity
Intuitiveness Perception on power of knowing o
r
understanding without cognitive effort
Physical discomfort Experience of physical discomfort in use
Enjoyableness Feeling of entertainment and enjoyment
Tabl e 1. Factors of usability (ISO 9241-11, 1998)
Factor Definition
Effectiveness The accuracy and completeness with whic
h
users achieve goals
Efficiency
The resources expended in relation to th
e
accuracy and completeness with which users
achieve goals
Satisfaction The freedom from discomfort, and positive
attitudes towards the user of the system
Tabl e 2. Metrics for evaluation of effectiveness
Metric Description
Task completion
rate
Ratio of tasks correctly completed in allotte
d
time
Error frequency Total number or errors made in task completion
Spatial accuracy Ratio of correct, on-target fixations to total

Vol.29,No.4.2010.8.31 PerformanceComparisonofManualandTouchInterfaceusingVideo‐basedBehaviorAnalysis657
as shown in Table 4.
4. Experiment design
An experiment was conducted for usability evaluation of
mobile interfaces based on user observation. Twenty people
(15 males and 5 females) with similar experiences and
proficiency with mobile devices and touchscreen interface
participated. With an eye gaze tracking and recording system,
video data were gathered as the subjects performed given
tasks.
4.1 Devices and tools
Two mobile phones with different control interfaces
were used for comparative evaluation. One had a button
keypad (LG-KH8000), and the other had a touchscreen
(SPH-W7700). The 2 mobile phones were similar in size,
and their measures are displayed in Figure 1.
Dikablis eye gaze tracking device was used for video
recording. The system has a headset that subjects can wear
while performing tasks, with 2 small cameras: an eye camera
and a field camera, as shown in Figure 2. The recordings
from the 2 cameras were then integrated using Dikablis
software. The movements of the pupil and the fingers were
clearly seen on the connected computer screen as shown in
Figure 3.
4.2 Process of experiment
All of the subjects performed the tasks in the same setting.
They wore the Dikablis headset and held the mobile phone
with both hands. A chin rest was given to eliminate noise
caused by excessive head movements.
Two types of tasks - number entry and menu selection -
were given to each subject to perform. For the number entry
task, a 10-digit number was given. For the menu selection
task, a sequence of items and menus were given verbally.
Both tasks were performed on all 2 devices, so each subject
performed 4 cycles of experiment. At the end of the
experiment, subjects filled out a questionnaire to evaluate
each interface in terms of satisfaction.
4.3 Gathering data
Video recordings were coded into quantitative data using
Noldus Observer XT, a software for accurate analysis of
observational data. Each subject's fingers and eye movements
were then sequentially recorded along with a corresponding
time dimension and description on behavioral characteristics,
as shown in Figure 4.
5. Analysis and results
5.1 Description of behavioral characteristics
Based on the coded information, the scanpaths of the
Figure 1. Devices used for the experimen
Figure 2. Components of Dikablis headset
Figure 3. Recording screen from Dikablis software

658ChaiwooLee·SangwooBahn·GaWon Kim·MyungHwanYun JESK
fingers and the eyes were drawn. The typical scanpaths are
illustrated in Figures 5 and 6, which revealed differences
between the 2 interfaces. Finger movements were much
simpler on button keypad in both tasks. On the touchscreen,
a larger number of unnecessary, inaccurate movements were
made. In terms of gaze, the length was longer on the button
keypad but the quantity was fewer.
5.2 Results
Coded data were used to produce results with the metrics
defined. Pairwise t-tests were done for comparison. Figures
7 and 8 summarize results from menu selection task. The
same tendency was found in the number entry task. Figure
9 shows results from satisfaction evaluation. The dark gray
bars show data for button keypad, and the light ones are for
touchscreen evaluation results.
A consistent result in which the button keypad is superior
in terms of usability was found in all of the metrics except
for intuitiveness and enjoyableness. With the button keypad,
tasks were done faster with more accurate movements,
fewer errors and less number of fixations. The touchscreen
required users to make unnecessary and longer movements
Figure 5. Summary of finger movements
Figure 7. Result of effectiveness evaluation
Figure 6. Summary of eye movements
Figure 8. Result of efficiency evaluation
Figure 9. Result of satisfaction evaluation
Figure 4. Coding video data using Observer XT

Vol.29,No.4.2010.8.31 PerformanceComparisonofManualandTouchInterfaceusingVideo‐basedBehaviorAnalysis659
spanning a larger area. The results showed that the button
keypad is generally better than the touchscreen in terms of
usability.
6. Conclusion
In the experiment, user behavior was closely monitored
by recording movements of the fingers and the eyes. By
coding the video data, quantitative analysis was conducted
to evaluate usability with metrics developed.
A comparative evaluation showed that the button keypad
is generally more effective, efficient and satisfactory than
the touchscreen. Also, observation of finger movements,
eye gaze analysis and questionnaire all showed consistent
results.
This study proposed a method for incorporating user
observation into usability evaluation of mobile interfaces.
It can be expected that the method of quantitative analysis
based on user observation can contribute by suggesting a
way for comprehensive evaluation of usability.
References
Goldberg, J. H. and Kotval, X. P., Computer interface evaluation using eye
movements: methods and constructs, International Journal of
Industrial Ergonomics, 24(6), 631-645, 1999.
Gonzalez, M. P., Lores, J. and Granollers, A., Enhancing usability testing
through datamining techniques: A novel approach to detecting
usability problem patterns for a context of use, Information and
Software Technology, 50(6), 547-568, 2008.
ISO 9241-11, Ergonomic Requirements for Office Work with Visual Display
Terminals (VDTs), Part 11: Guidance on Usability, International
Organization for Standardization, Geneva, 1998.
Jang, H., A study on patterns in the use of hands when handling with
screen-touch mobile multimedia devices, Unpublished Master's
Thesis, Kookmin University, 2008.
Kjeldskov, J. and Stage, J., New techniques for usability evaluation of
mobile systems, International Journal of Human-Computer Studies,
60(5-6), 599-620, 2004.
Lee, Y. S., Hong, S. W., Smith-Jackson, T. L., Nussbaum, M. A. and
Tomioka, K., Systematic evaluation methodology for cell phone user
interfaces, Interacting with Computers, 18(2), 304-325, 2006.
Nakamichi, N., Sakai, M., Shima, K., Hu, J. and Matsumoto, K., WebTracer:
A new web usability evaluation environment using gazing point
information, Electronic Commerce Research and Applications, 6(1),
63-73, 2007.
Oh, Y. S. and Lee, K. P., A comparative study of user observation methods
in actual mobile environment, Proceedings of the International
Conference of the Korean Society of Design Science, 2004.
Park, Y. J. and Kim, D. H., The influence of altering mobile phone
interface on the generation of mental model, Korean Journal of the
Science of Emotion and Sensibility, 11(4), 575-588, 2008.
Sawahata, Y., Khosla, R., Komine, K., Hiruma, N., Itou, T., Watanabe, S.,
Suzuki, Y., Hara, Y. and Issiki, N., Determining comprehension and
quality of TV programs using eye-gaze tracking, Pattern Recognition,
41(5), 1610-1626, 2008.
Schroeder, W., Testing web sites with eye-tracking, User Interface
Engineering Newsletter, 1998.
Author Listings
Chaiwoo Lee zep73@naver.com
Highest degree : M.S., Department of Industrial Engineering,
Seoul National Univ.
Position title : Researcher, Human Interface Systems Lab,
Department of Industrial Engineering,
Seoul National Univ.
Areas of interest : Usability, User-Centered Design
Sangwoo Bahn panlot@gmail.com
Highest degree : M.S., Department of Industrial Engineering,
Seoul National Univ.
Position title : Ph.D. Candidate, Department of Industrial
Engineering, Seoul National Univ.
Areas of interest : Affective Quality, UI Design & Evaluation, Usability
Ga Won Kim hietoile@gmail.com
Highest degree : B.S., Department of Industrial Engineering,
Seoul National Univ.
Position title : M.S. Candidate, Department of Industrial
Engineering, Seoul National Univ.
Areas of interest : Product Design, User-Centered Design
Myung Hwan Yun mhy@snu.ac.kr
Highest degree : Ph.D. Industrial and Manufacturing Engineering,
Penn State University
Position title : Professor, Department of Industrial Engineering,
Seoul National Univ.
Areas of interest : Human Factors, UCD, HCI, Kansei-Engineering
Date Received : June 6, 2010
Date Revised : July 22, 2010
Date Accepted : July 22, 2010