Content uploaded by Pei-Luen Patrick Rau
Author content
All content in this area was uploaded by Pei-Luen Patrick Rau on Feb 19, 2016
Content may be subject to copyright.
Leap-Motion Based Online Interactive System
for Hand Rehabilitation
Zhe Liu
1(&)
, Yingzhi Zhang
1
, Pei-Luen Patrick Rau
1
, Pilsung Choe
2
,
and Tauseef Gulrez
2
1
Department of Industrial Engineering, Tsinghua University, Beijing, China
{zheliu1992,zhangyz.0208}@gmail.com,
rpl@mail.tsinghua.edu.cn
2
Department of Mechanical and Industrial Engineering,
Qatar University, Doha, Qatar
pschoe@gmail.com, gtauseef@qu.edu.qa
Abstract. Insufficient recognition to hand afunction caused by overwork,
injury and geratic complications leads to inadequate auxiliary for hand reha-
bilitation. Patients’rehabilitative training is usually limited to rehabilitation
center in hospitals, leaving their time at home inefficient for precise recovery.
In this paper, we introduced an online interactive system for hand rehabilita-
tion based on Leap Motion controller. We established this system for doctor, on
which they can prescribe patient to imitate standard exercise motion and get
automatic feedback, such as score, according to similarity, thus enhance reha-
bilitation effect. In pilot study, we recruited 4 rehabilitation doctors and 8 patients
to investigate core requirements for rehabilitation and then developed this system
based on their expectation. After briefly presentation of the first version prototype
to doctors, we got evaluation showing that the Leap-Motion-Based interactive
system for hand rehabilitation can be effective for better independent training,
and designating a direction for future work as well.
Keywords: Hand rehabilitation Natural Free-Hand interaction Leap motion
controller Hand tracking
1 Introduction
Hands, as the most dexterous part of our body, are of vital importance to our everyday
life. However, since hands are extensively used in nearly all tasks, they are exposed in
more dangerous environment than any other parts. Overwork, injury and geratic
complications, such as stroke can all cause hand afunction, totally or partially, which
directly diminish the quality of life. According to the statistics from World Health
Organization (WHO) 2014, rehabilitation and palliative care has not been given enough
attention and essential basic equipment is still not available everywhere, not only in
most developing countries, but also in some developed countries [1], thus many
patients of hand afunction cannot be provided prevention, diagnosis or treatment.
In addition, unlike other diseases, hand impairment and afunction require sufficient
patient’s exercise besides doctor’s treatment. Exact motion exercise can profoundly
help with patient’s rehabilitation [2]. However, currently there is no other method than
©Springer International Publishing Switzerland 2015
P.L.P. Rau (Ed.): CCD 2015, Part II, LNCS 9181, pp. 338–347, 2015.
DOI: 10.1007/978-3-319-20934-0_32
doctor’s judgment telling patients what to exercise and how they exercise, and patients
only exercise their finger and thumb by repeatedly fisting or picking up small items
during their long time staying at home, without doctor’s instructions. Thus, there is a
potential demand of designing a hand rehabilitation system to utilize patients’time at
home to exercise accurate motion training.
The key procedure of designing such an exercising system is to reproduce the
training atmosphere in hospital for patients who need hand rehabilitation [3]. Previous
studies [4,5] have shown that following and imitating are effective in motivating
training and increasing exercising accuracy. We used the user-centered design
approach to innovate an interactive rehabilitation system for patients to encourage their
exercises at home and meanwhile provide them with direct instructions. Their finger
movements above the Leap Motion sensor are tracked and projected to a computer
screen where they can see standard motions and their own motions at the same time.
The Leap Motion-based rehabilitation system, both easy and efficient to exercise, is
considered to be very helpful in motivating patients to exercise at home as well as
correcting their inaccurate motions.
This article firstly introduces an analysis of interviews we conducted with patients,
doctors. We collected their opinions and preferences for both real and interactive
exercise to generate a list of requirements which is used in the later design of the
system. Shortcomings of current indoor sports equipment are summarized with a brief
introduction of the Leap Motion-based system in Sect. 4. A detailed illustration of the
design of the rehabilitation system is then presented. Some improvements based on a
usability test are suggested. Discussion and conclusions are summarized in the last
section.
2 Background and Related Work
Generally, hand rehabilitation, which focuses on recovery of both strength and nim-
bleness [2,3], causes for professional training in rehabilitation center and long-term
persistent self-exercise. According to present medical care situation in China, most
patient can only afford one or two treatments in hospitals per week, which is consid-
erably insufficient for rehabilitation. Improving the efficiency of patients’self-exercise
at home is a key method to reduce rehabilitation time and increase rehabilitation effect.
The human-computer interaction community has made attempts and carried out
researches to enhance hand rehabilitation with HCI technologies. Khademi, M. et al. [6]
modified the game “Fruit Ninja”with Leap Motion controller for patients with stroke to
practice arm and hand. Boulanger et al. [8] built a game, as well, using Microsoft
Surface’s hand position. These games and researches involve new or existing interaction
games and adapt them into hand control mode to encourage rehabilitation. Another
example of hand rehabilitation is an interactive glove proposed by Hallam et al. [9]to
help patients with stroke with hand reuniting. There is also another glove-based treat-
ment system called HandTutor [10] focusing on finger rehabilitation. The above
researches show examples about tangible and intangible interaction rehabilitation
designs. Yet, rehabilitation system using free-hand interaction for patients’self-exercise
and introducing doctor’s instruction still remains to be investigated.
Leap-Motion Based Online Interactive System for Hand Rehabilitation 339
3 User Requirement Gathering: Interview
System design was started by an interview of 4 doctors and 8 patients, who need hand
rehabilitation, in a Grade III-A Hospital in Beijing, China, which is a top grade hospital
according to Chinese grading system and runs fully functional rehabilitation center.
3.1 Interview with Doctors
To study the general procedure of hand rehabilitation and know better about hospital’s
demand for interactive system, we conducted a series of interviews with 4 doctors of
rehabilitation in January 2014. 3 diagnosticians and 1 physical therapist participated to
answer a list of 8 questions about their current treatment procedure and concerns they
had when carrying out therapeutic scheme. They are each in charge of diagnosing
injured parts and severity, designing rehabilitation project, and assisting rehabilitation
training.
During the interviews, we mainly investigated their common cases, general pro-
cedure, usual training motions and equipment, and opinions about self-exercise at
home. Since doctors’instructions will be introduced into this rehabilitation system, we
also surveyed their preference about operation process and interface about online
system. Additional concerns about rehabilitation they had were also included in gen-
erating a list of requirements we need to consider in designing the system.
They introduced the whole process of rehabilitation and gave their ideas about in
what stage they hope auxiliaries can help. They also showed around the rehabilitation
center about the common equipment and standard training plans. Besides, they stated
that, according to their clinical experience, flexibility recovery was far more important
than strength recovery, which most rehabilitation equipment failed to help, and that
they regarded patients’time at home was not effectively used for rehabilitation, because
they could only repeat simple motions, such as clenching fist, for fear that they
exercised wrong without professional instructions and feedback. As for the system
design, they mentioned the existing prescription system since they were quite familiar
to that and suggested the rehabilitation system to follow that procedure and be added to
the existing system.
3.2 Interview with Patients
We also randomly recruited 8 patients in the rehabilitation center who were under
treatment of those doctors. The 8 patients—consisted of two elderly people suffering
from geratic complications and six other patients who injured their hands.
During the investigation, we asked about their rehabilitation circumstances,
including the duration and frequency they came to hospital, their exercise at home, and
their attitude towards independent training and assisting auxiliary. This investigation
consists of more than 16 open-ended questions. Since the online system requires user to
operate by gestures, we also surveyed their IT background and experience of using
computers and interactive devices. Additional concerns about rehabilitation they had
340 Z. Liu et al.
were also included in generating a list of requirements we need to consider in designing
the system.
As identified, most patients regard their exercise at home useless and usually cannot
maintain a tight and long-term exercise schedule because they do not set regular time
for simple training and often forget to exercise daily. Instead of exercise at home, they
trust doctors much more, but most of them cannot afford a whole course with doctor’s
instruction. Generally, they spent the first few weeks at hospital, having professional
training every day, yet after they rehabilitate to some degree, they move back home and
visit hospital one or two times every week considering time and expense. All of them
take positive attitude towards self-exercise equipment, saying that they would like to
try such system if doctor permits. As for the type of training, they claimed that they
prefer a series of training lessons which each focus on one part of hand, so that they can
pertinently exercise according to their disease. When the interface and operation design
was mentioned, they worried about whether their IT background could handle the
system, since they may cannot use their finger flexibly and operate the mouse. They
further added a requirement to play some similar game because they hope to increase
entertainment.
3.3 User Requirements
Results from the interview shed light on the demand of a new system for online hand
rehabilitation with doctors’instructions. All subjects showed interests in the idea we
put forward about a series motions for exercises at home and were willing to have a try.
They expected to have an interaction device of good usability, great efficiency, high
safety, and social connectivity at an affordable price. A list of user requirements for the
hand rehabilitation is summarized as below.
•The system should have adequate but accurate exercises to be effective, but the
duration of each exercise motion should be limited to avoid overtiredness.
•The system should be easy to start with and self-explainable. It should be easy for
elderly people and people without abundant IT knowledge to understand.
•The system should apply gesture to interaction with computers considering the
patients’low level of control to their fingers.
•The system should provide specific training motions for exercise need for different
parts of hand. Different motions should enable patients to exercise different hand
parts, especially their fingers.
•The system should encourage patients to maintain a clear schedule by doing hand
rehabilitation every day. And if patients fail to follow the schedule, the system
should remind them and contact doctors to adjust their rehabilitation plan.
•The system should give clear feedbacks to patients for them to correct their gestures
and improve, and record their performance for themselves and their doctors to
check.
•The system would be better if social connectivity is provided between patients.
•The system should allow doctors to check their patients’performance and give
instructions easily and synchronously.
Leap-Motion Based Online Interactive System for Hand Rehabilitation 341
•The system should provide convenient access for doctors to manage and monitor
their patients.
4 Improvement of Existing Hand Rehabilitation Equipment
There are a large number of patients who need hand rehabilitation in the word today
according to status from WHO, and attentions on equipment for that are still inade-
quate. Currently, products designed for hand rehabilitation are limited, which we
proposed that with the help of some well-designed device and systems, patients can be
instructed and encouraged to rehabilitate safely, correctly and, most important, effi-
ciently by themselves.
4.1 Shortcomings of Existing Equipment
Current hand rehabilitation equipment, which consists of two types: wearable devices
and interactive video games, both have their own focuses. Thus when used by doctors
for assistance, they could have several inherent drawbacks as follows.
•Wearable devices, such as hand dynamometer, and gloves (e.g. MusicGlove [12],
Gloreha [11] as shown in Fig. 1) consist of mechanical parts and are fixed on hands
by elasticity or else, which may all add extra compression and cause reinjury.
•Interactive video games, for example Khademi’s Ninja game with Leap Motion
controller, focus only on encouraging users to exercise their hand, paying no
attentions on precise training motions, while nimbleness recovery which is of equal
importance causes for precise exercise.
•Both of these products are designed for patients to exercise all by themselves.
However, doctors’opinions and instructions may give much help for rehabilitation,
which should be taken into account.
Fig. 1. Product demos of musicglove snd gloreha
342 Z. Liu et al.
4.2 Improvement Using Leap Motion Controller
The Leap Motion controller is an excellent sensor which can tracking hands and fingers
with high speed and precision [13]. It enables a series of operation in interaction with
computers. These unique advantages could improve current hand rehabilitation prod-
ucts in following points.
•The Leap Motion Controller enables doctors to record standard exercise motion
beforehand by tracking their example motions through its camera and IR sensors
without contact, which can be shown to patients and instruct their self-training.
•Besides beforehand recording, the Leap Motion controller could also real-time
process the data it tracks. By comparing real-time data with recorded data, users’
motions can be judged whether matching with the standard motions or not.
As a result, the Leap Motion controller has been chosen to realize most of the
requirements doctors and patients proposed, because of its well performance in tracking
users’hand, as well as sending and processing real-time hand data. Users only need to
move their fingers above the Leap Motion controller. The prototype system refers a set
if sample codes in Leap Motion SDK, with 3D hands modeling and displaying.
5 Prototype Design and Development
5.1 Prototype Apparatus
The online hand rehabilitation system consists of four parts: users (include patients and
doctors), a Leap Motion controller, a personal computer and a display screen. Con-
necting with the Leap Motion controller and display screen, the PC processes the
real-time data of hands obtained from the Leap Motion controller and displays on the
screen when the system is running. Patients only needs to sit in front of the PC and hold
hand above the Leap Motion controller in order that his or her hands can be captured
and tracked. Patients are able to wave his or her hands to select and click on the buttons
presented on the screen and operate the system, as shown in Fig. 2.
Fig. 2. Prototype apparatus
Leap-Motion Based Online Interactive System for Hand Rehabilitation 343
5.2 System Design
This rehabilitation system has special designs for patients with hand afunction to help
them with hand rehabilitation and communicate with their doctors.
As the start of the whole rehabilitation process, doctor check patient’s hand situ-
ation and have a basic diagnosis. Then instead of writing a prescription and teaching
patients how to exercise, doctor log on his or her account and add a new patient’s
document under his account. (Of course, if this patient is come for a return visit, doctor
can open his or her existing document and work on that instead of creating a new one.)
Doctor needs to fill in patient’s basic information and diagnosis before he gives pre-
scription. He can choose among all the exercise motions, which have been recorded by
Leap Motion and saved in the system, for patients to exercise with instructions about
frequency and duration, etc., like prescriptions about taking medicine. After all these
operations end, doctor will generate an account for patient, with a unique username and
password, and rent a Leap Motion controller to patients. The main interface is shown in
Fig. 3, where doctors can add new patients and enter certain patient’s record. As for the
patient’s record, the interface arranges as shown in Fig. 4with patient’s basic infor-
mation, diagnose, prescription and score for each exercise. Here, doctor can add
comments as well.
Fig. 3. Doctor’s main interface
Fig. 4. Patient’s record interface
344 Z. Liu et al.
Then comes to the patient’s self-exercise. Patients can open the website and login
with the username and password which is given by their doctors. After a basic cali-
bration with Leap Motion, patients enter their homepage, as shown in Fig. 5where they
can choose to exercise with standard motion, view their score and play interactive
games which encourage them to move their finger and wrist. When entering into the
exercise part, as shown in Fig. 6, patients can see the standard motion, prescribed by
their doctor, presents on the left for them to follow and at the same time, their real
motion will present on the right with feedbacks showing where to improve. Moreover,
a score based on the similarity between the standard motion and real one will appear on
the left, for patients themselves and their doctor to estimate their rehabilitation
situation.
Fig. 5. Patients’homepage
Fig. 6. Patients’exercise page
Leap-Motion Based Online Interactive System for Hand Rehabilitation 345
6 Prototype Evaluation: An Interview
In order to check whether this online system can satisfy patients’and doctors’demand
and improve hand rehabilitation’sefficiency, a prototype system was developed and
evaluated by 2 doctors in June 2014.
The evaluation interview consisted of three steps: a brief introduction to the usage
and design of the system, a short operation trial for doctors and a follow-up interview to
collect their feedback. Since we don’t have enough time to recruit more patients, this
evaluation was carried out only with doctors.
According to doctors’feedback, they both acknowledged that the system met their
demand and they believed it to be useful for hand rehabilitation. Also, they liked the
interface and structure of the website. Besides, they mentioned that the interaction with
Leap Motion and computer was easy for ordinary person, but they were not sure
whether patients with hand afunction could also interact in this way easily.
7 Future Plan and Discussion
Compared with current wearable devices and interactive video games for hand reha-
bilitation, this online system is better for patients in the following three aspects.
First, the online interactive system for hand rehabilitation add no extra compression
to patients. Since patients who suffer from hand afunction have limited control to their
finger and are easier to be wounded, wearable devices may accidentally hurt patients’
hands. However, using Leap Motion as a distant sensing device, patients are free to
interact with the computer with their hand without touch, thus avoiding further damage.
Second, the online interactive system provides precise rehabilitation training,
comparing with existing hand interactive game. Those task oriented game, for example,
Fruit Ninja, usually set a task for players to finish, and the game only focuses on the
whether the task is finished instead of how to finish it, thus cannot instruct accurate
motion. However, this system teaches patients the rehabilitation motion and monitors
their performance, at the same time, gives feedbacks for patients to correct their motion
themselves. In this method, precise rehabilitation training can be realized.
Third, via the online system, doctors and patients can communicate at any time,
instead of only during weekly visit back to the hospital. Patients’video and score will
be shared with their doctors and doctors’comments and instructions appear on patients’
interface as well. This has not only increased efficiency of the rehabilitation, but also
make full use of patients’time at home.
We have to admit that because of time limitation, we didn’t carry out evaluation
experiment with patients for long term test. Although the two doctors showed their
great interest in this system after a brief introduction to this system, there may be some
problems we cannot find until the patients try this. So for the next stage, we need to
cooperate with some local hospitals and get more detailed feedbacks which can point to
the directions we need to consider to further improve our rehabilitation system.
In conclusion, in this study, we carried out a “user-entered design”for patients in
developing a Leap-Motion-based online interactive hand rehabilitation system. User
requirements were collected from an interview with both doctors in hand rehabilitation
346 Z. Liu et al.
and patients who need rehabilitation. Based on these requirements, a prototype system
has been designed and developed with Leap Motion sensor. We briefly evaluated the
system with an introduction and presentation to two relevant doctors, and furthermore,
a more detailed experiment with patients need to be carried out. Compared with the
existing wearable devices and interactive games, this online system performs better in
safety, accuracy, efficiency in assisting patients with hand afunction in rehabilitation
training.
References
1. World Health Statistics (2014). http://apps.who.int/iris/bitstream/10665/112738/1/9789240692671_
eng.pdf?ua=1
2. Mackin, E., Callahan, A.D.: Rehabilitation of the Hand, pp. 312–317. Mosby, Miles (1978)
3. Boian, R., Sharma, A., Han, C., Merians, A., Burdea, G., Adamovich, S., Poizner, H.:
Virtual reality-based post-stroke hand rehabilitation. Stud. Health Technol. Inform. 85,64–
70 (2002)
4. Cruz, E.G., Waldinger, H.C., Kamper, D.G.: Kinetic and kinematic workspaces of the index
finger following stroke. Brain 128(5), 1112–1121 (2005)
5. Seo, N.J., Rymer, W.Z., Kamper, D.G.: Altered digit force direction during pinch grip
following stroke. Exp. Brain Res. 202(4), 891–901 (2010)
6. Khademi, M., Mousavi Hondori, H., McKenzie, A., Dodakian, L., Lopes, C.V., Cramer, S.
C.: Free-hand interaction with leap motion controller for stroke rehabilitation. In: CHI 2014
Extended Abstracts on Human Factors in Computing Systems, pp. 1663–1668. ACM, April
2014
7. Grünert-Plüss, N., Hufschmid, U., Santschi, L., Grünert, J.: Mirror therapy in hand
rehabilitation: a review of the literature, the St Gallen protocol for mirror therapy and
evaluation of a case series of 52 patients. Brit. J. Hand Ther. 13(1), 4–11 (2008)
8. Boulanger, C., Boulanger, A., de Greef, L., Kearney, A., Sobel, K., Transue, R., Sweedyk, Z.,
Dietz, P.H., Bathiche, S.: Stroke rehabilitation with a sensing surface. In: Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems, New York, NY, USA,
pp. 1243–1246 (2013)
9. Hallam, J., Whiteley, V.: Interactive therapy gloves: reconnecting partners after a stroke. In:
CHI 2011 Extended Abstracts on Human Factors in Computing Systems, New York, NY,
USA, pp. 989–994 (2011)
10. Eli Carmeli, S.P.: HandTutorTM enhanced hand rehabilitation after stroke. Physiother. Res.
Int. J. Res. Clin. Phys. Ther. 16(4), 191–200 (2011)
11. Gloreha –Hand rehabilitation glove. http://www.gloreha.com/index.php/en/gloreha-en
12. MusicGlove by Flint Rehabilitation Devices LLC: Home. https://www.flintrehabilitation.
com/
13. Leap Motion | Mac & PC Motion Controller for Games, Designs, & More. https://www.
leapmotion.com/
Leap-Motion Based Online Interactive System for Hand Rehabilitation 347
