Visualisation of basic gesture types. 

Context in source publication

Context 1

... gestures typically from the face or head of the user. An example is presented in [3]: The control of an electric wheelchair is realised using a web cam fixated to it and allowing face detection, tracking and head gesture recognition in real time. Recognised gestures are then mapped to mo- tion control commands, such as speed up, slow down, or turn left/right. However, similar to voice-based systems, vision- based approaches raise privacy concerns and thus their social acceptance is limited. Moreover, both solutions have to deal with a very diversified, dynamically changing environment, posing difficult practical challenges. A promising alternative to the above systems is presented in [8]: A gesture controlled wheelchair user interface for people with motor impairments based on electromyography (EMG) and electrooculography (EOG). Eyebrow muscle activity from EMG signals is used for directional control commands, while EOG is used to ad- just speed. Although the system is wearable and rather unobtrusive (electrodes are integrated into a headband), the control interface itself is cumbersome and not natural. We present a novel sensing modality for hands-free control interfaces, addressing the shortcomings of the above mentioned approaches. Our system is based on the active capacitive sensing principle: Capacitive electrodes are integrated into a textile neckband. With our wearable system it is possible to continuously monitor the wearer’s head movements in an unobtrusive way. We investigate the potential of this technology to distinguish various head gestures and head postures. This is motivated by application scenarios such as the above mentioned electric wheelchair control interface, where recognised gestures or postures can be mapped to control commands. Moreover, we show that the capability of our system exceeds that of common solutions where usually only a few gestures are recognised: Quantitative evaluation shows that we can reliably distinguish between 15 head gestures. Finally, it is reasonable to expect that the social acceptance of our system is higher than that of, for example, voice or camera-based solutions. Overall, the paper provides the following main contributions: 1. We present a novel sensing modality for hands-free gesture controlled user interfaces, based on head gesture and head posture recognition. 2. To explore the capability of our system, we carried out a study involving 12 subjects and recording data from 15 head gestures and 19 different head postures. We present a quantitative evaluation based on this dataset. Our system is based on the active capacitive sensing principle, introduced recently for human activity recognition [1]. A capacitor consists of a dielectric material between two conductive planes. It can store energy in an electric field, the amount depending on thickness and composition of the dielectric material, and can be measured electrically. By plac- ing the planes around the human neck, its inside becomes the dielectric. Movement of muscles, tendons, blood vessels and other tissue can then be reflected as an electric value. Head movement consists mostly of the three main degrees of freedom of the cervical vertebrae (Figure 1). In order to infer different movements (which results in a broad range of changes inside the neck) multiple, carefully selected capacitive electrode positions are necessary. We propose the 4-electrode setup as depicted in Figure 1, where each pair of channels can reflect asymmetrical changes due to difference calculation. The implementation of the proposed scheme (hardware prototype) is shown in Figure 2. The four capacitive sensors, in form of conductive textile, are embedded in a non-conductive textile, building a stretchable neckband as proposed in [1]. Two sensors are centred directly above and under the larynx, the other two to the left and right on the neck. Electric ground is connected to the neck over partly exposed conductive textile and serves as a common second plane. The connected circuit boards are for amplification, noise reduction, analog- to-digital conversion and wireless data transmission [1]. From the monitored head movements, the goal is to distinguish between a set of gestures and postures, which then can be mapped to commands in a hands-free control interface. In order to evaluate the potential of our approach, we conducted two sets of experiments with 8 male and 4 female participants (aged from 22 to 35 years, with neck perimeter ranging from 29 cm to 42 cm). Only one of the test subjects was involved in designing the experiments. For each experiment, we adjusted the tightness of the neckband so that the participants felt comfortable and were not restricted in their head movement. For recording and automatic labelling of the data, we developed For the first set of experiments, subjects were asked to sit on a chair in front of a computer screen and perform 15 different head gestures upon timed screen and voice commands. Figure 3 shows the basic gesture types, chosen as feasible head movements for natural user interfaces. Each of the first three (“nod”, “tilt” and “look”) was performed in two directions. Moreover, we defined two movement types for these gestures, namely “slowly once” and “double”. We referred to the single and double click on a computer mouse as analogy. Therefore, 12 gestures ( e.g. “slowly look left once” or “double nod down”) were defined this way. In addition, we defined the gestures “circle head (counter) clockwise” and “double woodpecker move”, the latter resembling to the typical movement of a woodpecker. Participants were asked to carry out each gesture as they interpreted them, and return facing the screen once a gesture was performed in order to wait for the next command. Every subject recorded each of the 15 gestures 11 times in random order, during a single session. Figure 4 shows typical example signals from the 4 channels. For the second set of experiments, we placed a chair in fixed distance facing a wall with a grid of 3x5 points, named “A1- A5”, “B1-B5” and “C1-C5”. These points were spread over the average comfortable and typical facing range of a per- son sitting on the chair. Strips on the ceiling, the floor, the far left and far right marked extreme angles, defining four additional points for subjects to look at. The complete grid structure with approximate facing angles is depicted in Figure 5a. During this experiment participants were asked to look at the point called out via voice command. This was supported by a laser pointer mounted on top of the subject’s head, the participant had to hold the laser point until the next command. The laser pointer was calibrated to the subject be- fore experiments, and ensured reproducible head postures via well defined looking points. The entire experimental setup is shown in Figure 5b. Two sessions, divided by a break, were recorded with each subject. In every session, each of the 19 Due to the well defined experiment procedure and scripted data collection with automatic labelling, the start of each gesture is reliably marked in the dataset. In addition, the end of each gesture was marked in a semi-automatic way, thus resulting in isolated gesture segments (four examples are shown in Figure 4). Considering recorded data from head postures, automatic labels mark the start of the transition to the next posture ( cf. . Figure 6). We used these marking points to iso- late posture segments. Moreover, the end point of the transition in each posture segment was marked in a semi-automatic way. Relevant features ( e.g. mean, see below) were calculated by leaving the signal transition part out. A large number of approaches exist to recognise gestures from isolated signal segments. Most common algorithms use time domain matching or modelling such as hidden Markov model (HMM) [5], dynamic time warping (DTW) [4] or methods based on data dictionaries [2]. Another common tool used for gesture recognition relies on first extracting features from each signal segment and then use a classifier on the feature set. This latter method showed promising results recently, also in comparison to HMM or DTW [6, 9]. Therefore, we applied this approach in our evaluation. We defined the feature set based on visual inspection of the signal form and based on extracted features in related work ( cf. e.g. [9]). For head postures, signal level and its difference between certain channels includes the major informa- tion content ( cf. Figure 6). Therefore, the feature set for this task comprises of the features mean, min/max, ratio and difference between each pair of channels. For the head gesture recognition task, different signal shapes ( cf. Figure 4) need to be distinguished. Therefore, we computed various features in both time and frequency domain from each segment, such as mean, standard deviation, correlation between each pair of channels, spectral entropy, or dominant frequency. As for classification, we used decision tree-based classifiers since they automatically select the most relevant features from a large feature set. We compared three ensemble learners: bagging, boosting and random forests, each with 200 iterations. For comparison reasons, we also included a single decision tree and a kNN classifier in our evaluation ( k = 3 , all algo- rithm parameters were determined in preliminary studies on the training dataset). For quantitative results we performed user dependent 10-fold cross-validation for each user. Since both the head gesture and posture recognition task represents balanced classification problems, we used accuracy as performance measure. Figure 7 shows the mean accuracy of the 5 classifiers on both tasks, min/max values indicate user-specific variation. Overall it is clear that the decision tree-based ensemble learners perform best, with bagged trees achieving the highest accuracy. Figure 8 shows the confusion matrix of recognising the 15 head gestures ...