Fig 2 - uploaded by Lear Du
Content may be subject to copyright.
Source publication
Virtual reality requires high levels of interaction with the user, a type of human computer interaction. Interactions that match the way humans usually interact with their surroundings should improve training effectiveness. A 3D hand gesture based interface allows users to control the position and orientation of 3D objects by simply moving their ha...
Context in source publication
Context 1
... are many devices that provide hand pose data such as Intel RealSense, Leap Motion, Kinect etc. Due to the accuracy of Leap Motion and its compatibility with the Oculus Rift [8], we chose the pose data provided by the Leap Motion for gesture recognition. The system set-up is shown in Fig. 2 and the objects that the user sees are indicated in the box. A Leap Motion hand tracking device was mounted on an Oculus Rift VR headset using a custom mount that oriented the Leap Motion device to point 13° below a line perpendicular to the headset surface. The task involved manipulating a virtual hand to grab a virtual dice from a ...
Similar publications
As technology matures, human-computer interfaces have changed to meet the needs of interacting with more complex systems in user-friendly ways. Gesture-based interfaces, a type of natural user interface (NUI), allow users to use their bodies to interact with computers or virtual/augmented reality (VR/AR) and offer a more natural and intuitive user...
Citations
... For in-position, Leng et al. [11] showed the utilization of mid-air signals to collaborate with music in a virtual vivid climate. Additionally, Lin et al. [12] investigated different hand motions to control various articles in VR and examined the effect of stances on the related throughput. Yan et al. [13] showed a clever methodology for recovering items in VR utilizing hand signals like genuine getting. ...
p>User interfaces (UI) and menus in virtual reality (VR), which frequently replicate traditional UI for computers and smartphones, are not created factoring for individuals with low eyesight as they demand accurate pointing and good vision to engage effectively. As an alternative method of user interaction with UI, using gestures can be recommended. Comparing gesture-based interaction with the conventional point-and-click technique for changing system settings like volume, brightness, and window manipulation in order to test this hypothesis is employed. Accessibility, spatial awareness, and precision for those with low vision while lowering cognitive load and enhancing immersion for all users can be improved by leveraging gestures. The objective of the research work is to explore the framework of Gesture Elicitation in VR environments for users with low vision. In this research work the usage of gestures as a more effective and immersive means of interacting with menus, which will not only enhance the experience of normal VR users but also drastically reduce the friction experienced by those with visual impairments is proposed. User studies demonstrate a noticeable improvement in the aforementioned areas, with faster work completion times, more immersion, and better user satisfaction.</p
... In recent years, humans not only interact with their hands in the real world, but increasingly in different types of digitized multimedia environments, such as virtual or augmented reality [1][2][3] . Since hand movements serve as a crucial interaction interface, to make use of them in any of such scenarios as well as in digitally transmitted remote human-machine or human-human interactions through the Tactile Internet (TI) 4 , hand kinematics need to be well tracked 5,6 and in some cases modeled [7][8][9] , or predicted [10][11][12] . Although several useful databases of hand movements [13][14][15][16][17] exist, most come with certain limitations. ...
The Tactile Internet aims to advance human-human and human-machine interactions that also utilize hand movements in real, digitized, and remote environments. Attention to elderly generations is necessary to make the Tactile Internet age inclusive. We present the first age-representative kinematic database consisting of various hand gesturing and grasping movements at individualized paces, thus capturing naturalistic movements. We make this comprehensive database of kinematic hand movements across the adult lifespan (CeTI-Age-Kinematic-Hand) publicly available to facilitate a deeper understanding of intra-individual–focusing especially on age-related differences–and inter-individual variability in hand kinematics. The core of the database contains participants’ hand kinematics recorded with wearable resistive bend sensors, individual static 3D hand models, and all instructional videos used during the data acquisition. Sixty-three participants ranging from age 20 to 80 years performed six repetitions of 40 different naturalistic hand movements at individual paces. This unique database with data recorded from an adult lifespan sample can be used to advance machine-learning approaches in hand kinematic modeling and movement prediction for age-inclusive applications.
... When implementing grab gestures, the application needs to diferentiate between grab, rotation, movement and release [21]. Including press and drag provides the user with a natural object interaction [6]. ...
... The system supports multiple interaction tasks but it has relatively limited interaction accuracy (89.86%). Lin et al. [32] proposed hand gesture-based interaction in VE. They performed object grab, rotate, and release operations using hand gestures. ...
Simple and natural interaction has a vital role in any realistic virtual environment (VE). This research proposes a set of lightweight gesture-based techniques for interaction in VEs with a focus on high accuracy, performance, and usability. The proposed techniques use a single fingertip pose and state for object/task selection, translation, navigation, rotation, and scaling. Four different techniques are proposed for interaction, i.e., MSGE (Menu-based task selection and gesture-based task execution), GSGE (Gesture-based task selection and gesture-based task execution), SGTE (Single gesture for task selection and execution), and TSGE (Time slice-based task selection and gesture-based task execution). Keeping in mind the concept of re-usability, the index-tip spatial position is used for task operation in all techniques. For experimental evaluation of the proposed techniques, a VE is designed in Unity3D, while interaction is carried out using the Leap Motion controller. The experimental study was conducted with forty (40) volunteer participants and two experts (authors). Experimental results show improved accuracy for TSGE (participants 97.22%, and experts 97.22%) as compared to others (participants: SGTE 95.55%, GSGE 94.44%, and MSGE 92.75%, experts: SGTE 94.44%, GSGE 94.44%, and MSGE 91.67%). Similarly, the results show high task performance for TSGE (participants 112.9 seconds, SD 5.3, experts 101.75 seconds, SD 3.3) as compared to others (participants: SGTE 117.2 seconds, SD 5.7, GSGE 121.8 seconds, SD 8.0, and MSGE 126.7 seconds, SD 12.9 and experts: SGTE 107.0, SD 5.7, GSGE 113.25, SD 3.5, and MSGE 122.0 seconds, SD 3.6). In addition, usability analysis shows high usability for the proposed interaction techniques, i.e., TSGE (SUS score 98.5), SGTE (SUS score 95.75), GSGE (SUS score 95.25), MSGE (SUS score 94.75). Furthermore, a comparative study with state-of-the-art interaction techniques showed a high accuracy rate, multiple tasks, and reusability support, use of easy to learn and use and fewer features-based gestures (fingertip gestures), and multiple interaction techniques (four techniques) support for the proposed techniques.
... Hand gesture recognition methods have a significant number of applications, such as controlling unmanned aerial vehicles (UAVs) (Ma et al., 2017), interacting with autonomous vehicles (Holzbock et al., 2022), recognizing sign language (Pigou et al., 2015), or manipulating objects in virtual reality environments (Lin et al., 2017a) or in 3D design tools (Wang and Bao, 2007). In the case of applications such as object manipulation, it is necessary to track the pose of hand and fingers, whereas other applications have to classify the gesture into certain categories, which is the case of sign language recognition. ...
Existing research for the assistance of visually impaired people mainly focus on solving a single task (such as reading a text or detecting an obstacle), hence forcing the user to switch applications to perform other actions. This paper proposes an interactive system for mobile devices controlled by hand gestures that allow the user to control the device and use several assistance tools by making simple static and dynamic hand gestures (e.g., pointing a finger at an object will show a description of it). The system is based on a multi-head neural network, which initially detects and classifies the gestures, and subsequently, depending on the gesture detected, performs a second stage that carries out the corresponding action. This architecture optimizes the resources required to perform different tasks, it takes advantage of the information obtained from an initial backbone to perform different processes in a second stage. To train and evaluate the system, a dataset with about 40k images was manually compiled and labeled including different types of hand gestures, backgrounds (indoors and outdoors), lighting conditions, etc. This dataset contains synthetic gestures (whose objective is to pre-train the system to improve the results) and real images captured using different mobile phones. The comparison made with nearly 50 state-of-the-art methods shows competitive results as regards the different actions performed by the system, such as the accuracy of classification and localization of gestures, or the generation of descriptions for objects and scenes.
... The emergence of low-cost and high-precision interaction devices, such as Leap Motion (Bachmann et al. 2018) based on computer vision motion interaction, as well as Oculus (Ziak et al. 2017), HTC Vive VR (Niehorster et al. 2017), and data gloves based on wearable devices, has led to a tremendous development in virtual-hand interaction techniques (Al-Shamayleh et al. 2018). Currently, 3D interaction techniques have been implemented to track and recognize a series of gestures such as pointing (Schwind et al. 2018), selecting (Yu et al. 2018), sliding (Lin et al. 2017), pinching (Pfeuffer et al. 2017), rotating (Perng et al. 2020), shaping (Benko et al. 2016), grabbing (Lin et al. 2017), traveling, and zooming (Koutsabasis and Vogiatzidakis 2019). This type of NUI, using the body as a medium, provides a more intuitive and comfortable user experience (UX). ...
... The emergence of low-cost and high-precision interaction devices, such as Leap Motion (Bachmann et al. 2018) based on computer vision motion interaction, as well as Oculus (Ziak et al. 2017), HTC Vive VR (Niehorster et al. 2017), and data gloves based on wearable devices, has led to a tremendous development in virtual-hand interaction techniques (Al-Shamayleh et al. 2018). Currently, 3D interaction techniques have been implemented to track and recognize a series of gestures such as pointing (Schwind et al. 2018), selecting (Yu et al. 2018), sliding (Lin et al. 2017), pinching (Pfeuffer et al. 2017), rotating (Perng et al. 2020), shaping (Benko et al. 2016), grabbing (Lin et al. 2017), traveling, and zooming (Koutsabasis and Vogiatzidakis 2019). This type of NUI, using the body as a medium, provides a more intuitive and comfortable user experience (UX). ...
The virtual-hand interaction technique is a common input technique in virtual environments (VEs). The current application of virtual-hand interaction in VEs lacks specialized objective evaluation methods, making it difficult to establish a systematic functional goal orientation. To achieve a quantitative evaluation of the virtual-hand interaction system in VEs, we developed the modified evaluation method of goals, operators, methods, and selection rules (H-GOMS). The evaluation model contains five modules: analysis, decomposition, configuration, acquisition, and evaluation. To build the H-GOMS model, the relevant temporal parameters of operators in VEs were measured, and interactive rules were formulated for the configuration module. In addition to establishing the H-GOMS evaluation model, this paper demonstrates the development of a performance evaluation software (HI2ET) based on the Unity engine. We realized automatic retrieval and identification of the interactive behavior information from the software and applied the H-GOMS model algorithm for real-time visualization of the interactive process. The proposed method, with modeling of interactive tasks based on expert users, enables feasible and generally quantifiable performance evaluation for virtual-hand interaction systems in VEs.
... The use of hand tracking for training has many forms in VR, with a hand gesture design for specifying commands [14]. However, hand tracking has often been used in combination with other devices or techniques to detect hand gestures [15], including Leap motion, data glove, marker-based, and inside-out, which will have the following related tasks. ...
Virtual Reality (VR) technology is frequently applied in simulation, particularly in medical training. VR medical training often requires user input either from controllers or free-hand gestures. Nowadays, hand gestures are commonly tracked via built-in cameras from a VR headset. Like controllers, hand tracking can be used in VR applications to control virtual objects. This research developed VR intubation training as a case study and applied controllers and hand tracking for four interactions—namely collision, grabbing, pressing, and release. The quasi-experimental design assigned 30 medical students in clinical training to investigate the differences between using VR controller and hand tracking in medical interactions. The subjects were divided into two groups, one with VR controllers and the other with VR hand tracking, to study the interaction time and user satisfaction in seven procedures. System Usability Scale (SUS) and User Satisfaction Evaluation Questionnaire (USEQ) were used to measure user usability and satisfaction, respectively. The results showed that the interaction time of each procedure was not different. Similarly, according to SUS and USEQ scores, satisfaction and usability were also not different. Therefore, in VR intubation training, using hand tracking has no difference in results to using controllers. As medical training with free-hand gestures is more natural for real-world situations, hand tracking will play an important role as user input for VR medical training. This allows trainees to recognize and correct their postures intuitively, which is more beneficial for self-learning and practicing.
... 3D hand pose estimation is an important and active research topic due to its versatile applications in sign language recognition [49], HCI (human-computer interaction) [24][25][26], healthcare and entertainment [2,17,31]. Some HCI applications such as typing 1 highly rely on accurately estimated hand poses. ...
Estimating the 3D hand pose from a monocular RGB image is important but challenging. A solution is training on large-scale RGB hand images with accurate 3D hand keypoint annotations. However, it is too expensive in practice. Instead, we have developed a learning-based approach to synthesize realistic, diverse, and 3D pose-preserving hand images under the guidance of 3D pose information. We propose a 3D-aware multi-modal guided hand generative network (MM-Hand), together with a novel geometry-based curriculum learning strategy. Our extensive experimental results demonstrate that the 3D-annotated images generated by MM-Hand qualitatively and quantitatively outperform existing options. Moreover, the augmented data can consistently improve the quantitative performance of the state-of-the-art 3D hand pose estimators on two benchmark datasets. The code will be available at https://github.com/ScottHoang/mm-hand.
... As virtual reality (VR) technology has rapidly been advanced over the past decade, VR is widely used in many fields including education, professional training and gaming (Miller et al., 2013;Nakamura et al., 2016). VR interface provides great potential benefits as a new computer-human interface including its natural and intuitive interactions, immersive three-dimensional surroundings, and cost-effective simulation (Chen and Or, 2017;Lin et al., 2017). However, this new interface poses potential physical ergonomic risk factors which are associated with musculoskeletal discomfort and injuries especially in the neck and shoulder regions. ...
The objective of this study was to evaluate the effect of different target locations on musculoskeletal loading and task performance during virtual reality (VR) interactions. A repeated-measures laboratory study with 20 participants (24.2 � 1.5 years; 10 males) was conducted to compare biomechanical exposures (joint angle, moment, and muscle activity in the neck and shoulder), subjective discomfort, and task performance (speed and accuracy) during two VR tasks (omni-directional pointing and painting tasks) among different vertical target locations (ranged from 15 � above to 30 � below eye height). The results showed that neck flexion/extension angle and moment, shoulder flexion angle and moment, shoulder abduction angle, muscle activities of neck and shoulder muscles, and subjective discomfort in the neck and shoulder significantly varied by target locations (p's < 0.001). The target locations at 15 � above and 30 � below eye height demonstrated greater shoulder flexion (up to 52 �), neck flexion moment (up to 2.7Nm), anterior deltoid muscle activity, and subjective discomfort in the neck and shoulder as compared to the other locations. This result indicates that excessive vertical target locations should be avoided to reduce musculoskeletal discomfort and injury risks during VR interactions. Based on relatively lower biomechanical exposures and trade-off between neck and shoulder postures, vertical target location between eye height and 15 � below eye height could be recommended for VR use.
... More recently, studies have shifted the focus to sensory technologies that can further improve virtual reality immersion and interactivity. For example, to support high levels of interaction, hand gestures were developed to manipulate objects in virtual reality [37], and head tracking was integrated into virtual reality with the Oculus Rift headset to provide deep immersion game control [42]. Additionally, multimodal interactions have been developed that include visual, haptic, and brain-computer interfaces in virtual reality environments [35]. ...
Real-time adaptation is one of the most important problems that currently require a solution in the field of personalized human-computer interaction. For conventional desktop system interactions, user behaviors are acquired to develop models that support context-aware interactions. In virtual reality interactions, however, users operate tools in the physical world but view virtual objects in the virtual world. This dichotomy constrains the use of conventional behavioral models and presents difficulties to personalizing interactions in virtual environments. To address this problem, we propose the cross-object user interfaces (COUIs) for personalized virtual reality touring. COUIs consist of two components: a Deep Learning algorithm-based model using convolutional neural networks (CNNs) to predict the user’s visual attention from the past eye movement patterns to determine which virtual objects are likely to be viewed next, and delivery mechanisms that determine what should when and where be displayed on the user interface. In this chapter, we elaborate on the training and testing of the prediction model and evaluate the delivery mechanisms of COUIs through a cognitive walk-through approach. Furthermore, the implications for using COUIs to personalize interactions in virtual reality (and other environments such as augmented reality and mixed reality) are discussed.
