Content uploaded by Orestis Georgiou
Author content
All content in this area was uploaded by Orestis Georgiou on May 14, 2020
Content may be subject to copyright.
Touchless Haptic Feedback for Supernatural VR Exp e rien ces
!
Jonatan Martinez, Daniel Griffiths, Valerio Biscione*, Orestis Georgiou, and Tom Carter
Ultrahaptics Ltd., Bristol, United Kingdom
!
ABSTRACT
Haptics is an important part of the VR space as seen by the plethora
of haptic controllers available today. By using a novel ultrasonic
haptic device, we developed and integrated mid-air haptic
sensations without the need to wear or hold any equipment in a VR
game experience. The compelling experience combines visual,
audio and haptic stimulation in a supernatural narrative in which
the user takes on the role of a wizard apprentice. By using different
haptified patterns we could generate a wide range of sensations
which mimic supernatural interactions (wizard spells). We detail
our methodology and briefly discuss our findings and future work.
Keywords: haptics; HCI; VR; ultrasound; supernatural.
Index Terms: H.5.1 [Human computer interaction (HCI)]:
Interaction devices—Haptic devices; H.5.2 [Human computer
interaction (HCI)]: Interaction paradigms—Virtual reality
1 INTRODUCTION
To create fully immersive experiences in Virtual Reality (VR),
all 5 senses (sight, sound, touch, smell, and taste) must perceive the
digital environment. These senses need not always accurately
reproduce reality since our imagination can often fill in the gaps.
Moreover, many sensory modalities can be augmented,
supplemented, or even substituted. With great advancements in
current VR technologies, it is mostly the visual and auditory
modalities that have been adequately addressed to date. Haptics,
one of the most intimate of senses, has to date mostly been realized
in hand-held controllers that vibrate, or actuate within a limited set
of degrees of freedom. Possible peripheral alternatives include
especially designed controllers, haptic wearable gloves, electrical
muscle stimulation pads, and even full body exoskeleton suits.
Haptic feedback can significantly increase the performance of
VR applications [3]. In this extended abstract, we present and
demonstrate a touchless (i.e., non-wearable) haptic platform and its
effective use in rendering supernatural haptic feedback in VR. The
haptic device we have developed combines off-the-shelf hand
tracking solutions with the advanced manipulation of focused
ultrasound to remotely stimulate receptor structures in various parts
of the hand. This novel haptic technology has come a long way
since early haptic displays [1] and can now be actuated within an
expanded functional space as to adequately produce extraordinary
haptic effects that can be leveraged in VR applications.
The goal of our present research was not the accurate haptic
reproduction of complex 3D objects (e.g., a coffee cup) but rather
the haptification of abstract forces like the shooting of lightning
bolts from your hands (e.g., like a wizard), thereby enhancing the
overall VR experience. Since such effects are attributed to forces
beyond scientific understanding or the laws of nature (and hence
are supernatural) they present great challenges to developers with
regards to rendering them using existing haptic devices. Moreover,
since these haptic effects are not something we can relate to direct
past experiences we have to supplement our active ultrasonically
induced haptics with those of audio-visual pseudo-haptics [2].
The result of our research seems to suggest a significant
amplification and realism of the desired haptic sensations. Namely,
that we have adequately rendered the haptic perception of what one
might expect magical spells of lightning, fire, and wind to feel like.
This means that when tasked with computationally analyzing a
sample of haptic data points, ordered in both space and time relative
to the hand and fused together with other audio-visual haptic cues,
the brain was sufficiently convinced to believe and accept that this
is indeed a supernatural experience. Crucial to this illusionary
effect is that our haptic engine and actuator device have a large
enough functional space, and can rapidly explore and display it.
Here, we will limit the discussion to the demonstrator’s physical
and haptic sensation design, and defer further in-depth technical
and user studies for future work. Our main contribution is the
demonstration of the capabilities of state-of-the-art touchless mid-
air haptic feedback hardware and software, and a discussion on
best-practices for its efficient application in the rendering of
supernatural forces in VR/AR games and experiences.
2 DEMONSTRATION SETUP
The demonstration is set on a tabletop and consists of the HTC Vive
VR headset with headphones, a single Lighthouse and a gaming
laptop, an Ultrahaptics Evaluation Kit (UHEV1), a LEAP Motion
controller, and an adjustable pedestal stand tilted at an angle of 45
degrees (see Figure 1). The LEAP motion and UHEV1 are pre-
calibrated (with a fixed, known position relative to one another). To
align the graphics to the UHEV1 and the LEAP, we place the Vive
controller on a known position on the UHEV1. When the trigger on
the Vive controller is pressed, the camera is moved so that the
UHEV1 is aligned with a known point in the virtual scene (the stand
where the magic orbs appear).
We have designed the setup, the user interface and gameplay of
the demonstrator as to focus and show off the strengths of the
components being used in synergy. Namely, we made sure that the
position of the LEAP Motion controller camera has a direct and
clear field of view (FoV) of the user’s palms. Most of the game
interactions are also designed to be actioned with open palms. This
improves the hand tracking accuracy of the LEAP camera and also
improves the haptic sensations achieved by focused ultrasound
Valerio.Biscione@ultrahaptics.com
Figure 1: Photo of the demonstrator setup that includes the UHEV1
device, the LEAP mot ion controller, and VR capture screen
showing the young wizard shooting lightning out of his bare hands.
.!
vibrations since the Pacinian corpuscles are most dense there.
Finally, we decided on a sitting down game, rather than a standing
up one, since this can relax the user while performing her magical
spells. As a result, both the LEAP camera and the Ultrahaptics
device have been angled at 45 degrees (see Figure 1).
3 GAME AND INTERACTION TECHNIQUES
The game is designed to mimic the physical setup described above
but graphically dressing everything with a dungeon-like effect and
the user’s avatar hands are an all-white skeleton [4].
The game narrative was organized around the concept that the
user is an apprentice wizard learning the lightning spell. The book
of spells, resting on a pedestal in front of her, contains visual
instructions, and a voiceover was added to make the experience
more compelling. Each spell can be absorbed through a “magic
spherical orb”, which the user is prompted to touch. The use of the
lightning spells (triggered by a close-open hand gesture)
“accidentally” results in the release of some flying insects (in this
case scarabs) which the user is prompted to squash by first learning
the fire spell (absorbing the fire orb) and then hitting, grasping, or
slapping the scarabs in mid-air with either of her fiery hands. This
in turn “accidentally” results in part of the desk catching fire. The
user is then instructed to extinguish the fire with the wind spell
while hovering her open hands over of the fire. Once the fire is out
the user is informed that her session is over and the book of spells
magically closes. Screenshots are shown in Figure 2.
4 HAPTICS
A phased array composed of 16x16=256 ultrasonic transducers at
40 kHz was used to focus ultrasound at one or more specified
locations in three dimensions, 90 degrees wide and up to 70 cm far
from the device. The (x,y,z) coordinates for creating the focus point
(e.g., at the center of the user’s palm or fingertip) are obtained from
the LEAP Motion API. For a tactile sensation to be perceived, the
acoustic radiation force at a focus point is amplitude modulated at
200 Hz as to create small localized skin displacements that are
perceived by the user as strong haptic effect. The haptic refresh rate
of this device is an impressive 16 kHz which is fast enough to allow
very rapid updates of the focus point coordinates as to smoothly
display complex curves and patterns on the users’ hands (see Figure
3). These complex curves are what we have used to emulate the
desired supernatural effects. The Ultrahaptics device comes with an
API written in C++ and C# with Unity bindings which has (x,y,z)
position coordinates as one of its main inputs needed to generate up
to 8 amplitude modulated focus points, thereby making the real-
time haptification of hand gestures straight forward, as long as they
are i) detected by the LEAP Motion, and ii) within the focusing
region of the Ultrahaptics device.
All user interactions described in the previous section have been
haptified using focused ultrasound. By moving the focus of the
ultrasound across the palm with a certain pattern, we could generate
different haptic sensations. These haptic sensations were designed
using just a single focus point printed on each hand (two focus
points in total), moving along a predefined path or skipping through
several points on the surface of the palm, thus generating a “haptic
pattern” of movement. Therefore, we had to constantly update in
software the coordinate system at which the focus point was printed
as to form planes parallel to the user’s palms. Standard rotation and
translation matrices were used for this operation in order to create
a left and right palm coordinate system relative to the LEAP
motion’s frame of reference. The intensity of the haptic sensations
was kept fixed across all haptic effects.
We designed several haptic patterns in order to match the
following interactions: Hand hovers over the orb and touching the
orb (these varied slightly for the lightning, the fire, and the wind
orb); Lightning Spell, Fire Spell (when touching the scarabs to kill
them); Wind spell. A schematic of these patterns for the four of the
sensations is shown and described in Figure 3. The haptic
perception of these effects is enhanced by using different graphical
and auditory cues [3]. For example, for the fire spell (Figure 3c),
the movement of the spiral along the figure 8 path is synchronized
with a graphical representation of a small flame following the same
figure 8 path on the user’s palm, together with the typical crackling
and sizzling sound of flames, with a stereo 3D sound coming from
the user hands. As a further example, when absorbing the lightning
from the orb, sparks are seen to be generated at semi-random
positions on the avatar’s palm. The haptic sensation is enhanced by
also using a synchronized sound effect of sparkling electricity.
5 CONCLUSION
In this extended abstract, we have shown and described the
world’s first demonstration of mid-air haptics within a supernatural
narrative in VR. We have shown how a varied set of sensations can
be evoked by changing the movement pattern of an ultrasonic focus
point thus creating a compelling experience. To that end, we have
learned that a compelling haptic experience can be obtained by
combining different levels of moving paths, e.g., a haptic line
moving anti-clockwise, whose center is also following a path
around the palm. Such complex ultrasonic manipulations have only
recently been unlocked. As a future direction for this work, we
therefore propose that an even greater range of sensations can be
obtained by combining complex paths from multiple focus points.
REFERENCES
[1] Takayuki Iwamoto, et al., “Non-contact method for producing
tactile sensation using airborne ultrasound.” Haptics:
Perception, Devices and Scenarios (2008): 504-513.
[2] Anatole Lecuyer, “Simulating Haptic Feedback Using Vision:
a Survey of Research and Applications of Pseudo-Haptic
Feedback, Presence: Teleoperators and Virtual
Environments.” 18.1 (2009): 39-53.
[3] Robert Stone, "Haptic feedback: A brief history from
telepresence to virtual reality." Haptic Human-Computer
Interaction (2001): 1-16.
[4] https://www.youtube.com/watch?v=GqgFRPs0wxA !
Figure 3: Four examples of the haptic pattern used in the demo. All
the sensations were projected on the surface of the palm. a)
touching the magic orb: skipping through multiple random haptic
points; b) lightning spell: moving haptic point from the wrist to the
index finger tip; c) fire spell: spiral following the path of a figure of 8;
d) wind spell: a rotating line follows a rotating path.
Figure 2: Screenshots during the orb activation and lightning sp ell.
