No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Improved Situational Awareness and Performance with Dynamic Task-Based Overlays for Teleoperation
ResearchGate has not been able to resolve any citations for this publication.
Remote machine systems have drawn a lot of attention owing to accelerations of virtual reality (VR), augmented reality (AR), and the fifth generation (5G) networks. Despite recent trends of developing autonomous systems, the realization of sophisticated dexterous hand that can fully replace human hands is considered to be decades away. It is also extremely difficult to reproduce the sensilla of complex human hands. On the other hand, it is known that humans can perceive haptic information from visual information even without any physical feedback as cross modal sensation between visual and haptics sensations or pseudo haptics. In this paper, we propose a visual haptic technology, where haptic information is visualized in more perceptual images overlaid at the contact points of a remote machine hand. The usability of the proposed visual haptics was evaluated by subject's brain waves aiming to find out a new approach for quantifying "sense of oneness." In our proof-of-concept experiments using VR, subjects are asked to operate a virtual arm and hand presented in the VR space, and the performance of the operation with and without visual haptics information as measured with brain wave sensing. Consequently, three results were verified. Firstly, the information flow in the brain were significantly reduced with the proposed visual haptics for the whole α, β, and θ-waves by 45% across nine subjects. This result suggests that superimposing visual effects may be able to reduce the cognitive burden on the operator during the manipulation for the remote machine system. Secondly, high correlation (Pearson correlation factor of 0.795 at a p-value of 0.011) was verified between the subjective usability points and the brainwave measurement results. Finally, the number of the task successes across sessions were improved in the presence of overlaid visual stimulus. It implies that the visual haptics image could also facilitate operators' pre-training to get skillful at manipulating the remote machine interface more quickly.
This paper presents the design and integration of
3-Survivor: a rough terrain negotiable teleoperated mobile
rescue and service robot. 3-Survivor is an improved version of
two previously studied surveillance robots named Sigma-3 and
Alpha-N. In 3-Survivor, a modified double-tracked with
caterpillar mechanism is incorporated in the body design. A
passive adjustment established in the body balance enables the
front and rear body to operate in excellent synchronization.
Instead of using an actuator, a reconfigurable dynamic method
is constructed with a 6 DOF arm. This dynamic method is
configured with the planer, spatial mechanism, rotation matrix,
motion control of rotation using inverse kinematics and
controlling power consumption of the manipulator using
angular momentum. The robot is remotely controlled using a
handheld Radio Frequency (RF) transmitter. 3-Survivor is
equipped with a Raspberry Pi 12 MP camera which is used for
live streaming of robot operations. Object detection algorithms
are run on the live video stream. The object detection method is
built using a Faster R-CNN with VGGNet-16 architecture of
CNN. The entire operations of the robot are monitored through
a web control window. Therefore, the control portal provides a
brief scenario of the environment to run, control and steer the
robot for more precise operation. A very impressive 88.25%
accuracy is acquired from this module in a rescue operation.
Along with the ODM, the sensor system of the robot provides
information on the hazardous terrain. The feasibility of the 3-
Survivor is tested and presented by different experiments
throughout the paper.
Physical human-robot interaction(pHRI) is the most popular way to ensure the safety in robot surgery. Current research of pHRI in operation room focuses on the inside of the surgical area, which adapts virtual fixtures method for ensuring the safety of drag. But the safety problems caused by human error in the stage of dragging the robot from the outside surgical area to the inside surgical area are ignored. Therefore, a method of applying virtual fixtures to the outside of the surgical area is proposed to solve the safety issues during dragging stage outside surgical area. This method takes the kinematics model of human arm as the reference motion trajectory to construct the guided virtual fixtures, which is to restrict the robot movement in a defined area during the pHRI drag. This drag is based on admittance control method to improve safety and ensure flexibility. Experiment results show that the constructed guided virtual fixtures with the trajectory of the human arm model as the central axis, with radius 30mm, and restrict area 5mm can effectively limit the robot motion to a certain range. Simultaneously, the output speed of the robot in tangent direction of the central axis can well follow the change of the force applied by the doctor, and the output speed in the normal direction of the central axis can converge to zero stably at the pipeline boundary. Consequently, the purpose of improving the safety and flexibility of the surgical robot before surgical operation is realized.
This work presents a user-study evaluation of various visual and haptic feedback modes on a real telemanipulation platform. Of particular interest is the potential for haptic guidance virtual fixtures and 3D-mapping techniques to enhance efficiency and awareness in a simple teleoperated valve turn task. An RGB-Depth camera is used to gather real-time color and geometric data of the remote scene, and the operator is presented with either a monocular color video stream, a 3D-mapping voxel representation of the remote scene, or the ability to place a haptic guidance virtual fixture to help complete the telemanipulation task. The efficacy of the feedback modes is then explored experimentally through a user study, and the different modes are compared on the basis of objective and subjective metrics. Despite the simplistic task and numerous evaluation metrics, results show that the haptic virtual fixture resulted in significantly better collision avoidance compared to 3D visualization alone. Anticipated performance enhancements were also observed moving from 2D to 3D visualization. Remaining comparisons lead to exploratory inferences that inform future direction for focused and statistically significant studies.
The current study investigates individual differences that predict situation awareness (SA) in professional pilots. The aim of the study is twofold: to examine the roles of divided attention, inhibition, working memory, and expertise in predicting SA, and to demonstrate the relative contributions of these individual differences to online (Situation Awareness Present Method, SPAM) and offline (Situation Awareness Global Assessment Technique, SAGAT) SA measures. Thirty-six professional pilots completed a challenging flight scenario in a full-flight simulator. Divided attention, inhibition, working memory span, and expertise were measured using choice reaction time with dichotic listening, Stroop, and Automated Operation Span tasks, and flight hours in a full-flight simulator, respectively. Results indicated that offline and online SA measure were not correlated, supporting their concurrent use to obtain a comprehensive measure of SA. Offline SA scores were best predicted by working memory and level of expertise, while online SA scores were predicted by expertise, divided attention and inhibition. Results are discussed focusing on both theoretical contributions for defining and measuring SA and applications. Findings have implications for operators of critical domains and their interactions with automated systems, in which SA is crucial for performance and safety.
The article presents some aspects of a complex control system of a teleoperated military mobile robot Tactical Robotic System (TAROS) related to virtual reality and assistance to a human operator in general. Described is especially the unique and innovative system of virtual operator station which uses the HMD Oculus Rift to put the operator into a virtual space containing visual feedback from the robot and camera images, including stereovision. The virtual operator station serves as a cost-effective and portable replacement of what otherwise would be a large room with expensive equipment. Mentioned is also another system that helps the operator with remote manipulation tasks – the anti-collision system preventing damage done to the mechanical parts of the robot by incautious movements of the manipulator arm.
Objective:
The current status of human-robot interaction (HRI) is reviewed, and key current research challenges for the human factors community are described.
Background:
Robots have evolved from continuous human-controlled master-slave servomechanisms for handling nuclear waste to a broad range of robots incorporating artificial intelligence for many applications and under human supervisory control.
Methods:
This mini-review describes HRI developments in four application areas and what are the challenges for human factors research.
Results:
In addition to a plethora of research papers, evidence of success is manifest in live demonstrations of robot capability under various forms of human control.
Conclusions:
HRI is a rapidly evolving field. Specialized robots under human teleoperation have proven successful in hazardous environments and medical application, as have specialized telerobots under human supervisory control for space and repetitive industrial tasks. Research in areas of self-driving cars, intimate collaboration with humans in manipulation tasks, human control of humanoid robots for hazardous environments, and social interaction with robots is at initial stages. The efficacy of humanoid general-purpose robots has yet to be proven.
Applications:
HRI is now applied in almost all robot tasks, including manufacturing, space, aviation, undersea, surgery, rehabilitation, agriculture, education, package fetch and delivery, policing, and military operations.
Situation awareness (SA) has become an important criterion for systems evaluation efforts. Several measures of SA have been developed, the most widely used among them being the Situation Awareness Global Assessment Technique (SAGAT) and the Situational Awareness Rating Technique (SART). SAGAT provides an objective measure of SA based on queries during freezes in a simulation. SART provides a subjective rating of SA by operators. This paper presents a direct comparison of the two measures which were used within a display evaluation study. It was found that both SART and SAGAT contributed sensitivity and diagnosticity regarding the effects of the display concept. The SART measure was highly correlated with subjective measures of confidence level, a simple subjective SA measure and a subjective performance measure. The SAGAT and SART measures were not correlated with each other. The implications of these findings for the interpretation of subjective SA measures are discussed as well as advantages and disadvantages of both measurement approaches.
This report introduces the notion of virtual fixtures for use in telepresence systems. Tools and fixtures in the real world (e.g., a ruler guiding a pencil) enhance human performance by guiding manual operations, providing localizing references, reducing mental workload, and increasing precision. Virtual fixtures are computer-generated percepts overlaid on top of the reflection of a remote workspace which can provide similar benefits. Because such perceptual overlays are virtual constructions they can be diverse in modality, abstract in form, and custom tailored to individual situations. This study investigates the potential of virtual fixtures by implementing simple combinations of haptic surfaces and auditory sensations as virtual perceptual aids in a standardized telemanipulation task. Subjects viewed the remote environment through a vision system while wearing an upper-body exoskeleton. Eight subjects controlled a slave robot arm to perform standard Fitts' law peg-insertion tasks with and without the aid of a variety of virtual fixtures. Fixtures composed of haptic and auditory perceptual overlays increased operator performance up to 70%. Simple fixtures devised from basic elements can be powerful perceptual aids; a workstation environment might be developed to allow a teleoperator to design and implement virtual fixtures interactively. (Keywords: Telepresence, Telerobotics, Virtual Reality, Augmented Reality, Mixed Reality, AR, VR, MR, HCI )
A new theory, called thetilt constancy theory, claims that the Ponzo illusion is caused by the misperception of orientation induced by local visual cues. The theory relates
the Ponzo illusion—along with the Zöllner, Poggendorff, Wündt-Hering, and cafe wall illusions—to the mechanisms that enable
us to perceive stable orientations despite changes in retinal orientation or body orientation. In Experiment 1, the magnitude
of the misperception of orientation was compared with the magnitude of the Ponzo illusion. In Experiment 2, predictions of
the tilt constancy theory were compared with accounts based on (1) low spatial frequencies in the image, (2) memory comparisons
(pool-and-store model), and (3) relative size judgments. In Experiment 3, predictions of the tilt constancy theory were tested
against predictions of the assimilation theory of Pressey and his colleagues. In the final experiment, the orientation account
was compared with theories based on linear perspective and inappropriate size constancy. The results support the tilt constancy
theory.
In this article, we describe and demonstrate control algorithms for general motion constraints. These constraints are designed to enhance the accuracy and speed of a user manipulating in an environment with the assistance of a cooperative or telerobotic system. Our method uses a basis of preferred directions, created off-line or in real-time using sensor data, to generate virtual fixtures that may constrain the user to a curve, surface, orientation, etc. in space. Open loop virtual fixtures seek only to maintain user motion along preferred directions, whereas closed loop fixtures additionally guide the user toward a point, line, or surface. This article demonstrates and compares the effects of open and closed loop fixtures in both autonomous and human-machine cases.
This paper presents a theoretical model of situation awareness based on its role in dynamic human decision making in a variety of domains. Situation awareness is presented as a predominant concern in system operation, based on a descriptive view of decision making. The relationship between situation awareness and numerous individual and environmental factors is explored. Among these factors, attention and working memory are presented as critical factors limiting operators from acquiring and interpreting information from the environment to form situation awareness, and mental models and goal-directed behavior are hypothesized as important mechanisms for overcoming these limits. The impact of design features, workload, stress, system complexity, and automation on operator situation awareness is addressed, and a taxonomy of errors in situation awareness is introduced, based on the model presented. The model is used to generate design implications for enhancing operator situation awareness and future directions for situation awareness research.
Thesis (Ph. D.)--University of Southern California, 1990. Includes bibliographical references (leaves 119-128).
Teleoperated minimally invasive surgical robots can significantly enhance a surgeon's accuracy, dexterity and visualization. However, current commercially available systems do not include significant haptic (force and tactile) feedback to the operator. This paper describes experiments to characterize this problem, as well as several methods to provide haptic feedback in order to improve surgeon's performance. There exist a variety of sensing and control methods that enable haptic feedback, although a number of practical considerations, e.g. cost, complexity and biocompatibility, present significant challenges. The ability of teleoperated robot-assisted surgical systems to measure and display haptic information leads to a number of additional exciting clinical and scientific opportunities, such as active operator assistance through "virtual fixtures" and the automatic acquisition of tissue properties.
Fish-eye lenses are convenient in such computer vision applications where a very wide angle of view is needed. However, their use for measurement purposes is limited by the lack of an accurate, generic, and easy-to-use calibration procedure. We hence propose a generic camera model for cameras equipped with fish-eye lenses and a method for calibration of such cameras. The calibration is possible by using only one view of a planar calibration object but more views should be used for better results. The proposed calibration method was evaluated with real images and the obtained results are promising. The calibration software becomes commonly available at the author's Web page.
This paper presents the results of two cognitive load studies comparing three augmented reality display technologies: spatial augmented reality, the optical see-through Microsoft HoloLens, and the video see-through Samsung Gear VR. In particular, the two experiments focused on isolating the cognitive load cost of receiving instructions for a button-pressing procedural task. The studies employed a self-assessment cognitive load methodology, as well as an additional dual-task cognitive load methodology. The results showed that spatial augmented reality led to increased performance and reduced cognitive load. Additionally, it was discovered that a limited field of view can introduce increased cognitive load requirements. The findings suggest that some of the inherent restrictions of head-mounted displays materialize as increased user cognitive load.
This paper proposes an alternative approach to common teleoperation methods found in search and rescue (SAR) robots. Using a head mounted display (HMD) the operator is capable of perceiving rectified images of the robot world in 3-D, as transmitted by a pair of stereo cameras onboard the robot. The HMD is also equipped with an integrated head-tracker, which permits controlling the robot motion in such a way that the cameras follow the operator's head movements, thus providing an immersive sensation to him. We claim that this approach is a more intuitive and less error prone teleoperation of the robot. The proposed system was evaluated by a group of subjects, and the results suggest that it may yield significant benefits to the effectiveness of the SAR mission. In particular, the user's depth perception and situational awareness improved significantly when using the HMD, and their performance during a simulated SAR operation was also enhanced, both in terms of operation time and on successful identification of objects of interest.
We simulated a military mounted environment and examined the performance of the combined position of gunner and robotics operator and how aided target recognition (AiTR) capabilities (delivered either through tactile or tactile + visual cueing) for the gunnery task might benefit the concurrent robotics and communication tasks. Results showed that participants' teleoperation task improved significantly when the AiTR was available to assist them with their gunnery task. However, the same improvement was not found for semi-autonomous robotics task performance. Additionally, when teleoperating, those participants with higher spatial ability outperformed those with lower spatial ability. However, performance gap between those with higher and lower spatial ability appeared to be narrower when the AiTR was available to assist the gunnery task. Participants' communication task performance also improved significantly when the gunnery task was aided by AiTR. Finally, participant's perceived workload was significantly higher when they teleoperated a robotic asset and when their gunnery task was unassisted.
Abstract-This article accompanies a video that presents a bimanual haptic device composed of two DLR/KUKA Light weight Robot (LWR) arms. The LWRs have similar dimensions to human arms, and can be operated in torque and position control mode at an update rate of 1kHz. The two robots are mounted behind the user, such that the intersecting workspace of the robots and the human arms becomes maximal. In order to enhance user interaction, various hand interfaces and additional tactile feedback devices can be used together with the robots. The presented system is equipped with a thorough safety architecture that assures safe operation for human and robot. Additionally, sophisticated control strategies improve performance and guarantee stability. The introduced haptic system is well suited for versatile applications in remote and virtual environments, especially for large unsealed movements.
For 11 studies, we find that the detection of usability problems as a function of number of users tested or heuristic evaluators employed is well modeled as a Poisson process. The model can be used to plan the amount of evaluation required to achieve desired levels of thoroughness or benefits. Results of early tests can provide estimates of the number of problems left to be found and the number of additional evaluations needed to find a given fraction. With quantitative evaluation costs and detection values, the model can estimate the numbers of evaluations at which optimal cost/benefit ratios are obtained and at which marginal utility vanishes. For a â mediumâ example, we estimate that 16 evaluations would be worth their cost, with maximum benefit/cost ratio at four.
Graphic overlays in high-precision teleoperation: current and future work at JPL
- D B Diner
- Diner B.
Improved Situational Awareness in ROS Using Panospheric Vision and Virtual Reality
- V Vunder
- Vunder V.
Properties of cutaneous mechanoreceptors in the human hand related to touch sensation
- A B Vallbo
- A.B.