ArticlePublisher preview available

Development of a visual to audio and tactile substitution system for mobility and orientation of visually impaired people: a review

August 2023
Multimedia Tools and Applications 83(7):1-41

August 2023
83(7):1-41

Authors:

Maharashtra Academy of Engineering and Education Research

According to the World Health Organization (WHO), visual impairment is one of the most common problems affecting approximately one sixth of the world's population. It was found that 39 million of these sufferers have lost their vision completely and require supervision from other people to do their daily chores. Until today, several methodologies have been built to provide effective solutions to visually challenged people, mainly in terms of navigation. The assistive technologies developed for the visually challenged are generally very popular due to the benefits provided, but these technologies are limited in many aspects. One of the main problems with the technologies is that they are non-adaptable and cannot adjust to the changing needs of the participants. The demands of blind people are constantly growing, and there is no powerful technology that can meet all of these demands at the same time. The major constraints with assistive technologies are cost-effectiveness and user-friendliness. This paper presents a complete review of the assistive technologies introduced in the literature to deal with the navigation problems of blind people. Each and every literature work focuses on the requirements of the blind, and several features are considered accordingly. Thus, the presented review provides a detailed description of the evolution of such assistive technologies and the improvements brought in to satisfy the users. Apart from this, the complexities and limitations associated with the technologies are also unwounded to provide a clear perspective on the current scenario for future developers and researchers. Several constraints associated with different assistive technology are interpreted, and this review ends with the insights drawn from the study along with future scopes.

Methodology behind the review

…

Principle of basic robotic assistive technology

…

An example of an eye substitution device [23]

…

Reflection of ultrasonic pulse sequence between transmitter and receiver

…

+18

Dual ultra-sonic devices covering three ranges

…

Figures - available from: Multimedia Tools and Applications

This content is subject to copyright. Terms and conditions apply.

A preview of this full-text is provided by Springer Nature.

Learn more

Content available from Multimedia Tools and Applications

This content is subject to copyright. Terms and conditions apply.

Vol.:(0123456789)

Multimedia Tools and Applications (2024) 83:20387–20427

https://doi.org/10.1007/s11042-023-16355-0

1 3

Development ofavisual toaudio andtactile substitution

system formobility andorientation ofvisually impaired

people: areview

KomalMahadeoMasal1· ShripadBhatlawande2· SachinDattatrayaShingade3

Received: 13 September 2022 / Revised: 29 May 2023 / Accepted: 17 July 2023 /

Published online: 1 August 2023

Abstract

According to the World Health Organization (WHO), visual impairment is one of the

most common problems aﬀecting approximately one sixth of the world’s population. It

was found that 39 million of these suﬀerers have lost their vision completely and require

supervision from other people to do their daily chores. Until today, several methodolo-

gies have been built to provide eﬀective solutions to visually challenged people, mainly

in terms of navigation. The assistive technologies developed for the visually challenged

are generally very popular due to the beneﬁts provided, but these technologies are limited

in many aspects. One of the main problems with the technologies is that they are non-

adaptable and cannot adjust to the changing needs of the participants. The demands of

blind people are constantly growing, and there is no powerful technology that can meet

all of these demands at the same time. The major constraints with assistive technologies

are cost-eﬀectiveness and user-friendliness. This paper presents a complete review of the

assistive technologies introduced in the literature to deal with the navigation problems

of blind people. Each and every literature work focuses on the requirements of the blind,

and several features are considered accordingly. Thus, the presented review provides

a detailed description of the evolution of such assistive technologies and the improve-

ments brought in to satisfy the users. Apart from this, the complexities and limitations

associated with the technologies are also unwounded to provide a clear perspective on

the current scenario for future developers and researchers. Several constraints associated

with diﬀerent assistive technology are interpreted, and this review ends with the insights

drawn from the study along with future scopes.

Keywords Visually impaired people· Assistive systems· Barrier avoidance· Object

detection· Mobility and orientation

* Komal Mahadeo Masal

kmmasal@pict.edu

1 Department ofTechnology DOT, SPPU & PICT, Pune, Maharashtra411007, India

2 Vishwakarma Institute ofTechnology, Pune, Maharashtra411037, India

3 Department ofTechnology DOT, SPPU & MIT Academy ofEngineering, Pune,

Maharashtra411007, India

Content courtesy of Springer Nature, terms of use apply. Rights reserved.

The Aerial Guide Dog: A Low-Cognitive-Load Indoor Electronic Travel Aid for Visually Impaired Individuals

Article

Full-text available

Jan 2024
SENSORS-BASEL

Most navigation aids for visually impaired individuals require users to pay close attention and actively understand the instructions or feedback of guidance, which impose considerable cognitive loads in long-term usage. To tackle the issue, this study proposes a cognitive burden-free electronic travel aid for individuals with visual impairments. Utilizing human instinctive compliance in response to external force, we introduce the “Aerial Guide Dog”, a helium balloon aerostat drone designed for indoor guidance, which leverages gentle tugs in real time for directional guidance, ensuring a seamless and intuitive guiding experience. The introduced Aerial Guide Dog has been evaluated in terms of directional guidance and path following in the pilot study, focusing on assessing its accuracy in orientation and the overall performance in navigation. Preliminary results show that the Aerial Guide Dog, utilizing Ultra-Wideband (UWB) spatial positioning and Measurement Unit (IMU) angle sensors, consistently maintained minimal deviation from the targeting direction and designated path, while imposing negligible cognitive burdens on users while completing the guidance tasks.

A Systematic Review on Developing Computer Programming Skills for Visually Impaired Students

Article

Full-text available

Jan 2024

Technology and digital-based learning drive contemporary computer skill (CS) education advances. The absence of resources that are easily accessible and tailored to help the development of conceptual understanding in computers may present a number of issues for students who have visual impairments (VI) in the context of CS education. The objective of the research was to provide a concise overview of the existing evidence-based information about CS instruction for students with severe VI. A systematic review was performed to analyze the research papers published between January 2000 and September 2023. A total of 21 articles were extracted and classified into assistive technologies (ATs) and CSs. The study findings highlight the significance of the development of ATs and effective course design for supporting individuals with VI. It can be used by policy makers and educational institutions to present a useful environment for individuals with VI to improve their CSs. In future, the authors need to focus on ATs to enhance the computer programming abilities of students with VI. In addition, they will develop programming platforms and integrated development environments that are easily accessible and can be customized to meet a wide range of learning requirements.

Quantifying Digital Transformation Impact on Customer Service Automation in SMEs Using Knowledge Management Approach

Preprint

Full-text available

May 2024

Digital transformation has revolutionized business operations and customer service with the rise of automation. This research aims to quantify the influence of Digital Transformation (DT) on customer service automation and predict open innovation patterns in small and medium-sized enterprises (SMEs) using a hybrid approach. The study will use a combination of quantitative and qualitative analysis methods to assess the extent of DT in SMEs and its impact on customer service automation. Data will be obtained from a trial of SMEs through surveys and interviews to gather information on their adoption of digital technologies and the level of automation in their customer service processes. The convenience sampling method is used for the data collection. However, to successfully implement these technologies, SMEs need to adopt effective knowledge management practices that can enable them to acquire, store, transfer, and exploit the knowledge generated from their interactions with customers driven by a customer-centric approach. Furthermore, the enquiry will develop a predictive model to identify open innovation patterns in SMEs by combining machine learning algorithms with expert knowledge. This hybrid approach will leverage the power of automated data analysis while incorporating the domain expertise of professionals in the field. The utilization of Morphological-Linear Neural Network (MLNN) in conjunction with a Logistic Regression model for addressing complex pattern recognition and prediction tasks. This study predicts the high ratings of brands for the base of customer satisfaction. The proposed work involves a detailed analysis utilizing the Statistical Package for the Social Sciences (SPSS) software, specifically tailored for SMEs. Cronbach's Alpha is a quantity of internal consistency or reliability in established items in a questionnaire or survey. The sum of squares between people is 45189.032, with 944 degrees of freedom, resulting in a mean square of 47.870. The findings can help SMEs make informed decisions regarding their DT strategies and potentially drive more effective and efficient customer service operations.

A Comprehensive Review of Navigation Systems for Visually Impaired Individuals

Article

Full-text available

May 2024

Background: This review explores the evolutionary trajectory of navigation assistance tools tailored for the visually impaired, spanning from traditional aids like white canes to contemporary electronic devices. It underlines their pivotal role in fostering safe mobility for visually impaired individuals. Objectives: The primary aim is to categorize and assess the plethora of navigation assistance solutions available. Emphasis is placed on technological advancements, particularly in electronic systems employing sensors, AI, and feedback mechanisms. Furthermore, the review underscores the emerging influence of smartphone-based solutions and navigation satellite systems in augmenting independence and quality of life for the visually impaired. Methods: Navigation assistance solutions are segmented into four key categories: Visual Imagery Systems, Non-Visual Data Systems, Map-Based Solutions, and 3D Sound Systems. The integration of diverse sensors like Ultrasonic Sensors and LiDAR for obstacle detection and real-time feedback is scrutinized. Additionally, the fusion of smartphone technology with sensors to deliver location-based assistance is explored. The review also evaluates the functionality, efficacy, and cost-efficiency of navigation satellite systems. Results: Results indicate a significant evolution in navigation aids, with modern electronic systems proving highly effective in aiding obstacle detection and safe navigation. The convenience and portability of smartphone-based solutions are underscored, along with the potential of navigation satellite systems to enhance navigation assistance. Conclusions: In conclusion, the review advocates for continued innovation and technological integration in navigation tools to empower visually impaired individuals with increased independence and safe access to their surroundings. It accentuates the imperative of ongoing efforts to enhance the quality of life for those with visual impairments through futuristic technological solutions.

A Machine Learning Based Image to Object Detection and Audio Description for Blind People

Conference Paper

Full-text available

Dec 2023

Hybrid Deep Artificial Humming Bird Algorithm For Improved Real Time Blind Assistance with Advanced Jetson Nano GPU

Conference Paper

Nov 2023

A CNN-Based Wearable Assistive System for Visually Impaired People Walking Outdoors

Article

Full-text available

Oct 2021

In this study, we propose an assistive system for helping visually impaired people walk outdoors. This assistive system contains an embedded system—Jetson AGX Xavier (manufacture by Nvidia in Santa Clara, CA, USA) and a binocular depth camera—ZED 2 (manufacture by Stereolabs in San Francisco, CA, USA). Based on the CNN neural network FAST-SCNN and the depth map obtained by the ZED 2, the image of the environment in front of the visually impaired user is split into seven equal divisions. A walkability confidence value for each division is computed, and a voice prompt is played to guide the user toward the most appropriate direction such that the visually impaired user can navigate a safe path on the sidewalk, avoid any obstacles, or walk on the crosswalk safely. Furthermore, the obstacle in front of the user is identified by the network YOLOv5s proposed by Jocher, G. et al. Finally, we provided the proposed assistive system to a visually impaired person and experimented around an MRT station in Taiwan. The visually impaired person indicated that the proposed system indeed helped him feel safer when walking outdoors. The experiment also verified that the system could effectively guide the visually impaired person walking safely on the sidewalk and crosswalks.

Design and Development of a Wearable Assistive Device Integrating a Fuzzy Decision Support System for Blind and Visually Impaired People

Article

Full-text available

Sep 2021

Yassine Bouteraa

In this article, a new design of a wearable navigation support system for blind and visually impaired people (BVIP) is proposed. The proposed navigation system relies primarily on sensors, real-time processing boards, a fuzzy logic-based decision support system, and a user interface. It uses sensor data as inputs and provides the desired safety orientation to the BVIP. The user is informed about the decision based on a mixed voice–haptic interface. The navigation aid system contains two wearable obstacle detection systems managed by an embedded controller. The control system adopts the Robot Operating System (ROS) architecture supported by the Beagle Bone Black master board that meets the real-time constraints. The data acquisition and obstacle avoidance are carried out by several nodes managed by the ROS to finally deliver a mixed haptic–voice message for guidance of the BVIP. A fuzzy logic-based decision support system was implemented to help BVIP to choose a safe direction. The system has been applied to blindfolded persons and visually impaired persons. Both types of users found the system promising and pointed out its potential to become a good navigation aid in the future.

Analysis and Validation of Cross-Modal Generative Adversarial Network for Sensory Substitution

Article

Full-text available

Jun 2021
Int J Environ Res Publ Health

Visual-auditory sensory substitution has demonstrated great potential to help visually impaired and blind groups to recognize objects and to perform basic navigational tasks. However, the high latency between visual information acquisition and auditory transduction may contribute to the lack of the successful adoption of such aid technologies in the blind community; thus far, substitution methods have remained only laboratory-scale research or pilot demonstrations. This high latency for data conversion leads to challenges in perceiving fast-moving objects or rapid environmental changes. To reduce this latency, prior analysis of auditory sensitivity is necessary. However, existing auditory sensitivity analyses are subjective because they were conducted using human behavioral analysis. Therefore, in this study, we propose a cross-modal generative adversarial network-based evaluation method to find an optimal auditory sensitivity to reduce transmission latency in visual-auditory sensory substitution, which is related to the perception of visual information. We further conducted a human-based assessment to evaluate the effectiveness of the proposed model-based analysis in human behavioral experiments. We conducted experiments with three participant groups, including sighted users (SU), congenitally blind (CB) and late-blind (LB) individuals. Experimental results from the proposed model showed that the temporal length of the auditory signal for sensory substitution could be reduced by 50%. This result indicates the possibility of improving the performance of the conventional vOICe method by up to two times. We confirmed that our experimental results are consistent with human assessment through behavioral experiments. Analyzing auditory sensitivity with deep learning models has the potential to improve the efficiency of sensory substitution.

Solar-Powered Deep Learning-Based Recognition System of Daily Used Objects and Human Faces for Assistance of the Visually Impaired

Article

Full-text available

Nov 2020

This paper introduces a novel low-cost solar-powered wearable assistive technology (AT) device, whose aim is to provide continuous, real-time object recognition to ease the finding of the objects for visually impaired (VI) people in daily life. The system consists of three major components: a miniature low-cost camera, a system on module (SoM) computing unit, and an ultrasonic sensor. The first is worn on the user's eyeglasses and acquires real-time video of the nearby space. The second is worn as a belt and runs deep learning-based methods and spatial algorithms which process the video coming from the camera performing objects' detection and recognition. The third assists on positioning the objects found in the surrounding space. The developed device provides audible descriptive sentences as feedback to the user involving the objects recognized and their position referenced to the user gaze. After a proper power consumption analysis, a wearable solar harvesting system, integrated with the developed AT device, has been designed and tested to extend the energy autonomy in the different operating modes and scenarios. Experimental results obtained with the developed low-cost AT device have demonstrated an accurate and reliable real-time object identification with an 86% correct recognition rate and 215 ms average time interval (in case of high-speed SoM operating mode) for the image processing. The proposed system is capable of recognizing the 91 objects offered by the Microsoft Common Objects in Context (COCO) dataset plus several custom objects and human faces. In addition, a simple and scalable methodology for using image datasets and training of Convolutional Neural Networks (CNNs) is introduced to add objects to the system and increase its repertory. It is also demonstrated that comprehensive trainings involving 100 images per targeted object achieve 89% recognition rates, while fast trainings with only 12 images achieve acceptable recognition rates of 55%.

Computer Vision-based Assistance System for the Visually Impaired Using Mobile Edge Artificial Intelligence

Conference Paper

Jun 2021

Designing 3D Printed Audio-Tactile Graphics: Recommendations from Prior Research

Chapter

Jul 2021

3D printed audio-tactile graphics are interactive materials aimed at the accessibility of people with visual impairment. There is not yet a consensus on how to design 3D printed audio-tactile graphics and most publications on this topic follow an empirical approach, experimenting first and learning with the process. However, these research publications show meaningful knowledge that can serve as the base for a set of compiled design recommendations. The goal of this paper is to present recommendations that were analyzed and extracted from 32 publications found in a systematic literature review. In total, 57 recommendations compose the set.

Arts activity and well-being for visually impaired military veterans: a narrative discussion of current knowledge

Article

May 2021

Objectives: The objective of this study was to offer an overview of literature relating to the topic of arts as activity within the context of military and veteran health and to consider the implications of current knowledge on future research with visually impaired veterans. Study design: A search for literature addressing the topic of visual art activities with visually impaired veterans was undertaken. No research addressing this topic was identified. A review of literature on the related topics of mental health and well-being in military veterans, visual impairment and mental health and well-being, and art therapy for veteran populations was carried out to offer an overview of current knowledge. Results: While there is growing evidence of the benefits of arts engagement among both general and military populations, the role of the visual arts in the everyday lives of broader veteran samples, and the impact of these activities on holistic well-being, remains underexplored. The current article highlights the need for art as activity to be differentiated from art as therapy and argues that the former might offer a tool to positively impact the holistic well-being of visually impaired veterans. Conclusions: Future research relating to the use of visual art activities in the context of veteran health and well-being should endeavour to explore the potential impacts of engagement on holistic well-being. Research is needed to build on anecdotal evidence of the positive impact of arts engagement on visually impaired veterans by systematically exploring if, and how, holistic well-being in this population may be impacted by engagement with visual art activities.

A Novel Marker Detection System for People with Visual Impairment Using the Improved Tiny-YOLOv3 Model

Article

Jun 2021
COMPUT METH PROG BIO

Background and Objective Daily activities such as shopping and navigating indoors are challenging problems for people with visual impairment. Researchers tried to find different solutions to help people with visual impairment navigate indoors and outdoors. Methods We applied deep learning to help visually impaired people navigate indoors using markers. We propose a system to help them detect markers and navigate indoors using an improved Tiny-YOLOv3 model. A dataset was created by collecting marker images from recorded videos and augmenting them using image processing techniques such as rotation transformation, brightness, and blur processing. After training and validating this model, the performance was tested on a testing dataset and on real videos. Results The contributions of this paper are: (1) We developed a navigation system to help people with visual impairment navigate indoors using markers; (2) We implemented and tested a deep learning model to detect Aruco markers in different challenging situations using Tiny-YOLOv3; (3) We implemented and compared several modified versions of the original model to improve detection accuracy. The modified Tiny-YOLOv3 model achieved an accuracy of 99.31% in challenging conditions and the original model achieved an accuracy of 96.11 %. Conclusion The training and testing results show that the improved Tiny-YOLOv3 models are superior to the original model.

Evaluation of walkability and mobility requirements of visually impaired people in urban spaces

Article

Dec 2020

The objective of this work is to evaluate the walkability of urban environments from the perspective of visually impaired people, developing a qualitative analysis based on Commented Paths Method (CPM) to derive subjective evaluations from the practical experience of specific paths. Multiple-choice questions, in applying Likert scale were used to assess physical characteristics (e.g. slope surface, sidewalk width) related to infrastructural elements of the analyzed path and their influence in the difficulty of movement, as well as interferences (e.g. need to stop, help from other people) and feelings (e.g. anxiety, tiredness) experimented along it. The elaboration of single judgments allowed the calculation of disaggregated and global Walkability Indices and the elaboration of associated thematic maps, as an innovative integration of the method, to highlight the critical aspects related to each stretch in which the path was divided. Results deriving by the implementation of a case study in the city of Enna (Italy) have revealed a greater weight to infrastructural criticalities with respect to perceived interferences, demonstrating coherence with the current situation of the analyzed area and increasing the study's validity. The research moves towards more inclusive public policies, capturing the subjective point of view of this specific target of users with visual disabilities, which is fundamental in the construction of the knowledge and crucial to improved walkability for all.

Malaysian Banknote Reader For Visually Impaired Person

Conference Paper