Figure - available from: SN Computer Science
This content is subject to copyright. Terms and conditions apply.
Source publication
Purpose: With an indisputable complexity of communication for hearing and speaking impaired people, most sign language recognition systems utilize virtual reality or onscreen robots. This paper presents the design and development of a special and low-cost humanoid robot that can perform as a sign language interpreter. To the best of our knowledge,...
Similar publications
In recent years, the synaptic properties of transistors have been extensively studied. Compared with liquid or organic material-based transistors, inorganic solid electrolyte-gated transistors have the advantage of better chemical stability. This study uses a simple, low-cost solution technology to prepare In2O3 transistors gated by AlLiO solid ele...
Citations
... In accordance with how an SL is expressed, the corresponding SL dataset can consist of images or videos. In many researches, traditional Convolutional neural network (CNN) and Recurrent neural network (RNN) based deep learning systems are proven to have extraordinary performance in SL recognition [3][4][5]. However, the majority of automatic SL recognition systems rely on large-scale data training. ...
... In the classical training phase, we train a powerful CNN architecture named DenseNet-121 [18]. This model was selected on the basis of our previous research [5,19] that found that DenseNet extracts better features in SL recognition. We trained DenseNet-121 on the seen class dataset-BdSL. ...
Sign language is a unique form of communication in which hand or other body part gestures are used to express oneself. A large proportion of the world’s population has speech and hearing impairments and communicates through sign language. Sign language, like verbal language, varies from country to country. Recent researches on automatic recognition focus on specific sign language of a country and require a large dataset. However, a prevalent issue arises when there is plenty of data available for some sign languages, while other sign languages suffer from data scarcity or non-existence of resources. To tackle this issue, our study presents a novel solution by proposing a few-shot learning approach for automatic sign language recognition. This approach involves training the model using data from a single sign language and then leveraging the acquired knowledge to recognize other sign languages, even when limited data is available for those languages. By bridging the gap between limited data availability and accurate recognition of new Sign Languages via employing this few-shot learning technique, our approach contributes to enhancing communication accessibility for the global sign language community. Our experimental results demonstrate promising performance, showcasing the potential of our model in overcoming the challenges of cross-lingual sign language recognition.
... It has an overall accuracy of 99.22%. Nihal et al. [13] describe the design and creation of a unique, reasonably priced humanoid robot that can serve as an interpreter for sign language. They built an image dataset containing 950 images. ...
Bangla Sign Language (BdSL), also known as Bengali or Bangladeshi Sign Language, is the sign language that is commonly used for communication with the deaf people in Bangladesh and some parts of India, e.g., West Bengal (formally known as Calcutta), Assam, Tripura, and the Andaman and Nicobar Islands. Bangla is one of the sweetest languages in the world, according to the UNESCO survey, and the 7th most spoken language in the world. In Bangladesh, about 13.7 million people have a hearing problem, and the Bangladesh National Federation of the Deaf (BNFD) was established on December 25, 1963, to provide training, skill development, and education to deaf people. In this study, we have applied transfer learning in machine learning for BdSL recognition that reuses a pre-trained classifier named YOLOv5 to detect and classify BdSL. Transfer learning speeds up the training process with a minimum number of instances and improves classification performance. Transfer learning for deep learning has become very popular for solving computer vision and natural language processing (NLP) tasks nowadays. Initially, we collected a 24.2 GB video of BdSL from SignBD-word and extracted images from the video using the FFmpeg multimedia framework. Then, we annotated the images using the LabelImg image annotation tool. Next, we checked the image annotation and built the model using YOLOv5. YOLOv5 is a family of compound-scaled object detection models, which stands for “You Only Look Once”, commonly used for detecting objects by separating images into a grid structure. Finally, we have implemented the prototype with an average accuracy of 91.62%.
... Representative examples include studies using humanoids and animations. In studies using humanoids, the humanoids use sign language so that humans can understand them [20,21]. Animation-based sign language education, similar to humanoid education, expresses input words as animations so that students can observe and reproduce them [22]. ...
This study proposes the design and application of wearable gloves that can recognize sign language expressions from input images via long short-term memory (LSTM) network models and can learn sign language through finger movement generation and vibration motor feedback. It is difficult for nondisabled people who do not know sign language to express sign language accurately. Therefore, we suggest the use of wearable gloves for sign language education to help nondisabled people learn and accurately express sign language. The wearable glove consists of a direct current motor, a link (finger exoskeleton) that can generate finger movements, and a flexible sensor that recognizes the degree of finger bending. When the coordinates of the hand move in the input image, the sign language motion is fed back through the vibration motor attached to the wrist. The proposed wearable glove can learn 20 Korean sign language words, and the data used for learning are configured to represent the joint coordinates and joint angles of both the hands and body for these 20 sign language words. Prototypes were produced based on the design, and it was confirmed that the angle of each finger could be adjusted. Through experiments, a sign language recognition model was selected, and the validity of the proposed method was confirmed by comparing the generated learning results with the data sequence. Finally, we compared and verified the accuracy and learning loss using a recurrent neural network and confirmed that the test results of the LSTM model showed an accuracy of 85%.
... Sign language recognition has attracted much attention in the field of computer vision, with the goal to accurately recognize the movement of gestures and understand the meaning of sign language for communicators or machines. The recognition of sign language actions has huge application in robot perception [1], improving the lives of people with speech impairments [2] and enriching nonverbal information transfer. However, since sign language movements are generally too fast, the use of traditional frame cameras introduces great challenges, such as blur and overlap and computational complexity. ...
Sign language recognition has been utilized in human–machine interactions, improving the lives of people with speech impairments or who rely on nonverbal instructions. Thanks to its higher temporal resolution, less visual redundancy information and lower energy consumption, the use of an event camera with a new dynamic vision sensor (DVS) shows promise with regard to sign language recognition with robot perception and intelligent control. Although previous work has focused on event camera-based, simple gesture datasets, such as DVS128Gesture, event camera gesture datasets inspired by sign language are critical, which poses a great impediment to the development of event camera-based sign language recognition. An effective method to extract spatio-temporal features from event data is significantly desired. Firstly, the event-based sign language gesture datasets are proposed and the data have two sources: traditional sign language videos to event stream (DVS_Sign_v2e) and DAVIS346 (DVS_Sign). In the present dataset, data are divided into five classification, verbs, quantifiers, position, things and people, adapting to actual scenarios where robots provide instruction or assistance. Sign language classification is demonstrated in spike neuron networks with a spatio-temporal back-propagation training method, leading to the best recognition accuracy of 77%. This work paves the way for the combination of event camera-based sign language gesture recognition and robotic perception for the future intelligent systems.
... 2016 yılında gerçekleştirilen bir diğer çalışmada NINO adı verilen insansı robot ile Tayvan işaret dili ifadeleri başarılı bir şekilde gerçekleştirildiği rapor edildi [39]. 2021 yılında yapılan bir diğer çalışmada ise Bangla işaret dili ifadeleri geliştirilen insansı robot ile başarılı bir şekilde gerçekleştirildi [40]. ...
Highlights: Graphical/Tabular Abstract Turkish Sign Language Deep Learning Humanoid Robots in Education In this study, a humanoid robot was developed for Turkish Sign Language learning. The working principle flow chart of the humanoid robot developed within the scope of the study is shown in Figure A. Figure A. Flowchart of EC-Tema humanoid robot working principle Purpose: Development of humanoid robot for Turkish Sign Language education. Theory and Methods: While there are approximately 466 million hearing and speech-impaired individuals worldwide, there are approximately 1.5 million hearing and speech-impaired individuals in Türkiye. Learning the sign language used by people with disabilities to communicate with each other or with their immediate environment at an early age is extremely important in terms of both cognitive and intellectual development and academic development. Furthermore, humanoid robots, which are an output of developing technology, have different usage areas. It can be shown to be used in education among these usage areas. It has been reported that humanoid robots have been used for educational purposes in subjects such as sign language learning, and successful results have been obtained. Results: Within the scope of the study, an improvable humanoid robot for Turkish Sign Language education was developed, and a high success rate was achieved. Conclusion: A humanoid robot was developed for sign language learning for the first time in Türkiye, and it will be aimed to eliminate the identified deficiencies within the scope of the study as well as to increase human-robot interaction in the follow-up study.
... 2016 yılında gerçekleştirilen bir diğer çalışmada NINO adı verilen insansı robot ile Tayvan işaret dili ifadeleri başarılı bir şekilde gerçekleştirildiği rapor edildi [39]. 2021 yılında yapılan bir diğer çalışmada ise Bangla işaret dili ifadeleri geliştirilen insansı robot ile başarılı bir şekilde gerçekleştirildi [40]. ...
Dünya genelinde 466 milyon Türkiye’de ise yaklaşık 1.5 milyon işitme ve konuşma engelli bireyin kendi aralarında ve diğer bireyler ile iletişimlerini sağlamak için kullandıkları dil işaret dilidir. El, yüz ve vücut mimikleri ile gerçekleştirilen bu dil, ülkeden ülkeye ve konuşulduğu ülkelerde ise bölgeden bölgeye değişiklik gösterebilen dinamik bir dildir. Bireyin daha küçük yaşta işaret dili öğrenmesi, bireyin hem bilişsel ve entelektüel gelişimi hem de akademik gelişimi açısından önemli olması bu dilin önemini gözler önüne sermektedir. Bu nedenle işaret dili öğrenimi konusunda teknolojinin kullanılmasına dönük geliştirilen bir dizi çalışma bulunmaktadır. Geliştirilen insansı robotların işaret dili öğreniminde başarılı bir şekilde icra edildiği kanıtlandı. Ülkemizde işaret dili öğrenimi üzerine geliştirilen herhangi bir insansı robotun bulunmaması ve bu alanda yapılan güncel çalışmaların literatürde yer edinmesi üzerine Türkçe İşaret Dili ifadelerini gerçekleştiren insansı robot çalışması yapıldı. Çalışma kapsamında insansı robotun geliştirme aşamaları detaylı bir şekilde tartışılmaktadır.
... From paper [9], the test accuracy of the DenseNet architecture model used for recognizing Bangladesh Sign Language is 88.6%, as shown in table 1. The proposed model has been able to improve the efficiency of the model using the same architecture upto 98.07% to recognize American Sign Language images and convert them to English letters. ...
This paper presents the design of a low-cost, portable robotic arm for American Sign Language (ASL) interpretation and sentiment analysis. To mitigate the complexity of communication between the deaf and dumb and common beings, a two-way communication and interpretation model has been designed to translate the English language into ASL symbols on the robotic arm and ASL into the English language using image recognition. Along with translation, the model can also analyse the sentiments of the mute people and has an auto-text completion feature for ease of typing, thus improving the entire calibre of interaction and making it more human-like.
... They implemented segmentation of hand gestures, binary masking and SVM classification and found 99.8 accuracies for static hand poses. The most recent notable work done by Nihal et al [49] developed a robust humanoid that can capture real-time image, video data and identify 38 alphabets of BdSL with 93.8% accuracy in the available image dataset and with 98.19% accuracy in their selfmade dataset. Additionally, video-based medical sign recognition achieved 87.5% test accuracy. ...
Despite sign language being the primary medium of conversation with deaf and dumb (DD) individuals, it is hard to grasp for any class of people, making communication immensely challenging. Bengali being one of the most spoken languages globally, significant research works on Bangla Sign Language (BdSL) have come to light to deal with this challenge. In recent years, researchers from diverse backgrounds have proposed solutions to automate the process of BdSL recognition. In this literature review, we have explored the trends of research on BdSL by comparing incorporated features and evaluation results of systems between 2002-2021 and techniques applied on different existing and new datasets. We have also collected and combined metadata from datasets of all BdSL Alphabets and numerics used so far. The literature findings of this paper indicate that most proposed models perform well on images containing static and single-handed signs but, performance falls significantly in complex backgrounds. Moreover, we focused on identifying insights and similarities of the existing system, finding research gaps and propose possible future directions. We expect this article will draw the attention of more researchers into this domain and this is the first identifiable academic literature review of BdSL recognition systems to the best of our knowledge.
... Over years, several attempts have been taken to develop Bangla sign language (BdSL) recognition system using Convolutional Neural Network (CNN) [1], Artificial Neural Network (ANN) [2], Geometric hashing [3], and so on. Like several other sign languages (e.g., American / British sign language), Bangla sign language can be expressed by both: showing special signs for words/sentences and finger-spelling using sign alphabets [4,5]. Fingerspelling helps deaf and dumb children learn how to read and write SL. ...
Bangla, being the fifth most spoken language in the world has its own distinct sign language with two methods (one-handed and two-handed) of representation. However, a standard automatic recognition system of Bangla sign language (BdSL) is still to be achieved. Though widely studied and explored by researchers in the past years, certain unaddressed issues like identifying unseen signs and both types of BdSL or lack of evaluation of the models in versatile environmental conditions demarcate the real-world implementation of the automatic recognition of BdSL. To find a probable solution to the shortcomings in the existing works, this paper proposes two approaches based on conventional transfer learning and contemporary Zero-shot learning (ZSL) for automatic BdSL alphabet recognition of both seen and unseen data. The performance of the proposed system is evaluated for both types of Bangla sign representations as well as on a large dataset with 35,149 images from over 350 subjects, varying in terms of backgrounds, camera angle, light contrast, skin tone, hand size, and orientation. For the ZSL approach, a new semantic descriptor dedicated to BdSL is created and a split of the dataset into seen and unseen classes is proposed. Our model achieved 68.21%, 91.57%, and 54.34% of harmonic mean accuracy, seen accuracy, and zero-shot accuracy with six unseen classes respectively. For the transfer learning-based approach, we found pre-trained DenseNet201 architecture to be the best performing feature extractor and Linear Discriminant Analysis as the best classifier with an overall accuracy of 93.68% on the large dataset after conducting quantitative experimentation on 18 CNN architectures and 21 classifiers. The satisfactory result from our models supports its very probative potential to serve extensively for the hearing and speaking impaired community.
