TWO-WAY INDIAN SIGN LANGUAGE COMMUNICATION ASSISTANT USING MACHINE LEARNING

Abstract: It can be sometimes a tough challenge for the Indian Sign Language (ISL) user to communicate with the non- ISL learner. In this regard, we decided to address the issue by creating a Two Way Indian Sign Language Communication Assistant using Machine Learning. Instead of simply converting the sign to speech, our system focuses on both sides of communication. The system also provides the extra functionality for ISL learners. A webcam is used for capturing the sign gestures. The points of interest for hands, face and body are identified using MediaPipe Holistic. All these points are taken under consideration for recognizing the gestures through LSTM model analysis of multiple video frames. Signs recognized are then converted to text and spoken. In case of communication by speech, it takes input from the microphone, converts to text, and shows corresponding signs in the form of images/videos/animations. Apart from this, there is a learning mode, where one can practice and get familiar with the sign gestures of ISL. All these features combine sign recognition, speech conversion and learning of sign language into a single application. The goal of this project is to ease the communication between ISL users and non-signers and also to create more awareness about the language.

Keywords: Indian Sign Language , Sign Language Recognition, Two-Way Communication, Speech-to-Sign Translation, LSTM, MediaPipe Holistic, Gesture Recognition, Human-Computer Interaction, Assistive Technology, Sign Language Learning