AI Powered Sign Language to Text and Speech

Abstract: Communication between individuals who use sign language and those unfamiliar with it can be challenging, particularly in everyday interactions. This paper presents an AI-powered sign language recognition system designed to convert hand gestures into text and speech in Kannada, Hindi, and Tamil. The proposed system uses a webcam to capture static hand gestures and MediaPipe to detect hand landmarks and extract spatial features. A Multilayer Perceptron (MLP) model classifies the extracted features into supported sign characters. The recognized characters are then combined to form words and sentences using language-specific text composition rules. Large Language Model (LLM) based text refinement is incorporated to improve the grammatical correctness and readability of the generated text. The refined text is displayed through a user interface and converted into speech using a Text-to-Speech (TTS) module. The proposed system demonstrates the potential of computer vision, machine learning, and natural language processing in supporting multilingual assistive communication and improving accessibility for sign language users.

Keywords: Artificial Intelligence, Sign Language Recognition, Gesture Recognition, Text to Speech, Large Language Model