A Review of Deep Learning for Real-Time Sign Language Recognition

Abstract: Communication is the fundamental exchange of thoughts and messages through speech, signals, visuals, and behavior. For deaf and hard-of-hearing individuals, sign language serves as the primary medium for interaction. However, because most non-disabled individuals do not understand sign language, a significant communication barrier exists. This paper presents a user-friendly Human-Computer Interface (HCI) software system designed to translate American Sign Language (ASL) fingerspelling hand gestures into text and speech in real time. The proposed system captures gesture images via a webcam, extracts hand landmarks using the MediaPipe library, and projects them onto a clean white canvas to overcome real-world lighting and background variations. A Convolutional Neural Network (CNN) model is trained on these skeletal representations to recognize individual alphabets (A–Z) and form complete words. The recognized text is then converted into spoken audio using the pyttsx3 library. The proposed approach achieves 97% recognition accuracy under standard conditions and over 99% accuracy in controlled environments.

Keywords: American Sign Language, Convolutional Neural Network, MediaPipe, Human-Computer Interface, Gesture Recognition, Text-to-Speech Conversion.