Classification of Sign Language Gestures using Machine Learning

Abstract: The low awareness of sign language among the general public presents a hurdle in communication with the deaf and dumb communities and their integration within society. A sign language translator application can help reduce this communication gap. This paper presents a system developed to detect fingerspelling gestures from video and give their English equivalent letter using machine learning, with a focus towards developing a potential solution for everyday use. A classifier is trained on a dataset of 24 fingerspelling gestures using the bag of visual words approach, wherein features detected in the images are clustered to form a codebook, then each image is expressed as a histogram denoting the frequency of observed codewords in that image. The SURF and BRISK algorithms are explored for automatic feature detection. Four classification algorithms are evaluated, namely K-nearest neighbours, logistic regression, Naive Bayes and Support Vector Machine. The best performing model has been used to classify sign language gestures from video frames.

Keywords: Bag of visual words, codebook, descriptors, histogram of codewords, image processing, feature detection, machine learning, sign language, speeded up robust features