Interactive Multimodal Visual Search Using Voice Command on Mobile Device

Abstract: Mobile phones have involved into powerful image and video processing devices equipped with built-in cameras, color displays, and hardware-accelerated graphics. These more features allow users to give multimodal queries for searching information on the go from the world wide web. In this paper, we propose a multimodal image search system that fully utilized multimodal and multi-touch functionalities of smart phones. The system allows searching images on the web by using an existing image query or a speech query with the help of existing image search engine. If the user doesn�t have an existing image query or captured photo, they can input a speech query that clearly represents a picture description in the user�s mind. The proposed system enhances the mobile search experience and increases relevance of search results. It involves a natural interactive process through which user has to express their search content very well.

Keywords: multimodal search, visual search, mobile phone, interactive search, information retrieval