Responsive Job Scheduling for Map-Reduce in Hadoop Framework

Abstract: Hadoop is a framework which is used to store and process large amount of data. Scheduling of Jobs is very much important to achieve high performance in Hadoop cluster. Hadoop scheduler is pluggable module which is used to manage the tasks for executions. Most commonly used schedulers are FIFO, Fair and Capacity scheduler. In this paper we have implemented Responsive Job Scheduler based on locality of Reference, it would add fair and capacity scheduler�s job selection features to our algorithm. Data locality ensures that the map tasks will be executed on the node encompassing the input data. Though the proposed algorithm is designed specifically for map reduce framework and it can be very well implemented in Hadoop environment.

Keywords: Hadoop, MapReduce, Job Scheduler, Responsive Job Scheduling.