How will you decide whether you need to use the Capacity Scheduler or the Fair Scheduler

Question

I know how fair scheduler will handle more than one job, but doesn't know how to decide the use of Capacity Scheduler. Can you help me out with this.

Answer 1

Fair Scheduling is the process in which resources are assigned to jobs such that all jobs get to share equal number of resources over time. Fair Scheduler can be used under the following circumstances -

i) If you wants the jobs to make equal progress instead of following the FIFO order then you must use Fair Scheduling.

ii) If you have slow connectivity and data locality plays a vital role and makes a significant difference to the job runtime then you must use Fair Scheduling.

iii) Use fair scheduling if there is a lot of variability in the utilization between pools.

Capacity Scheduler allows runs the Hadoop mapreduce cluster as a shared, multi-tenant cluster to maximize the utilization of the hadoop cluster and throughput. Capacity Scheduler can be used under the following circumstances -

i) If the jobs require scheduler determinism then Capacity Scheduler can be useful.

ii) CS's memory based scheduling method is useful if the jobs have varying memory requirements.

iii) If you want to enforce resource allocation because you know very well about the cluster utilization and workload then use Capacity Scheduler.