How to increase wait time to launch data-local task

Question

I am running long tasks and I see poor locality. I was doing some research and found that this can be optimized using locality wait. But the problem is that I don't know how to increase it. Please help.

Answer 1

You can increase the locality wait time using the following command:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.locality.wait= <time in seconds>