Remove idle executor automatically

Question

I want to configure my application such that if an executor has been idle for more than a particular time, then it should be removed. Is this possible?

Answer 1

Yes, it is possible and is already enabled. I don't think you have observed but when an executor is idle for more than 60s, it is automatically removed. This is means that the default time set is 60s. You can change it like this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.dynamicAllocation.executorIdleTimeout= <new time>