How to prevent executor from self-destructing

Question

Hi. The executors running are getting self-destructed. I tried to trace the problem and I think it is because some task had not stopped running. How to prevent this?

Answer 1

I think there is a timeout set after which if the task does not stop running, then the executor will self-destruct. Try disabling it as follows:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.task.reaper.killTimeout=-1