How to enable worker cleanup in Spark

Question

I have a Stand-alone Spark application running on my VM. I want to enable the cleanup feature to periodically clean up the worker directories. How to do this?

Answer 1

To enable cleanup, open the spark shell and run the below commands:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.worker.cleanup.enabled=true

Note that this will work only for stand-alone applications