Spark logs not overwriting

Question

Hi. I had a created a Spark application a long time ago and Spark has stored the logs. I don't need them now. I am running another application with the same name and what I want is that the logs of the new application should overwrite the logs of the previous application. How can I make this happen?

Answer 1

Spark does not allow you to overwrite the files by default. There are two ways you can handle the situation.

You can either go to the location  file:///tmp/spark-events delete the directory of the previous application or its logs. 

Or you can just make Spark allow you to overwrite the log files:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.eventLog.overwrite=true