SparkContext addFile not able to update file

Question

I want to make changes to a file and I am using the sparkContect.addFile() for it. I am using this method and using the same name as the existing file but the file is not updating. Please help. 

Answer 1

Spark by default won't let you overwrite the file using the sparkContect.addFile() method. To enable the overwrite, you need to make changes to the property:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.files.overwrite=true