How to disable executor from fetching file from cache

Question

Hi. I am running a Spark application and there are many executors running. I think the executors are picking up data from the cache due to which the application is going into some problem with updating new data. Is there a way to disable this?

Answer 1

When a Spark application is running, the executors by default will fetch data from a local cache. To change this and make the executors get data own copy of files, you have to modify the spark.files.useFetchCach property. You can do it like this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.files.useFetchCache=false