How to disable existing directory check

Question

Can someone tell me how to disable the check whether the output directory exists or not? I keep getting an exception every now and then and it is frustrating. Thanks.

Answer 1

To disable this, run the below commands:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.hadoop.validateOutputSpecs=false

I don't suggest you to disable this because it does different checks. Instead, you can use Hadoop's FileSystem API to delete output directories