Why is Spark map output compressed

Question

I am using Spark for MapReduce and I see that the output file after map phase is always compressed. Why is this happening?

Answer 1

Spark thinks that it is a good idea to compress output files and it is in fact right. The reason for the compression of output files is due to the property spark.shuffle.compress. This property is used to decide whether the output file should be compressed or not and by default is set to true. If you do not want the output to be changed then you can change this property dynamically:

./bin/spark-submit --conf spark.shuffle.compress=false