Hive How to compress intermediate jobs

Question

Hi. I have set compression for MapReduce job output as follows:

$ hive --hiveconf hive.exec.compress.output=true

But I have observed that the output is being compressed and not the intermediates. Is there a way to compress the intermediates too?

Answer 1

The option hive.exec.compress.output will only compress the output. To compress the intermediates, you have to enable intermediate compression as follows:

$ hive --hiveconf hive.exec.compress.output=true hive.exec.compress.intermediate=true