How many files can hive create in HDFS duringbMapReduce job

Question

Is there a limit on how many files Hive can create on HDFS during a MapReduce job or can any number of files be created? I am writing a MapReduce program that involves creating files on HDFS. I wanted to know if there are any limits so that my program does not exceeds it.

Answer 1

There is a limit on how many files you can create on HDFS during MapReduce. The maximum number of files by default is set to 100000. But you can change this limit as follows:

$ hive --hiveconf hive.exec.max.created.files=<set the new limit>