How to copy file to multiple hdfs directory

Question

I have a file in local system that I want to copy to many hdfs directories. I had to copy it 3 directories so I did it manually. I was just wondering, what if I had to copy the file into more directories, say 100. Is there a faster way to do this?

Answer 1

You can use loop to copy the file to multiple directories, I found a shell script which does the same:

m=10
for (( i = 0; i < 100; i++ )); do
   sh hdfs_cp_script partition$(($i*$m + 1)) & sh hdfs_cp_script partition$(($i*$m + 2) & ... & sh hdfs_cp_script partition$(($i*$m + 10))
done