Is there a way to copy data from one one Hadoop distributed file system HDFS to another HDFS

Question

I have two HDFS setup and want to copy (not migrate or move) some data from HDFS1 to HDFS2. How to do I copy data from one HDFS to another HDFS? Is it possible via Sqoop or other command line?

Answer 1

The distributed copy command, distcp, is a utility that allows you to copy large data sets between two DFS cluster. The distcp command submits a regular MapReduce job that performs a file-by-file copy.

Usage:

hadoop distcp <source> <destination>

Example:

hadoop distcp hdfs://namenode1:9820/source_dir hdfs://namenode2:9820/destination_dir