What are the various ways to import files into HDFS

Question

I am new to Hadoop and I want to know what are the different ways through which I can bring data into HDFS.

I have a web application that is creating log files, how can i import the log files into HDFS?

Answer 1

There are various tools and frameworks available in Hadoop Ecosystem, using which you can import and export data to HDFS & from HDFS respectively.

let me tell you few of them and their use cases:

1. hdfs dfs -put - simple way to insert files from local file system to HDFS
2. HDFS Java API
3. Sqoop - for bringing data to/from databases
4. Flume - streaming files, logs
5. Kafka - distributed queue, mostly for near-real time stream processing
6. Nifi - incubating project at Apache for moving data into HDFS without making lots of changes

So for your purpose, the best solution for bringing web application logs to HDFS is through Flume.