How to integrate Hadoop with Elasticsearch

Question

I want to integrate hadoop with elastic search so, for this I have set up a Hadoop cluster with 3 DataNodes and 1 NameNode with elastic search installed on one of the DataNodes. But I’m not able to access HDFS using elastic search. Some one tells me that there is a separate plugin for hadoop and elastic search. Can someone tell me which plugin should I download and steps to integrate hadoop with elastic search.

Thanks in advance!

Answer 1

The HDFS snapshot/restore plugin comes in three flavors:

• Default / Hadoop 1.x The default version contains the plugin jar alongside Hadoop 1.x (stable) dependencies
• Yarn / Hadoop 2.x The hadoop2 version contains the plugin jar plus the Hadoop 2.x (Yarn) dependencies.
• Light The light version contains just the plugin jar, without any Hadoop dependencies.

What version to use?

It depends on whether you have Hadoop installed on your nodes or not. If you do, then it may be easiest to expose Hadoop to the Elasticsearch classpath (typically by creating symbolic links to the client jars within the Elasticsearch lib/ directory) and using the light version. This guarantees the existing libraries and configuration are being picked up by the plugin. If you do not have Hadoop installed, then select either the default version (for Hadoop stable/1.x) or, if you are using Hadoop 2, the hadoop2 version.

Below are the two repo links given which will help you to setup Hadoop with Elasticsearch.

https://github.com/elastic/elasticsearch-hadoop/tree/master/repository-hdfs

https://github.com/elastic/elasticsearch-hadoop