Hadoop Unable to load native-hadoop library for your platform warning

Question

I installed Hadoop on a server running centos. Whenever I run start-dfs.sh or stop-dfs.sh, I get the following error:

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

My Hadoop version is - Hadoop 2.2.0.

Can anyone tell me why I m getting this warning, I am not sure what to do.

I've also added these two environment variables in hadoop-env.sh:

export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/usr/local/hadoop/lib/"

export HADOOP_COMMON_LIB_NATIVE_DIR="/usr/local/hadoop/lib/native/"

Any suggestions?

Answer 1

modify the glibc version.CentOS provides safe softwares tranditionally,it also means the version is old such as glibc,protobuf ...

ldd --version
ldd /opt/hadoop/lib/native/libhadoop.so.1.0.0

You can compare the version of current glibc with needed glibc.

Secondly: If the version of current glibc is old,you can update the glibc. 

If the version of current glibc id right,you can append word native to your HADOOP_OPTS

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

You can get a better understanding with the Online big data courses.

Answer 2

There may be two possibilities:

1. You have installed the wrong Java JDK8 package. Please ensure to download the 64-bit JDK8 and remove your current 32-bit JDK8.
2. It's a warning due to the Hadoop libraries being compiled for 32bits. You are probably running on a 64bit OS.

It's safe to ignore this warning.

You can even check out the details of Hadoop Ecosystem tools with the Online big data courses. 

Answer 3

I had the same issue. It's solved by adding following lines in .bashrc:

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

Comments

its not working

---

Hi! Did you try the solutions mentioned in other answers?

---

It is not working

---

Could you mention the steps you followed to install Hadoop?

Answer 4

In my case , after I build hadoop on my 64 bit Linux mint OS, I replaced the native library in hadoop/lib. Still the problem persist. Then I figured out the hadoop pointing to hadoop/lib not to the hadoop/lib/native. So I just moved all content from native library to its parent. And the warning just gone.

Comments

how do you know what lib directory Hadoop is pointing to? I have this issue and all the env variables cannot solve it.

---

Hadoop has native implementations of certain components for performance reasons and for non-availability of Java implementations. These components are available in a single, dynamically-linked native library called the native hadoop library.

So, you have to add native directory in .bashrc file.

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

Answer 5

This also would work:

export LD_LIBRARY_PATH=/usr/lib/hadoop/lib/native

Comments

this works fine very good

---

works fine thanks mate...

Answer 6

Hello. I am using OSX and face the same problem. Any idea how to solve it?

Comments

Follow these steps replacing the path and Hadoop version where appropriate

wget
tar xvf hadoop-2.7.1-src.tar.gz
cd hadoop-2.7.1-src
mvn package -Pdist,native -DskipTests -Dtar
mv lib /usr/local/Cellar/hadoop/2.7.1/

then update hadoop-env.sh with

export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc= -Djava.library.path=/usr/local/Cellar/hadoop/2.7.1/lib/native"

Answer 7

Make sure you have set proper paths:

# JAVA env
#
export JAVA_HOME=/user/sys/jdk
export JRE_HOME=/user/sys/jdk/jre

export PATH=${JAVA_HOME}/bin:${JRE_HOME}/bin:${PATH}:.

# HADOOP env
#
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

Answer 8

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

Answer 9

Update your jdk and try again

Answer 10

Add this path:

export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH

source ~./bashrc

Answer 11

Hadoop Unable to load native-hadoop library for your platform warning