Zookeeper server going down frequently

Question

Out of 3 servers, one ZK server going down frequently. It was up and running successfully for quite a long time. There were no traces of errors in the log file also. Kindly help me !!!

Comments

Is there any other way, instead of dropping the already running one?
Because out of 3, 2 are running fine and only one is giving trouble. If I remove the zookeeper, it would be a huge impact across the cluster.

Kindly advise on this hypothetic situation.

Answer 1

Hi@B,

There may be several reasons behind this. First, check the firewall. If it is fine then, there may be a problem with the log file. You may delete the previous log file. Or check the log file properly. It should show the error message.

Comments

Hi MD,

There is no firewall enabled. I checked the log files, they are empty. Not evidence for issue anywhere in the ZK server.

---

Hi@B,

I think there may be a problem with permission. Check if the user has permission or not.

---

Required permissions are there.
Anything specific user permission need to be validated?

FYI., A couple of days ago I have upgraded the HDP version to 2.6.5; since then I am facing this issue.

---

Hi@B,

You have upgraded your HDP. But your other tools remain the same. The thing is we should check which one is compatible with which one be upgrading. Do one thing shut down your zookeeper and create a new one and try to connect again.

---

Is there any other way, instead of dropping the already running one?
Because out of 3, 2 are running fine and only one is giving trouble. If I remove the zookeeper, it would be a huge impact across the cluster.

Kindly advise on this hypothetic situation.

---

Hi@B,

You can use tcpdump command to troubleshoot. I mean run the tcpdump command to that node and capture the network packet. And see what it is showing.

---

Now this issue also has been resolved. Issue was due to old Zookeeper process was running and blocking the new request from Ambari; that's why whenever I attempt to start the server, it comes up and goes down after few seconds.

Throwed this cmd #ps -ef | grep -i zookeeper; kill -9 <pid>, started the faulty zookeeper server and issue got resolved by that way.

Anyways, thanks a lot for spending your valuable time in advising the possible solutions.

Really appreciated your help!!!