Zookeeper server going down frequently

Question

Out of 3 servers, one ZK server going down frequently. It was up and running successfully for quite a long time. There were no traces of errors in the log file also. Kindly help me !!!

Is there any other way, instead of dropping the already running one?
Because out of 3, 2 are running fine and only one is giving trouble. If I remove the zookeeper, it would be a huge impact across the cluster.

Kindly advise on this hypothetic situation. — B, Nov 11, 2020

MD · Answer 1 · Nov 9, 2020

Hi@B,

There may be several reasons behind this. First, check the firewall. If it is fine then, there may be a problem with the log file. You may delete the previous log file. Or check the log file properly. It should show the error message.

answered Nov 9, 2020 by MD
• 95,460 points

Hi MD,

There is no firewall enabled. I checked the log files, they are empty. Not evidence for issue anywhere in the ZK server.

commented Nov 10, 2020 by B
• 210 points

Hi@B,

I think there may be a problem with permission. Check if the user has permission or not.

commented Nov 10, 2020 by MD
• 95,460 points

Required permissions are there.
Anything specific user permission need to be validated?

FYI., A couple of days ago I have upgraded the HDP version to 2.6.5; since then I am facing this issue.

commented Nov 11, 2020 by B
• 210 points

Hi@B,

You have upgraded your HDP. But your other tools remain the same. The thing is we should check which one is compatible with which one be upgrading. Do one thing shut down your zookeeper and create a new one and try to connect again.

commented Nov 11, 2020 by MD
• 95,460 points

Is there any other way, instead of dropping the already running one?
Because out of 3, 2 are running fine and only one is giving trouble. If I remove the zookeeper, it would be a huge impact across the cluster.

Kindly advise on this hypothetic situation.

commented Nov 11, 2020 by B
• 210 points

Hi@B,

You can use tcpdump command to troubleshoot. I mean run the tcpdump command to that node and capture the network packet. And see what it is showing.

commented Nov 11, 2020 by akhtar
• 38,260 points

Now this issue also has been resolved. Issue was due to old Zookeeper process was running and blocking the new request from Ambari; that's why whenever I attempt to start the server, it comes up and goes down after few seconds.

Throwed this cmd #ps -ef | grep -i zookeeper; kill -9 <pid>, started the faulty zookeeper server and issue got resolved by that way.

Anyways, thanks a lot for spending your valuable time in advising the possible solutions.

Really appreciated your help!!!