unable to start Kubernetes due to so many open files in system

Question

I tried creating bunch of pods, services and deployment using Kubernetes, but getting the following errors when I run the kubectl describe command.

for "POD" with RunContainerError: "runContainer: API error (500): Cannot start container bbdb58770a848733bf7130b1b230d809fcec3062b2b16748c5e4a8b12cc0533a: [8] System error: too many open files in system\n"

I terminated all pods and try restarting the machine, but it's not working.I was wondering how to get all the open files and close them at once? Can anyone help me with this?

Answer 1

You can try the following steps:

You can find which process is hogging file descriptors by running the following command:

lsof | awk '{print $2}' | sort | uniq -c | sort -n

You will get a sorted list of open FD counts with the pid of the process. Then you can look up each process w/

ps -p <pid>

If the main hogs are docker/kubernetes, then I would recommend following along on the issue th caesarxuchao reference.

I think it would be helpful for you.