Question

what is the impact of uprading kubelet if we leave the pods on the worker nodes - will it break running pods? why?

Answer 1

• Restarting kubelet, which has to happen for an upgrade will cause all the Pods on the node to stop and be started again. 

• It’s generally better to drain a node because that way Pods can be gracefully migrated, and things like Disruption Budgets can be honored. 

• The problem is that `kubectl` keeps up with the state of all running pods, so when it goes away the containers don’t necessarily die, but as soon as it comes back up, they are all killed so `kubectl` can create a clean slate. 

• As kubelet communicates with the apiserver, so if something happens in between of upgrade process, rescheduling of pods may take place and health checks may fail in between the process. 

• During the restart, the kubelet will stop querying the API, so it won’t start/stop containers, and Heapster won’t be able to fetch system metrics from cAdvisor. 

• Just make sure it’s not down for too long or the node will be removed from the cluster!