Why do jobs have to be idempotent - kubernetes cron jobs

Question

The documentation says the following:

A cron job creates a job object about once per execution time of its schedule. We say “about” because there are certain circumstances where two jobs might be created, or no job might be created. We attempt to make these rare, but do not completely prevent them. Therefore, jobs should be idempotent

Could anyone explain why would this happen and how can i get it fixed?

Answer 1

To understand why this happens lets for instance take the node hosting the CronJobController, this node fails at the time a job is supposed to run.

The reason why this occurs is because there are no idemopotency-related issues under the area/batch label in the k8s repo, so one would guess not.