Does spark streaming provides checkpoint

Question

Hi,

Does Spark streaming provides checkpoint to recovery or fault tolerance?

Thank You

Answer 1

Hi@akhtar,

Yes, Spark streaming uses checkpoint. Checkpoint is the process to make streaming applications resilient to failures. There are mainly two types of checkpoint one is Metadata checkpoint and another one is Data checkpoint.

Metadata checkpoint is used for recovery from a node failure.

Data checkpoint is used for fault tolerance in HDFS.

Thank You