Log every block update in Spark

Question

How should I configure my Spark job to log all the events for every block update? Right now it logs only limited (or less) logs.

Answer 1

By default, Spark does not log all the events for block update. But you can configure it to log them. To do this, change the value of the property  spark.eventLog.logBlockUpdates.enabled to true

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.eventLog.logBlockUpdates.enabled=true

Note: This will increase the size of the log file too much and is not suggested.