When is speculation enabled in Spark

Question

Hi. I have a doubt regarding Task speculation in Spark. When does the speculation happen? When is it enabled?

Answer 1

Speculation is enabled when a fraction of task is completed. By default, it is set to 0.75. So, when 75% of the task is complete, speculation is enabled. If you want to change this fraction, you can do it like this:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.speculation.quantile=<fraction>