Increase Yarn wait time for Sparkcontext

Question

Hi. I want to increase the time that the Yarn application master waits for the SparkContext to be initialized. What is the command I have to use to do this?

Answer 1

The default time that the Yarn application waits for the SparkContext is 100s. If you want to change it, open the Spark shell and run the following command by setting the new wait time:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.yarn.am.waitTime=<new wait time>