Loading Spark properties dynamically

Question

Hi. I have different masters running on Spark. I need the same configurations for 3 masters but a different configuration for the 4th master. Due to this, I do not want to change the configuration in the conf file. How can I use different configuration only for this master?

Answer 1

First, create an empty conf using this code:

val sc = new SparkContext(new SparkConf())

Then, you can supply configuration values at runtime. For example:

./bin/spark-submit --name "My app" --master local[4] --conf spark.eventLog.enabled=false
  --conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" myApp.jar