How to change block manager port for driver only

Question

I am facing a problem with the port of Block manager. I changed the port from random to  5545 to which the Block managers listen. This set the port for executors and the drivers. Now, the problem is that the drivers can not use this port because it is being used by another process. Is it possible to change the port only for the driver?

Answer 1

Yes, it is possible to change the port only for the driver. To do this, run the following in the Spark shell:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.driver.blockManager.port= <new port number>