How to increase Spark listener bus event queue capacity

Question

I am running a scheduler in Spark. I am facing a problem with the listener bus. The problem is that the listener events are getting dropped. So, I want to increase the capacity of the listener bus to hold more event queue. How can I do this?

Answer 1

The default capacity of listener bus is set to 10000. Try changing it to 20000:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.scheduler.listenerbus.eventqueue.capacity=20000

Comments

Is it spark.scheduler.listenerbus.eventqueue.size or spark.scheduler.listenerbus.eventqueue.capacity?

---

I think, it is spark.scheduler.listenerbus.eventqueue.capacity. In spark documentation, there is no property available as spark.scheduler.listenerbus.eventqueue.size. You can go through the below link.

https://people.apache.org/~pwendell/spark-nightly/spark-master-docs/latest/configuration.html