Getting buffer limit exceeded exception inside Kryo

Question

Hi. I am running a Spark application and I am facing some problem in serialization. During the serialization, I am getting the following error in Kyro:

buffer limit exceeded

Answer 1

Seems like the object being sent for serialization is exceeding the buffer size of Kyro. By default the maximum allowed size is 64MiB and to increase this, you can do the following:

val sc = new SparkContext(new SparkConf())

./bin/spark-submit <all your existing options> --spark.kryoserializer.buffer.max=<new size>