Pyspark rdd How to get partition number in output

Question

z = sc.parallelize([1,2,3,4,5,6], 2)

How to get partition number in output?

Answer 1

The glom function is what you are looking for:

glom(self): Return an RDD created by coalescing all elements within each partition into a list.

a = sc.parallelize(range(10), 5)
a.glom().collect()
#output:[[0, 1], [2, 3], [4, 5], [6, 7], [8, 9]]

To know more about Pyspark, it's recommended that you join PySpark course today.