Question

What are the levels of parallelism in spark streaming

Answer 1

> In order to reduce the processing time, one needs to increase the parallelism.
> Spark Streaming provides three ways to increase the parallelism :
(1) Increase the number of receivers: If there are too many records for a single receiver (single machine) to read in and distribute so that is a bottleneck. So we can increase the no. of the receiver depending on the scenario.
(2) Re-partition the receive data: If one is not in a position to increase the no. of receivers, in that case, redistribute the data by re-partitioning.
(3) Increase parallelism in aggregation

Comments

Clusters will not be fully utilized unless the level of parallelism for each operation is high enough. Spark automatically sets the number of partitions of an input file according to its size and for distributed shuffles. By default spark create one partition for each block of the file in HDFS it is 64MB by default