what is Paired RDD and how to create paired RDD in Spark

Question

Can anyone explain how to create paired RDD in Spark?

Answer 1

Hi,

Paired RDD is a distributed collection of data with the key-value pair. It is a subset of Resilient Distributed Dataset So it has all the features of RDD and some new feature for the key-value pair. There are many transformation operations available for Paired RDD. These operations on Paired RDD are very useful to solve many use cases that require sorting, grouping, reducing some value/function.
Commonly used operations on paired RDD are: groupByKey() reduceByKey() countByKey() join() etc.

Here is an example regarding creating of paired RDD:

val pRDD:[(String),(Int)]=sc.textFile(“path_of_your_file”)
.flatMap(line => line.split(” “))
.map{word=>(word,word.length)}