Difference between map and mapPartitions function in Spark

Question

Hi everyone,

Can someone tell me, what is the basic difference between map() and mapPartitions() in Spark??

Please give an example.

Thank You

Answer 1

Hi@ akhtar,

Both map() and mapPartitions() are the transformation present in spark rdd.

Consider, You have a file which contains 50 lines and there are five partitions. Each partitions contains 10 lines. 

If you use map(func) to rdd, then the func() will be applied on each and every line and in this particular case func() will be called 50 times. So, it will take more time to process.

If, you use mapPartitons(func) to rdd. then the func() will be applied on each partitions and in this case func() will be called 5 times. So, the processing speed will be more.

Hope this will help you

Thank You

Comments

then why we have map function

---

Hi,

It depends on the use cases. Maybe sometimes we require a map and sometimes mapPartitions. You need to use according to your requirement.