Query regarding a spark split logic

Question

I have a csv file containing 10 lines. I need to have a spark code to split even and odd numbers of lines into 2 text files results. Could you help me out?

Answer 1

First, import the data in Spark and add IDs to it. Run these commands in scala console:

val df = spark.read.csv("file.csv")
val df1 = df.withColumn("id",monotonicallyIncreasingId)

Then use this logic to split:

val df2 = df1.filter($"id"%2!==0)
val df2 = df1.filter($"id"%2!==0)