How to perform a word count on a dataframe column

Question

I have created a dataframe of two columns id and text, I want to perform a wordcount on the text column of the dataframe. What code can I use to do this using PySpark?

Answer 1

You can use the below code to do this:

rdd=sc.textfile(inputfile)

wc=rdd.flatMap(lambda line: line.split()).map(lambda w : (w,1)).reduceByKey(lambda a,b:a+b).map(lambda (a,b) : (b,a)).sortByKey(ascending=True)

output=sc.collect()


for (count, word) in output:

       print("%s %i",(word, count))