Spark to check if a particular string exists in a file

Question

Please provide me the spark code to check if a particular word exists in a file or not.

Answer 1

You can use this:

lines = sc.textFile(“hdfs://path/to/file/filename.txt”);

def isFound(line):

if line.find(“mykeyword”) > -1:

return 1;

return 0;

foundBits = lines.map(isFound);

sum = foundBits.reduce(sum);

if sum > 0:

print “Exists”;

else:

print “Does not exist”;