Hadoop hdfs How to count number of lines

Question

I have stored a file in my hdfs. The data in the hdfs is distributed and I want to count the number of lines in the file that I stored in the hdfs (the original file). How can I do this?

Answer 1

You can count the number of lines using spark:

myfile = spark.textFile("hdfs://...")
mycount = myfile.count();
mycount.dump();

This will output the number of lines in the file.