Grep across multiple files in Hadoop

Question

I need to find which of ~100 files in my Hadoop filesystem contain a certain string.

I can see the files I wish to search like this:

bash-3.00$ hadoop fs -ls /apps/edureka/community/real-time

..which returns several entries like this:

-rw-r--r--   3 community edureka 1073741824 2019-07-01 22:50 /apps/edureka/community/HADOOP_consolidated_RT_v1x0_20120716_aa
-rw-r--r--   3 community edureka 1073741824 2019-07-01 22:50 /apps/edureka/community/HADOOP_consolidated_RT_v1x0_20120716_ab

Answer 1

You are looking to applying to grep command on hdfs folder

hdfs dfs -cat /user/coupons/input/201807160000/* | grep -c null

here cat recursively goes through all files in the folder and I have applied grep to find count.

Comments

But it doesn't print the name of the file where it found the string.. so most of the times not useful..

is it possible to print the file name?

---

Hi,

Yes, you can't list down the file names using the cat command. This command is used to read the data from a file. You can use the ls command instead of the cat command.

$ hdfs dfs -ls /path