Hadoop Spark How to iterate hdfs directories

Question

How to iterate hdfs directories using spark?

Answer 1

You can use org.apache.hadoop.fs.FileSystem.

Using SPARK:

FileSystem.get(sc.hadoopConfiguration()).listFiles(..., true)

Answer 2

Using PySpark 

hadoop = sc._jvm.org.apache.hadoop

fs = hadoop.fs.FileSystem
conf = hadoop.conf.Configuration() 
path = hadoop.fs.Path('/hivewarehouse/disc_mrt.db/unified_fact/')

for f in fs.get(conf).listStatus(path):
    print f.getPath()

Answer 3

import  org.apache.hadoop.fs.{FileSystem,Path}

FileSystem.get( sc.hadoopConfiguration ).listStatus( new Path("hdfs:///tmp")).foreach( x => println(x.getPath ))