Writing File into HDFS using spark scala

Question

Hi, I am trying to create a new file in HDFS and write some content into it. Here is my code snippet:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import java.io.PrintWriter;

object App {

def main(args : Array[String]) {

val conf = new Configuration()
val fs= FileSystem.get(conf)
val output = fs.create(new Path("hdfs://quickstart.cloudera:8020/user/cloudera/sample.txt"))
val writer = new PrintWriter(output)
writer.write("Hellow World")
}
}

The file is getting created in the hdfs but the problem is I can't find the content that I was trying to write into the file sample.txt. Please tell me what is the problem.

Answer 1

The reason you are not able to find the content that you are trying to write into the file is because you are not closing the writer object. So, just add the following line in your code and you will be able to write the data as well:

writer.close()

Hope it helps!

If you want to know more about Apache Spark Scala, It's highly recommended to go for Apache Spark certification course today.

Thanks!!