How to read data from S3 in a regular inetrval using Spark Scala

Question

I was trying to find application for my need and found one java application which dump data(in csv file) to s3 on daily basis.

This application create folder in S3 based on system date like(MM-DD-YYYY format) and then add files to the folder created.

Now i want to read those files from S3 on regular interval like

val fileFromS3= sc.textFile("s3a://digital/MM-DD-YYYY/abc.csv")

Now the script should replace 'MM-DD-YYYY' with the system date.

Please suggest possible solution or any other way to achieve this.

Answer 1

Inorder to get it done first you need to import calender

First you need some imports:

import java.util.Calendar
import java.text.SimpleDateFormat
import java.util.Date

val now = Calendar.getInstance().getTime()  //you can get the current Time:

val formatter = new SimpleDateFormat("MM-dd-yyyy")

val dateAsString = formatter.format(now)   //use your formatter to get the date as String

Then you can load your resource with the dateAsString value using String interpolation:

val fileFromS3= sc.textFile(s"s3a://digital/${dateAsString}/abc.csv")

Hope this helps!

If you want to know more about Apache Spark Scala, It's highly recommended to go for Spark certification course today.

Thanks!!