I want to read csv file from S3 and load write the same data to cassandra

Question

I want to read the .csv file from S3 and load/write the same data to Cassandra.
FYI: I have Access Key and Secret Key to connect to S3.
Any help?

Answer 1

Hi@Raj,

You need to use the connector that connects your S3 bucket to your database. After that, you can use the commands.

First, read the file from s3 into an RDD in your system.

val rdd = sc.textFile("s3n://mybucket/file_name")

After that, load the red to Cassandra table. Also, check the proper format.

rdd.saveToCassandra()

Comments

@MD, I am running my code on EC2. Using Spark, How do I write it to Cassandra, Please share your thoughts.

---

Hi,

As you said your spark is in EC2 instance. So first read the dataset in a data frame and then use the format of Cassandra to save the dataset in Cassandra table. You can use the below command.

val DF = sqlContext.applySchema(rowRDD, schema)
DF.write.format("org.apache.spark.sql.cassandra").options(Map( "table" -> "name", "ID" -> "test")).save()