how to run spark job from EC2 to EMR

Question

I have created a django application in EC2 instance and for analytics I used pyspark in django application. Now for spark jobs i want to utilize EMR. How it can be done?

Answer 1

Hi,

You can follow the below-given steps to run your spark code in the Amazon EMR cluster.

• Upload files on Amazon S3.

• Open the Amazon EMR console.

• Choose Create cluster.

• On the General Configuration section, enter the cluster name, choose the S3 bucket you created (the logs will be stored in this bucket), and check Step execution.

• On the Add steps section, select Spark application, click Configure, and fill the popup like this.

• On the Software Configuration section, use the default release.

• On the Hardware configuration section, choose the instance type and the number of instances.

• On the Security and access section, use the default values.

• Click on Create cluster

• Now go back to the S3 console and you will see the output directory in which the result has been stored, you can click on it and download its contents