java lang ClassNotFoundException in Spark on Amazon EMR

Question

Hi Guys,

When I use custom JAR files in a spark-submit or PySpark job on Amazon EMR, I got this below error. 

java.lang.ClassNotFoundException error.

How can I solve this problem?

Answer 1

Hi@akhtar,

In /etc/spark/conf/spark-defaults.conf, append the path of your custom JAR to the class names. 

Open vim /etc/spark/conf/spark-defaults.conf and add jars.

spark.driver.extraClassPath :/home/hadoop/extrajars/*
spark.executor.extraClassPath :/home/hadoop/extrajars/*

Hope this will work.