java lang IllegalArgumentException Unable to instantiate SparkSession with Hive support because Hive classes are not found

Question

I had tried to use hive tables in spark sql the folowing is the sparksesion instance

spark_Session = SparkSession
.builder()
.appName("Spark_Sql_Project")
.master("local[*]")
.enableHiveSupport()
.config("spark.sql.warehouse-dir","/uer/hive/warehouse")
.getOrCreate()

If I run the code in scala ide I am getting following error

"java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found."

Answer 1

 Try using the below build.sbt, code.scala and command for connecting to hive from spark.

build.sbt

name := "Hive Spark"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.0"

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"

hive_connect.scala

import org.apache.spark.sql.Row

import org.apache.spark.sql.SparkSession

import org.apache.spark.sql.types._

object hive_connect{

def main(args: Array[String]) {

val sparkSession = SparkSession.builder.master("local").appName("demo").enableHiveSupport().getOrCreate()

sparkSession.sqlContext.sql(“use sayni”)

sparkSession.sqlContext.sql(“show tables”).show()

}

}

And your command should look something like this:

spark2-submit --class hive_connect --deploy-mode client /mnt/home/edureka_425640/hive_connect/target/scala-2.11/hive-spark_2.11-1.0.jar