FileStreamSink Error while looking for metadata directory java lang IllegalArgumentException java net UnknownHostException hive

Question

Hi,

I tried to read one csv file in spark sql. But I am getting the below error.

val dfs = sqlcontext.read.format("com.databricks.spark.csv").option("header", "true").option("inferschema","true").load("hdfs://hive/bike")
20/02/13 09:38:05 WARN streaming.FileStreamSink: Error while looking for metadata directory.
java.lang.IllegalArgumentException: java.net.UnknownHostException: hive
  at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)
  at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
  at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
  at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:668)
  at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:604)
  at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2598)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2632)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2614)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:547)
  at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:545)
  at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
  at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
  at scala.collection.immutable.List.flatMap(List.scala:355)
  at org.apache.spark.sql.execution.datasources.DataSource.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:545)
  at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:359)
  at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:223)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:211)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
  ... 49 elided
Caused by: java.net.UnknownHostException: hive
  ... 73 more
                               

Answer 1

Hi@akhtar,

You tried to read file from your hdfs cluster. So you have to give the full path of the file along with master ip and port.

$  val dfs = sqlcontext.read.format("com.databricks.spark.csv").option("header", "true").option("inferschema","true").load("hdfs://master ip:port/hive/bike")

Hope it will solve your problem.

Thank You