How to read a dataframe based on an avro schema

Question

I am unable to import from_avro in Pyspark. Trying to run a spark-submit job by invoking the external package for Avro. 

Eg: spark-submit --packages org.apache.spark:spark-avro_2.12:3.0.1 test1.py

My test1.py file contains the import statement

from pyspark.sql.avro.functions import from_avro, to_avro
Getting ImportError: NO module names avro.function

Please help!!! Need to import from_avro using python code

Answer 1

Hi,

I am able to understand your requirement. But your error says you don't have the avro.function package in your system. First, check all the available packages in your Spark.

Comments

How can I read a dataframe using avro schema?

---

Hi@ana,

While working with spark-shell, you can also use --packages to add spark-avro_2.12 and its dependencies directly.

$ ./bin/spark-shell --packages org.apache.spark:spark-avro_2.12:2.4.4

we should use DataSource format as “avro” or “org.apache.spark.sql.avro” and load() is used to read the Avro file.

$ val personDF= spark.read.format("avro").load("person.avro")

---

Thank you @ MD for providing the package details.