How to read Avro Partition Data

Question

Hi Guys,

I have Avro partition data. I want to read the files. So that I can perform my operations. How can I do that?

Answer 1

Hi@akhtar,

When we try to retrieve the data from the partition, It just reads the data from the partition folder without scanning entire Avro files.

spark.read
      .format("avro")
      .load("person_partition.avro")
      .where(col("dob_year") === 2010)
      .show()