How to remove duplicate records from Hive table

Question

Hi,

What is the query to remove duplicate records from the hive table and save only the unique records in the hive table?

Answer 1

A record is duplicate if there are occurrences of the same entire record multiple times. We can use distinct to view unique records.

select distinct * from <table_name>;

and then use this:

insert overwrite table duplicate_test select distinct * from duplicate_test;