How can I remove duplicate dict in list in Python

Question

I have a list of dicts, and I'd like to remove the dicts with identical key and value pairs.

For this list: [{'a': 123}, {'b': 123}, {'a': 123}]

I'd like to return this: [{'a': 123}, {'b': 123}]

Answer 1

You can preserve the Order, then you can do:

from collections import OrderedDict
print OrderedDict((frozenset(item.items()),item) for item in data).values()
# [{'a': 123, 'b': 1234}, {'a': 3222, 'b': 1234}]

If the order doesn't matter, then you can do:

print {frozenset(item.items()):item for item in data}.values()
# [{'a': 3222, 'b': 1234}, {'a': 123, 'b': 1234}]

Answer 2

[dict(t) for t in {tuple(d.items()) for d in l}]

or

a = [{'a': 123}, {'b': 123}, {'a': 123}]
b = []
for i in range(0, len(a)):
    if a[i] not in a[i+1:]:
        b.append(a[i])

Comments

This is so much easier than @Nymeria's answer. The logic is clear and understandable. Thanks!

Answer 3

use list comprehension. much simpler:

a = [{'a': 123}, {'b': 123}, {'a': 123}]
b = []
[b.append(item) for item in a if item not in b]
b