How to Remove duplicates and value with None from the list

Question

Hello I have bulk of list of dict and would like to remove duplicate dict and value with None. Within the data structure, bunch dict are duplicate. I want to keep pair of devices.

BulkData

[
    {"name": "PAIR-05|PAIR-06", "device": "oob-01"},
    {"name": "PAIR-05|PAIR-06", "device": "oob-01"},
    {"name": "PAIR-01|PAIR-02", "device": "oob-03"},
    {"name": "PAIR-01|PAIR-02", "device": "oob-03"},
    {"name": None, "device": "oob-01"},
    {"name": None, "device": "oob-01"},
    {"name": None, "device": "oob-01"},
    {"name": None, "device": "oob-01"},
]

Non-working code:

for key, value in bulkpairdata.items():
  t = []
  if bulkpairdata.get("name") != None:
    if bulkpairdata.get("name") not in t:
      result.append(bulkpairdata)
      t.append(bulkpairdata.get("name"))
    else:
      bulkpairdata.remove(bulkpairdata.get("name"))

Result:

[
    {"name": "PAIR-05|PAIR-06", "device": "oob-01"},
    {"name": "PAIR-01|PAIR-02", "device": "oob-03"},
]

Answer 1

You can try something like this:

bulk_data = [
    {"name": "PAIR-05|PAIR-06", "device": "oob-01"},
    {"name": "PAIR-05|PAIR-06", "device": "oob-01"},
    {"name": "PAIR-01|PAIR-02", "device": "oob-03"},
    {"name": "PAIR-01|PAIR-02", "device": "oob-03"},
    {"name": None, "device": "oob-01"},
    {"name": None, "device": "oob-01"},
    {"name": None, "device": "oob-01"},
    {"name": None, "device": "oob-01"},
]

result = []

for item in bulk_data:
    any_none = any([True if value is None else False for key, value in item.items()])
    if not any_none and item not in result:
        result.append(item)

which yields following result:

[{'name': 'PAIR-05|PAIR-06', 'device': 'oob-01'}, {'name': 'PAIR-01|PAIR-02', 'device': 'oob-03'}]

Answer 2

Try this:

bulkdata = [
    {"name": "PAIR-05|PAIR-06", "device": "oob-01"},
    {"name": "PAIR-05|PAIR-06", "device": "oob-01"},
    {"name": "PAIR-01|PAIR-02", "device": "oob-03"},
    {"name": "PAIR-01|PAIR-02", "device": "oob-03"},
    {"name": None, "device": "oob-01"},
    {"name": None, "device": "oob-01"},
    {"name": None, "device": "oob-01"},
    {"name": None, "device": "oob-01"},
]


[dict(t) for t in {tuple(d.items()) for d in list(filter(lambda x: None not in x.values(), bulkdata))}]

This prints:

[{'name': 'PAIR-05|PAIR-06', 'device': 'oob-01'},
 {'name': 'PAIR-01|PAIR-02', 'device': 'oob-03'}]

The function works by first filtering your list to only keep dicts, where all values are not None. Afterwards the duplicate dicts are removed. I used the function as described in the link below.

For reference: How to remove duplicate dicts: https://stackoverflow.com/a/9427216/5240684

How the filter function works: https://www.programiz.com/python-programming/methods/built-in/filter

Answer 3

bulkdata = [
    {"name": "PAIR-05|PAIR-06", "device": "oob-01"},
    {"name": "PAIR-05|PAIR-06", "device": "oob-01"},
    {"name": "PAIR-01|PAIR-02", "device": "oob-03"},
    {"name": "PAIR-01|PAIR-02", "device": "oob-03"},
    {"name": None, "device": "oob-01"},
    {"name": None, "device": "oob-01"},
    {"name": None, "device": "oob-01"},
    {"name": None, "device": "oob-01"},
    ]
    
result = []

for entry in bulkdata:
    if None not in entry.values() and entry not in result:
        result.append(entry)

print(result)

result:

[{'name': 'PAIR-05|PAIR-06', 'device': 'oob-01'}, {'name': 'PAIR-01|PAIR-02', 'device': 'oob-03'}]