Python - Load multiple Pickle objects into a single dictionary

Question

So my problem is this... I have multiple Pickle object files (which are Pickled Dictionaries) and I want to load them all, but essentially merge each dictionary into a single larger dictionary.

E.g.

I have pickle_file1 and pickle_file2 both contain dictionaries. I would like the contents of pickle_file1 and pickle_file2 loaded into my_dict_final.

EDIT As per request here is what i have so far:

for pkl_file in pkl_file_list:
    pickle_in = open(pkl_file,'rb')
    my_dict = pickle.load(pickle_in)
    pickle_in.close()

In essence, it works, but just overwrites the contents of my_dict rather than append each pickle object.

Thanks in advance for the help.

Answer 1

my_dict_final = {}  # Create an empty dictionary
with open('pickle_file1', 'rb') as f:
    my_dict_final.update(pickle.load(f))   # Update contents of file1 to the dictionary
with open('pickle_file2', 'rb') as f:
    my_dict_final.update(pickle.load(f))   # Update contents of file2 to the dictionary
print my_dict_final

Answer 2

You can use the dict.update function.

pickle_dict1 = pickle.load(picke_file1)
pickle_dict2 = pickle.load(picke_file2)
my_dict_final = pickle_dict1
my_dict_final.update(pickle_dict2)

Python Standard Library Docs

Answer 3

@Nunchux, @Vikas Ojha If the dictionaries happen to have common keys, the update method will, unfortunately, overwrite the values for those common keys. Example:

>>> dict1 = {'a': 4, 'b': 3, 'c': 0, 'd': 4}
>>> dict2 = {'a': 1, 'b': 8, 'c': 5}

>>> All_dict = {}                   
>>> All_dict.update(dict1)          
>>> All_dict.update(dict2)          

>>> All_dict                        
{'a': 1, 'b': 8, 'c': 5, 'd': 4}

If you'd like to avoid this and keep adding the counts of common keys, one option is to use the following strategy. Applied to your example, here is a minimal working example:

import os 
import pickle
from collections import Counter 

dict1 = {'a': 4, 'b': 3, 'c': 0, 'd': 4}
dict2 = {'a': 1, 'b': 8, 'c': 5}

# just creating two pickle files: 
pickle_out = open("dict1.pickle", "wb") 
pickle.dump(dict1, pickle_out) 
pickle_out.close() 

pickle_out = open("dict2.pickle", "wb")  
pickle.dump(dict2, pickle_out) 
pickle_out.close()  

# Here comes: 
pkl_file_list = ["dict1.pickle", "dict2.pickle"]

All_dict = Counter({})  
for pkl_file in pkl_file_list:  
    if os.path.exists(pkl_file):  
        pickle_in = open(pkl_file, "rb")  
        dict_i = pickle.load(pickle_in)  
        All_dict = All_dict + Counter(dict_i)  

print (dict(All_dict))

This will happily give you:

{'a': 5, 'b': 11, 'd': 4, 'c': 5}