import shelve
from collections import deque
with shelve.open('123', writeback=True) as DATA:
DATA['TEST'] = deque()
DATA['TEST'].append(1)
DATA['TEST'].append(2)
This code creates a deque and, after the programm ends, copies it to the file '123.dat' (at least that is what I understand). What I want to do is that after running DATA['TEST'].append(1), the file '123.dat' gets updated, so if you open it you are able to see the deque with the value 1. Then, after running DATA['TEST'].append(2), the file '123.dat' gets updated again with the new deque but just the value 2 get added, i.e., value 1 is not overwritten and only value 2 is added to the file.
The problem is that in the code that I'm working on handles a fairly heavy deque (over 10gb) and runs in one loop. In each iteration an element is added to the deque, but when closing the program it takes a long time to update the file depending on how long it was running. As it is a neural network I need it to train it for several hours, but I don't want to wait so long every time I want to pause the training.
Does anyone know how to update the file at each iteration only adding the new value obtained?