Shelve module: How to instantly write on memory when using 'deque.append()'

30 views Asked by At
import shelve
from collections import deque
with shelve.open('123', writeback=True) as DATA:
    DATA['TEST'] = deque()
    DATA['TEST'].append(1)
    DATA['TEST'].append(2)

This code creates a deque and, after the programm ends, copies it to the file '123.dat' (at least that is what I understand). What I want to do is that after running DATA['TEST'].append(1), the file '123.dat' gets updated, so if you open it you are able to see the deque with the value 1. Then, after running DATA['TEST'].append(2), the file '123.dat' gets updated again with the new deque but just the value 2 get added, i.e., value 1 is not overwritten and only value 2 is added to the file.

The problem is that in the code that I'm working on handles a fairly heavy deque (over 10gb) and runs in one loop. In each iteration an element is added to the deque, but when closing the program it takes a long time to update the file depending on how long it was running. As it is a neural network I need it to train it for several hours, but I don't want to wait so long every time I want to pause the training.

Does anyone know how to update the file at each iteration only adding the new value obtained?

0

There are 0 answers