How does one properly close a captain proto struct to avoid memory leaks?

Question

I think I have memory issues and I want to close the file I've read using pycapnp. How is the best way to do this? Is doing:

    def __getitem__(self, idx: int) -> DagNode:
        # gets the file idx for the value we want
        file_idx = self.get_file_index(idx)
        file_name = self.list_files_current_split[file_idx]
        f = open(file_name)
        capnp_file = dag_api_capnp.Dag.read_packed(f, traversal_limit_in_words=2 ** 64 - 1)
        # do stuff with it
        node = DagNode(capnp_file.field1)
        # close both files
        capnp_file.finish()
        f.close()
        return node

in my scenario I have a data set saved in the captian proto format and I want to close the captain proto file after I've extracted the information I need from it (so to avoid memory issues and not waste space!). What is the proper way to do this, the

.finish()

method guaranteed to do this? I don't have to call the garbage collector...right?

Thanks in advance!

reference:

• http://capnproto.github.io/pycapnp/capnp.html#capnp._DynamicResizableListBuilder.finish
• https://github.com/capnproto/pycapnp/issues/245
• https://github.com/capnproto/pycapnp/issues/251

Answer 1

have you tried python memory-mapped file support? mmap It works much like os handling huge swap files. It's extremely fast and efficient. use readBytes to collect chunks of data (guess it's traversal_limit_in_words ?)

In short, all data needed is available at right time in your program memory space and further loading of data is taken care by mmap.

hope this helps!