Append to golang gob in a file on disk

Question

I'm trying to save gob-encoded data in a file on disk as a simple datastore. However, when I open it next time the gob encoder just ignores whatever data is already in the file, and starts over sending definitions of already sent formats before sending data. Seeing that gob.Encoder takes an io.Writer, rather than an io.ReadWriter, this makes sense. The encoder has no idea what's in the file, because it cannot read from it.

This is, however, quite inefficient. It's also unnecessarily hard to parse, since you have to reset the decoder each time the encoder restarted when writing to the file, since the gob type id's might've changed.

How can I continue to write to the end of a file containing gob data? Do I have to create a new file, moving all data over so the gob encoder knows what types it already sent, or is there a way for me to tell the encoder what types it should know about already, or some other way? I need it to work across software restarts.

Answer 1

It is not possible to reconstruct an Encoder with the state that some other Encoder had when writing a stream of gob values. If you cannot use the original Encoder when appending values, then you have a couple of options:

• Move the data over as you suggest.
• Use a framing mechanism to store multiple gob streams to the single file. For example, you can prefix each gob stream with the length of that stream.