Do I Need To Store Length Info With Protobuf?

Question

I am storing Protobufs inside of some non-volatile memory in order to save configuration information. I am using NanoPB to decode/encode them. Since I do not know how large the encoded Protobufs are, when I go to grab a serialized proto from memory and decode, I just grab the maximum number of bytes that the enocded protobuf could be, even if it takes up fewer bytes.

My question is: do I have to store some data describing how many bytes the protobuf is so that I can decode properly?

Or is there a way for me to determine that myself if I 0-pad the serialized buffer or use some other method. As it stands, NanoPB is failing to decode the bytes I am giving it, most likely because there is some garbage data after the end of the encoded proto and I have no way of telling how long the serialized data is.

Answer 1

Yes, the end of protobuf messages has to be indicated somehow, as the format is not self-delimiting.

Nanopb provides two built-in methods for this:

1. pb_encode_nullterminated() and pb_decode_nullterminated(), which append a 0x00 byte to the encoded data and detect it when decoding. It is a neat way to do it, but does not have a widespread support in other protobuf libraries. Therefore it is mostly useful when nanopb is used for both encoding and decoding.

2. pb_encode_delimited() and pb_decode_delimited(), which prepend the size before the message. This is supported by several other protobuf libraries also. A downside is that the encoder has to first determine the length of the message before encoding it, which currently gives a small performance penalty in nanopb.

One can also use custom methods, such as storing the message length at a separate memory location.