using callbacks for nested and repeated fields in a protobuf using nanopb in c

Question

*Edit: updated * My message is defined as:

message Repeat {
  int32 inum = 1;
  float fnum = 2;
}
message NotSimpleMessage {
  repeated Repeat repeat = 1;
}

I'm trying to write a decoder and encoder using the callback option. I think my encoding works fine, but my decoder fails. My code is: definitions:

 typedef struct{
        Repeat rep[MAX_NUMBERS];
        int32_t numbers_count;
    }Messer;

typedef struct{
    Mess mess[MAX_NUMBERS];
    int32_t numbers_count;
}MessList;

void mess_add_number(MessList * list, int32_t inum, float fnum)
{
    if (list->numbers_count < MAX_NUMBERS)
    {
        (list->mess[list->numbers_count]).inumber = inum;
        (list->mess[list->numbers_count]).fnumber = fnum;
        list->numbers_count++;
    }
}

    void messer_add_number(Messer * list, int32_t inum, float fnum)
    {
        if (list->numbers_count < MAX_NUMBERS)
        {
            (list->rep[list->numbers_count]).inum = inum;
            (list->rep[list->numbers_count]).fnum = fnum;
            (list->rep[list->numbers_count]).has_inum = true;
            (list->rep[list->numbers_count]).has_fnum = true;
            list->numbers_count++;
        }
    }

encoder/decoder functions:

bool NestedMessage_encode_numbers(pb_ostream_t *ostream, const pb_field_t *field, void * const *arg)
{
    Messer * source = (Messer*)(*arg);
        int i;
        // encode all numbers
        for ( i = 0; i < source->numbers_count; i++)
        {
            if (!pb_encode_tag_for_field(ostream, field))
            {
                const char * error = PB_GET_ERROR(ostream);
                printf("SimpleMessage_encode_numbers error: %s\n", error);
                return false;
            }

            if (!pb_encode_submessage(ostream, Repeat_fields, &(source->rep[i])))
            {
                const char * error = PB_GET_ERROR(ostream);
                printf("SimpleMessage_encode_numbers error: %s\n", error);
                return false;
            }
        }

        return true;
c}
bool NestedMessage_decode_numbers(pb_istream_t *istream, const pb_field_t *field, void **arg)
{
    MessList * dest = (MessList*)(*arg);
    Repeat rep;
    // decode single number
    Mess decmess;
    printf("decoding...\n");
    if (!pb_decode(istream, Repeat_fields ,&rep))
    {
        const char * error = PB_GET_ERROR(istream);
        printf("decode error: %s\n", error);
        return false;
    }

    // add to destination list
    mess_add_number(dest, rep.inum, rep.fnum);
    return true;
}

and the main is:

int main(void) {


    uint8_t buffer[128];
    size_t total_bytes_encoded = 0;

    // encoding
    // prepare the actual "variable" array
    Messer actualData = { 0 };
    messer_add_number(&actualData, 123, 1.2);
    messer_add_number(&actualData, 456, 2.3);
    messer_add_number(&actualData, 789, 3.4);
    printf("Size: %d\n",actualData.numbers_count);
    printf("data to be encoded: %d - %f, %d-%f, %d-%f\n",actualData.rep[0].inum,actualData.rep[0].fnum,
            actualData.rep[1].inum, actualData.rep[1].fnum,
            actualData.rep[2].inum,actualData.rep[2].fnum);
    // prepare the nanopb ENCODING callback
    NotSimpleMessage msg = NotSimpleMessage_init_zero;
    msg.repeat.arg = &actualData;
    msg.repeat.funcs.encode = NestedMessage_encode_numbers;

    // call nanopb
    pb_ostream_t ostream = pb_ostream_from_buffer(buffer, sizeof(buffer));
    if (!pb_encode(&ostream, NotSimpleMessage_fields, &msg))
    {
        const char * error = PB_GET_ERROR(&ostream);
        printf("pb_encode error: %s\n", error);
        return EXIT_FAILURE;
    }

    total_bytes_encoded = ostream.bytes_written;
    printf("Encoded size: %d\n", total_bytes_encoded);


    // decoding

    // empty array for decoding
    Messer decodedData = { 0 };

    // prepare the nanopb DECODING callback
    NotSimpleMessage msgdec = NotSimpleMessage_init_zero;
    msgdec.repeat.arg = &decodedData;
    msgdec.repeat.funcs.decode = NestedMessage_decode_numbers;

    // call nanopb
    pb_istream_t istream = pb_istream_from_buffer(buffer, total_bytes_encoded);
    if (!pb_decode(&istream, NotSimpleMessage_fields, &msgdec))
    {
        const char * error = PB_GET_ERROR(&istream);
        printf("pb_decode error: %s", error);
        return EXIT_FAILURE;
    }

    printf("Bytes decoded: %d\n", total_bytes_encoded - istream.bytes_left);
    printf("decoded data: %d - %f, %d-%f, %d-%f\n",decodedData.rep[0].inum,decodedData.rep[0].fnum,
            decodedData.rep[1].inum, decodedData.rep[1].fnum,
            decodedData.rep[2].inum,decodedData.rep[2].fnum);
}

the output I get is:

Size: 3 data to be encoded: 123 - 1.200000, 456-2.300000, 789-3.400000 Encoded size: 29 Bytes decoded: 1 decoded data: 0 - 0.000000, 0-0.000000, 0-0.000000

print of the encoded buffer:

0a07087b15ffffff9affffff99ffffff993f0a0808ffffffc80315333313400a0808ffffff950615ffffff9affffff995940

I've tried some different structs inside the decoder but it just doesn't work. pretty sure it some dumb small thing I'm missing, but I'm clueless about it.

Answer 1

Ah, there is a small gotcha in encoding/decoding submessages in callbacks.

When decoding, pb_decode() works fine because the submessage tag and length has already been parsed by nanopb. However, when encoding, the length of the message needs to be calculated and encoded separately. So instead of pb_encode(), you need to use pb_encode_submessage() here:

        if (!pb_encode_submessage(ostream, Repeat_fields, &(source->rep[i])))
        {
            const char * error = PB_GET_ERROR(ostream);
            printf("SimpleMessage_encode_numbers error: %s\n", error);
            return false;
        }

(For reference, here is a relevant part of an example.)

--

Regarding your update, this hex text:

0a07087b15ffffff9affffff99ffffff993f0a0808ffffffc80315333313400a0808ffffff950615ffffff9affffff995940

is somewhat corrupted, because your printing function seems to print "ffffff9a" instead of just "9a". Probably a signed to unsigned cast behaving unexpectedly. But that can be fixed with a simple search & replace, which gives:

0a07087b159a99993f0a0808c80315333313400a08089506159a995940

Decoding this with protoc:

echo 0a07087b159a99993f0a0808c80315333313400a08089506159a995940 | xxd -r -p | protoc --decode=NotSimpleMessage test.proto

Gives:

repeat {
  inum: 123
  fnum: 1.2
}
repeat {
  inum: 456
  fnum: 2.3
}
repeat {
  inum: 789
  fnum: 3.4
}

So seems your encoding is now working correctly.

Not sure what is causing the decode end so early (only 1 byte read) without error message. Maybe try stepping through it with a debugger and see what is going on. One reason might be if the data in the buffer would somehow get corrupted before the decode call, but can't see why that would happen.

Answer 2

In these definitions what structure is for Repeat and Mess? It´s important to try to compile.