using a bytes field as proxy for arbitrary messages

Question

Hello nano developers,

I'd like to realize the following proto:

message container {
    enum MessageType {
        TYPE_UNKNOWN =  0;
        evt_resultStatus  =  1;
    }
    required MessageType mt = 1;
    optional bytes cmd_evt_transfer = 2;
}

message evt_resultStatus {
    required int32 operationMode = 1;
}

...

The dots denote, there are more messages with (multiple) primitive containing datatypes to come. The enum will grow likewise, just wanted to keep it short.

The container gets generated as:

typedef struct _container { 
    container_MessageType mt; 
    pb_callback_t cmd_evt_transfer; 
} container;

evt_resultStatus is:

typedef struct _evt_resultStatus { 
    int32_t operationMode; 
} evt_resultStatus;

The field cmd_evt_transfer should act as a proxy of subsequent messages like evt_resultStatus holding primitive datatypes. evt_resultStatus shall be encoded into bytes and be placed into the cmd_evt_transfer field. Then the container shall get encoded and the encoding result will be used for subsequent transfers.

The background why to do so, is to shorten the proto definition and avoid the oneof thing. Unfortunately syntax version 3 is not fully supported, so we can not make use of any fields.

The first question is: will this approach be possible?

What I've got so far is the encoding including the callback which seems to behave fine. But on the other side, decoding somehow skips the callback. I've read issues here, that this happened also when using oneof and bytes fields.

Can someone please clarify on how to proceed with this?

Sample code so far I got:

bool encode_msg_test(pb_byte_t* buffer, int32_t sval, size_t* sz, char* err) {
    evt_resultStatus rs = evt_resultStatus_init_zero;
    rs.operationMode = sval;
    pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));

    /*encode container*/
    container msg = container_init_zero;
    msg.mt = container_MessageType_evt_resultStatus;
    msg.cmd_evt_transfer.arg = &rs;
    msg.cmd_evt_transfer.funcs.encode = encode_cb;
    if(! pb_encode(&stream, container_fields, &msg)) {
        const char* local_err = PB_GET_ERROR(&stream);
        sprintf(err, "pb_encode error: %s", local_err);
        return false;
    }

    *sz = stream.bytes_written;
    return true;
}

bool encode_cb(pb_ostream_t *stream, const pb_field_t *field, void * const *arg) {
    evt_resultStatus* rs = (evt_resultStatus*)(*arg);

//with the below in place a stream full error rises
//    if (! pb_encode_tag_for_field(stream, field)) {
//        return false;
//    }

    if(! pb_encode(stream, evt_resultStatus_fields, rs)) {
        return false;
    }

    return true;
}

//buffer holds previously encoded data
bool decode_msg_test(pb_byte_t* buffer, int32_t* sval, size_t msg_len, char* err) {
    container msg = container_init_zero;
    evt_resultStatus res = evt_resultStatus_init_zero;

    msg.cmd_evt_transfer.arg = &res;
    msg.cmd_evt_transfer.funcs.decode = decode_cb;
    pb_istream_t stream = pb_istream_from_buffer(buffer, msg_len);
    if(! pb_decode(&stream, container_fields, &msg)) {
        const char* local_err = PB_GET_ERROR(&stream);
        sprintf(err, "pb_encode error: %s", local_err);
        return false;
    }
    *sval = res.operationMode;
    return true;
}

bool decode_cb(pb_istream_t *istream, const pb_field_t *field, void **arg) {
    evt_resultStatus * rs = (evt_resultStatus*)(*arg);

    if(! pb_decode(istream, evt_resultStatus_fields, rs)) {
        return false;
    }

    return true;
}

I feel, I don't have a proper understanding of the encoding / decoding process.

Is it correct to assume:

• the first call of pb_encode (in encode_msg_test) takes care of the mt field
• the second call of pb_encode (in encode_cb) handles the cmd_evt_transfer field

If I do:

bool encode_cb(pb_ostream_t *stream, const pb_field_t *field, void * const *arg) {
    evt_resultStatus* rs = (evt_resultStatus*)(*arg);

    if (! pb_encode_tag_for_field(stream, field)) {
         return false;
    }

    if(! pb_encode(stream, evt_resultStatus_fields, rs)) {
        return false;
    }

    return true;
}

then I get a stream full error on the call of pb_encode.

Why is that?

Answer 1

Yes, the approach is reasonable. Nanopb callbacks do not care what the actual data read or written by the callback is.

As for why your decode callback is not working, you'll need to post the code you are using for decoding.

(As an aside, Any type does work in nanopb and is covered by this test case. But the type_url included in all Any messages makes them have a quite large overhead.)