Recursive data structure unmarshalling gives error "cannot parse invalid wire-format data" in Go Lang Protobuf

Question

OS and protobuf version

go1.18.1 linux/amd64, github.com/golang/protobuf v1.5.2

Introduction

I am trying to use recursive proto definitions.

.proto file

message AsyncConsensus {
  int32 sender = 1;
  int32 receiver = 2;
  string unique_id = 3; // to specify the fall back block id to which the vote asyn is for
  int32 type = 4; // 1-propose, 2-vote, 3-timeout, 4-propose-async, 5-vote-async, 6-timeout-internal, 7-consensus-external-request, 8-consensus-external-response, 9-fallback-complete
  string note = 5;
  int32 v = 6 ; // view number
  int32 r = 7;// round number
  message Block {
    string id = 1;
    int32 v = 2 ; // view number
    int32 r = 3;// round number
    Block parent = 4;
    repeated int32 commands = 5;
    int32 level = 6; // for the fallback mode
  }
  Block blockHigh = 8;
  Block blockNew = 9;
  Block blockCommit = 10;
}

The following is how I Marshal and Un-Marshal

func (t *AsyncConsensus) Marshal(wire io.Writer) error {
    data, err := proto.Marshal(t)
    if err != nil {
        return err
    }
    lengthWritten := len(data)
    var b [8]byte
    bs := b[:8]
    binary.LittleEndian.PutUint64(bs, uint64(lengthWritten))
    _, err = wire.Write(bs)
    if err != nil {
        return err
    }
    _, err = wire.Write(data)
    if err != nil {
        return err
    }
    return nil
}

func (t *AsyncConsensus) Unmarshal(wire io.Reader) error {

    var b [8]byte
    bs := b[:8]
    _, err := io.ReadFull(wire, bs)
    if err != nil {
        return err
    }
    numBytes := binary.LittleEndian.Uint64(bs)
    data := make([]byte, numBytes)
    length, err := io.ReadFull(wire, data)
    if err != nil {
        return err
    }
    err = proto.Unmarshal(data[:length], t)
    if err != nil {
        return err
    }
    return nil
}

func (t *AsyncConsensus) New() Serializable {
    return new(AsyncConsensus)
}

My expected outcome

When marshaled and sent to the same process via TCP, it should correctly unmarshal and produce correct data structures.

Resulting error

error "cannot parse invalid wire-format data"

Additional information

I tried with non-recursive .proto definitions, and never had this issue before.

Answer 1

This is not a bug with Protobuf, but its a mater of how you marshal and unmarshal protobuf structs.

As a concrete guideline, never concurrently marshal and unmarshal protobuf structs as it my lead to race conditions.

In the specific example you have provided, I see recursive data structs, so even if you use a separate struct for each invocation of marshal and unmarshal, it's likely that the pointers in the parent can lead to shared pointers.

Use a deep copy technique to remove any dependency so that you do not run in to race conditions.

func CloneMyStruct(orig *proto.AsyncConsensus_Block) (*proto.AsyncConsensus_Block, error) {
    origJSON, err := json.Marshal(orig)
    if err != nil {
        return nil, err
    }

    clone := proto.AsyncConsensus_Block{}
    if err = json.Unmarshal(origJSON, &clone); err != nil {
        return nil, err
    }

    return &clone, nil
}

Answer 2

The stupidest error I can think about is that the wire.Write(bs) don’t write as many bytes as the io.ReadFull(wire, bs) read - so I’d just make sure that their return value is actually 8 in both cases.

Then I don’t know the golang/protobuf very well, but I guess it should be able to do this. Shouldn’t you create the go-code and then call out to it? I’m not sure how to call it.

If you think that it’s actually a problem in the protobuf implementation, there are some online protobuf-decoders, which can help. But they sometimes interpret the stream incorrectly, which could be the case here with a recursive pattern, so you have to be careful. But at least they helped me to debug the dedis/protobuf package more than once.

As a last resort you can make a minimal example with recursive data, check if it works, and then slowly add fields until it breaks…