c#: strange shift in data buffer during communication with device

Question

In my app I have to receive and process some data from device, connected through COM port. I do it partially. In that particular device first two bytes are the length of the packet (minus 2 since it doesn't take into account these very two bytes; so it is length of the rest of the packet after all). Then, since I know that device tends to send me its data slowly, I read rest of the packet in the loop, until all data has been read. But right here I encountered strange problem. Let's assume the entire packet (including these first two bytes with length) looks like this: ['a', 'b', 'c', 'd', 'e']. When I read first two bytes ('a' and 'b'), I'd expect rest of the packet to look like this: ['c', 'd', 'e']. But instead, it looks like this: ['b', 'c', 'd', 'e']. How come second byte of the response is still in the read buffer? And why just the second one, without the previous one?

The code below shows how do I handle the communication process:

//The data array is some array with output data
//The size array is two-byte array to store frame-length bytes
//The results array is for device's response
//The part array is for part of the response that's currently in read buffer
port.Write(data, 0, data.Length);
//Receiving device's response (if there's any)
try
{
    port.Read(size, 0, 2); //Read first two bytes (packet's length) of the response
    //We'll store entire response in results array. We get its size from first two bytes of response
   //(+2 for these very bytes since they're not counted in the device's data frame)
    results = new byte[(size[0] | ((int)size[1] << 8)) + 2];
    results[0] = size[0]; results[1] = size[1]; //We'll need packet size for checksum count
    //Time to read rest of the response
    for(offset = 2; offset < results.Length && port.BytesToRead > 0; offset += part.Length)
    {
        System.Threading.Thread.Sleep(5); //Device's quite slow, isn't it
        try
        {
            part = new byte[port.BytesToRead];
            port.Read(part, 0, part.Length); //Here's where old data is being read
        }
        catch(System.TimeoutException)
        {
                //Handle it somehow
        }
        Buffer.BlockCopy(part, 0, results, offset, part.Length);
    }
    if(offset < results.Length) //Something went wrong during receiving response
        throw new Exception();
}
catch(Exception)
{
    //Handle it somehow
}

Answer 1

You are making a traditional mistake, you cannot ignore the return value of Read(). It tells you how many bytes were actually received. It will be at least 1, not more than count. However many are present in the receive buffer, BytesToRead tells you. Simply keep calling Read() until your happy:

int cnt = 0;
while (cnt < 2) cnt += port.Read(size, cnt, 2 - cnt);

Just use the same code in the 2nd part of your code so you don't burn 100% core without the Sleep() call. Do keep in mind that TimeoutException is just as likely when you read the size, more likely actually. It is a fatal exception if it is thrown when cnt > 0, you can't resynchronize anymore.

Answer 2

Well, strange enough, but when I read first two bytes separatedly:

port.Read(size, 0, 1); //Read first two bytes (packet's length) of the response
port.Read(size, 1, 1); //Second time, lol

Everything works just fine, no matter what kind of data pack do I receive from device.

Answer 3

The documntation for SerialPort contains the following text:

Because the SerialPort class buffers data, and the stream contained in the BaseStream property does not, the two might conflict about how many bytes are available to read. The BytesToRead property can indicate that there are bytes to read, but these bytes might not be accessible to the stream contained in the BaseStream property because they have been buffered to the SerialPort class.

Could this explain why BytesToRead is giving you confusing values?

Personally, I always use the DataReceived event and in my event handler, I use ReadExisting() to read all immediately available data and add it to my own buffer. I don't attempt to impose any meaning on the data stream at that level, I just buffer it; instead I'll typically write a little state machine that takes characters out of the buffer one at a time and parses the data into whatever format is required.

Alternatively, you could use the ReactiveExtensions to produce an observable sequence of received characters and then layer observers on top of that. I do it with a couple of extension methods like this:

public static class SerialObservableExtensions
    {
    static readonly Logger log = LogManager.GetCurrentClassLogger();

    /// <summary>
    ///     Captures the <see cref="System.IO.Ports.SerialPort.DataReceived" /> event of a serial port and returns an
    ///     observable sequence of the events.
    /// </summary>
    /// <param name="port">The serial port that will act as the event source.</param>
    /// <returns><see cref="IObservable{Char}" /> - an observable sequence of events.</returns>
    public static IObservable<EventPattern<SerialDataReceivedEventArgs>> ObservableDataReceivedEvents(
        this ISerialPort port)
        {
        var portEvents = Observable.FromEventPattern<SerialDataReceivedEventHandler, SerialDataReceivedEventArgs>(
            handler =>
                {
                log.Debug("Event: SerialDataReceived");
                return handler.Invoke;
                },
            handler =>
                {
                // We must discard stale data when subscribing or it will pollute the first element of the sequence.
                port.DiscardInBuffer();
                port.DataReceived += handler;
                log.Debug("Listening to DataReceived event");
                },
            handler =>
                {
                port.DataReceived -= handler;
                log.Debug("Stopped listening to DataReceived event");
                });
        return portEvents;
        }

    /// <summary>
    ///     Gets an observable sequence of all the characters received by a serial port.
    /// </summary>
    /// <param name="port">The port that is to be the data source.</param>
    /// <returns><see cref="IObservable{char}" /> - an observable sequence of characters.</returns>
    public static IObservable<char> ReceivedCharacters(this ISerialPort port)
        {
        var observableEvents = port.ObservableDataReceivedEvents();
        var observableCharacterSequence = from args in observableEvents
                                          where args.EventArgs.EventType == SerialData.Chars
                                          from character in port.ReadExisting()
                                          select character;
        return observableCharacterSequence;
        }
}

The ISerialPort interface is just a header interface that I extracted from the SerialPort class, which makes it easier for me to mock it when I'm unit testing.