Why does ReadAsync on a NetworkStream read data twice?

Question

I am currently working an a piece of TcpIp communication in which we have to send/receive JSON strings back and forth. While sending the data I first send the length of the JSON string, (call it the header) and then the JSON string itself (call it the data). The receiving part of course first reads the header (which is a fixed size message containing the length of the JSON string) and then tries to read the data (which is the JSON string). Most of the times it works well but every now and then I observe that the header data is not removed from the NetworkStream after the ReadAsync call and thus ends up being part of the data (i.e. in the JSON string). Deserializing the JSON string then throws an exception.

        public virtual async Task<object> ReceiveRequestAsync(SerializationBinder binder)
        {
            Trace?.In($"binder={binder}");
            var _binder = binder;
            byte[] headerData = new byte[sizeof(int)];
            dynamic message = null;
            string json = "";
            try
            {
                if (IsConnected)
                {
                    // First we read the length of the JSON string from the stream.
                    int bytesRead = 0;
                    while (bytesRead < sizeof(int))
                    {
                        bytesRead += await TcpStream.ReadAsync(headerData, bytesRead, sizeof(int) - bytesRead);
                    }
                    LoggingService.Message($"Header: [{string.Join(", ", headerData)}]");

                    // Now that we now how many bytes there are in the JSON string we can create and array
                    // and read the data.
                    byte[] data = new byte[BitConverter.ToInt32(headerData, 0)];
                    bytesRead = 0;
                    while (bytesRead < data.Length)
                    {
                        bytesRead += await TcpStream.ReadAsync(data, bytesRead, data.Length - bytesRead);
                    }
                    LoggingService.Message($"Data: [{string.Join(", ", data)}]");

                    // Now that we have the full JSON data we can deserialize it into a request message.
                    json = Encoding.UTF8.GetString(data, 0, bytesRead);
                    message = JsonConvert.DeserializeObject<dynamic>(json, new JsonSerializerSettings
                    {
                        TypeNameHandling = TypeNameHandling.Auto,
                        Binder = _binder
                    });
                }
                else
                {
                    LoggingService.Error($"Not connected {ServerInfo.ServerIp}:{ServerInfo.ServerPort}");
                }

            }
            catch (Exception e)
            {
                LoggingService.Error($"Error occured {e.Message}, message contained: {json}");
            }

            Trace?.Out($"binder={binder}", $"{message}");
            return message;
        }

I was not expecting to read the same data twice. I suspect that the filepointer in the stream is not updated directly after the ReadAsync but I don't seem to get my hands on it. I am using .Net Framework 4.8.1 for this.