How do I use the PipeReader for reading a JSON flatfile?

Question

I am trying to learn this new system.io.pipelines, and the new webapi strategy for deserializing json...

I wrote my own JsonConverter, but I can't figure out the correct way to initialize a Utf9JsonReader from a json flat file fixture.

here is the test:

    [Fact]
    public void WhenGivenJsonObjectThenEntityDTOReturned() {
                    
        using(var stream = new FileStream("Fixtures/BookStoreJson.json", FileMode.Open))
        {
            var pipe = PipeReader.Create(stream);
            ReadResult bytes;
            pipe.TryRead(out bytes);
            var reader = new Utf8JsonReader(bytes.Buffer);
            var target = new EntityDTOConverter();
            reader.Read();
            var actual = target.Read(ref reader, typeof(EntityDTO), new JsonSerializerOptions());
            
            Assert.True(actual.Props.ContainsKey("name"));
        }
    

    }

When I debug this, the bytes.buffer is set to 0 bytes, even though the BookStoreJson.json file contains the following:

{
    "name": "Tattered Cover",
    "store":{
       "book":[
          {
             "category":"reference",
             "author":"Nigel Rees",
             "title":"Sayings of the Century",
             "price":8.95
          },
          {
             "category":"fiction",
             "author":"Evelyn Waugh",
             "title":"Sword of Honour",
             "price":12.99
          },
          {
             "category":"fiction",
             "author":"J. R. R. Tolkien",
             "title":"The Lord of the Rings",
             "isbn":"0-395-19395-8",
             "price":22.99
          }
       ],
       "bicycle":{
          "color":"red",
          "price":19.95
       }
    }
 }

Answer 1

Apologies,I didn't realise about async process, I was testing from console app. besides not sure how much this answer is going to help you out, basically you can run async tasks on sync by accessing results. also there is a limitation on buffer size, if the json file size is higher you may need to create custom pool and use AdvanceTo to options to read to end of buffer to get the stream.

using System;
using System.Buffers;
using System.IO;
using System.IO.Pipelines;
using System.Text;
using System.Text.Json;
using System.Threading.Tasks;
using Xunit;
 

            public class Person
            {
                public string Email { get; set; }
                public bool Active { get; set; }
                public string CreatedDate { get; set; }
                public string[] Roles { get; set; }
            }
    
            [Fact]
            public void WhenGivenJsonObjectThenEntityDTOReturned()
            {
                //{
                //    "Email": "[email protected]",
                //    "Active": true,
                //    "CreatedDate": "2013-01-20T00: 00: 00Z",
                //    "Roles": [
                //    "User",
                //    "Admin"
                //    ]
                //}
    
                using (var stream = new FileStream(@"c:\temp\json.json", FileMode.Open))
                {
                    var reader = PipeReader.Create(stream, 
                        new StreamPipeReaderOptions(bufferSize:4096 *2));
    
                    ValueTask<ReadResult> readResult = reader.ReadAsync();
                    ReadOnlySequence<byte> buffer = readResult.Result.Buffer;
                    Assert.True(readResult.IsCompleted);
                    var jsonStreamReader = new Utf8JsonReader(buffer);
                    var expectedJson = JsonSerializer
                        .Deserialize<Person>(Encoding.UTF8.GetString(buffer.ToArray()));
                    Assert.Equal("[email protected]", expectedJson.Email);
                }
            }

Answer 2

You can read a file by PipeReader in this way -

            using (var stream = new FileStream(@"test.json", FileMode.Open))
            {
                var pipeReader = System.IO.Pipelines.PipeReader.Create(stream);

                while (true)
                {
                    var pipeReadResult = await pipeReader.ReadAsync();
                    var buffer = pipeReadResult.Buffer;

                    try
                    {
                        //process data in buffer
                        Console.WriteLine(buffer.Length.ToString());

                        if (pipeReadResult.IsCompleted)
                        {
                            break;
                        }
                    }
                    finally
                    {
                        pipeReader.AdvanceTo(buffer.End);
                    }
                }
            }

If you set the buffer size for PipeReader larger than the size of the file to read, you do not need to loop, but that beats the purpose of using PipeReader - to process the data on the fly, piece by piece.

I do not think PipeReader is suitable to your case. Utf8JsonReader can directly work with a file stream. I think this is what you need -

https://learn.microsoft.com/en-us/dotnet/standard/serialization/write-custom-serializer-deserializer#read-from-a-stream-using-utf8jsonreader