In my Blazor server app I read a file on the server line by line but it takes too long. How can I shorten the time?

Question

In my Blazor server app I want to show the content of a server side file in my razor page. Th file is 5 MB and has appr. 100000 lines. (My app and the file are on the same server, Win2019) I read the file on the server line by line with "File.ReadLines". Reading of the file takes over 20 seconds.I don't know if that is normal for such a file size or amount of lines.

I wanted to know whether I can shorten the reading process, or even if there is a better way to show the content of the file (read only) on my razor page. The file is a dynamic file, that means its content is changing.

My razor code

<table>
    @if(TagService.Error_log_loaded == true)
    {
    @for (int f = 0; f < TagService.Error_log.Count; f++)
    {
        <tr>
            <td class=".log_file"><pre class="preformatted">@TagService.Error_log[f]</pre></td>
            </tr>
        }            
    }       
</table>

My Blazor code:

public async Task Read_Error_Log()
    {
        try
        {
            TagService.Error_log.Clear();
            Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
            var lines = File.ReadLines(CommonClass.error_path, Encoding.GetEncoding("Windows-1254"));
            var line_trimmed = "";

            foreach (var line in lines)
            {
              TagService.Error_log.Add(line);
            }
            TagService.Error_log_loaded  = true;
            StateHasChanged();
        }
        catch (Exception e)
        {
            using (StreamWriter sw = File.AppendText(CommonClass.error_path))
            {
                sw.WriteLine("Read_Error_Log() Error " + e + " " + Convert.ToString(DateTime.Now));
            }
        }            
    }    

Answer 1

Not sure how you are doing this but here's a demo which uses async line by line reading to load the data into a List<string> and the Virtualize component to control the display rows retrieved. The data file I ran it on contains 1,000,000 lines. It loads the display immediately with the first set of data, while the service gets the rest of the data in the background.

FileService registered as a scoped service.

using Microsoft.AspNetCore.Components.Web.Virtualization;

namespace SO76986556.Data
{
    public class FileService
    {
        private string _fileName = $"{System.IO.Directory.GetCurrentDirectory()}{@"\wwwroot\data.txt"}";
        private DateTime _lastWriteTime = DateTime.MinValue;

        private List<string>? _lines;

        public Task FileLoadTask = Task.CompletedTask;

        public async Task ReadFileAsync()
        {
            // Refresh if the data is older than 2 mins
            // probably check the fiel write time or some other method
            // to determine if the log has updated
            if (_lastWriteTime > DateTime.Now.AddMinutes(-2))
            {
                return;
            }

            // Read the data asynchronously into the list
            _lines = new List<string>();
            using (StreamReader reader = new StreamReader(_fileName))
            {
                string? line;

                while ((line = await reader.ReadLineAsync()) != null)
                {
                    _lines.Add(line);
                }
            }
        }

        // The get method for Virtualize
        public async ValueTask<ItemsProviderResult<string>> GetItemsAsync(ItemsProviderRequest request)
        {
            // if there's no lines the we need to load the data so start, assign the load task 
            // amd wait for a while for the first few hundred lines to be loaded so we have a valid display
            if (_lines == null)
            {
                FileLoadTask = ReadFileAsync();
                await Task.Delay(250);
            }
            if (_lines == null)
                _lines = new List<string>();

            var lines = _lines.Skip(request.StartIndex).Take(request.Count);

            return new ItemsProviderResult<string>(lines, _lines.Count());
        }
    }
}

Registered Services:

builder.Services.AddScoped<FileService>();
builder.Services.AddHttpContextAccessor();

And the demo page.

@page "/fetchdata"
@inject FileService service;
@inject IHttpContextAccessor HttpContextAccessor

@using System.Linq.Expressions;
@using SO76986556.Data;
@using System.Diagnostics;

@if (_isServerRender)
{
    <div class="alert alert-warning">Pre-Rendering</div>
}
else
{
    <Virtualize TItem="string" ItemsProvider="service.GetItemsAsync">
        <ItemContent>
            <div>@context</div>
        </ItemContent>
        <Placeholder>
            <div class="alert alert-warning">The List is currently loading</div>
        </Placeholder>
    </Virtualize>
}

@code {
    private Task? loadingTask;
    private bool _isServerRender;

    protected override void OnInitialized()
    {
        // check if the page is being pre-rendered
        _isServerRender = !(HttpContextAccessor.HttpContext is not null && HttpContextAccessor.HttpContext.Response.HasStarted);
    }
}

It will need a little tuning and sorting for your situation.

There's a demo Repo here : https://github.com/ShaunCurtis/SO76986556