Slow loading of .CSV files using EPPLUS

Question

I have loads of .csv files I need to convert to .xslx after applying some formatting.

A file containing approx 20 000 rows and 7 columns takes 12 minutes to convert. If the file contains more than 100 000 it runs for > 1 hour.

This is unfortunately not acceptable for me.

Code snippet:

        var format = new ExcelTextFormat();
        format.Delimiter = ';';
        format.Encoding = new UTF7Encoding();
        format.Culture = new CultureInfo(System.Threading.Thread.CurrentThread.CurrentCulture.ToString());
        format.Culture.DateTimeFormat.ShortDatePattern = "dd.mm.yyyy";

        using (ExcelPackage package = new ExcelPackage(new FileInfo(file.Name))){
            ExcelWorksheet worksheet = package.Workbook.Worksheets.Add(Path.GetFileNameWithoutExtension(file.Name));
            worksheet.Cells["A1"].LoadFromText(new FileInfo(file.FullName), format);
        }

I have verified that it is the LoadFromText command that spends the time used.

Is there a way to speed things up? I have tried without the "format" parameter, but the loadtime was the same.

What loadtimes are you experiencing?

Answer 1

My suggestion here is to read the file by yourself and then use the library to create the file.

The code to read the CSV could be as simple as:

List<String> lines = new List<String>();
using (StreamReader reader = new StreamReader("file.csv"))
{
    String line; 
    while((line = reader.ReadLine()) != null)
    {
        lines.add(line);
    }
}

//Now you got all lines of your CSV

//Create your file with EPPLUS

foreach(String line in lines)
{
    var values = line.Split(';');
    foreach(String value in values)
    {
        //use EPPLUS library to fill your file
    }
}

Answer 2

I ran into a very similar problem with LoadFromCollection. EPPlus has to account for all situations in their methods to load data generically like that so there is a good deal of overhead. I ended up narrowing done the bottleneck to that method and ended up just manually coverting the data from the collection to Excel Cell object in EPPlus. Probably saved several minutes in my exports.

Plenty of examples on how to read csv data:

C# Read a particular value from CSV file