Balance availability and performance: Iterate and process over 10k MS Office Files (1 TB) with .NET on a shared network drive

Question

I wrote a Visual Basic .NET application in Visual Studio to parse MS Powerpoint and Word Files, transform slides to jpgs and to store the content in ElasticSearch. I want to ensure that when parsing a shared network drive that neither the server or my client dramatically slow down. How can I monitor the execution and adapt the processing accordingly? Any basic techniques to get me started?

Answer 1

In my answer I assume that the load mainly consists of reading the PowerPoint and Word files. So enumerating directories, generating the JPGs and writing the result isn't an issue. If not, the approach can be extended.

A simple approach would be:

1. Figure out how much IO load you want to generate at most, e.g. at most 5 MB/s. This is your read rate you do not want to exceed.

2. Retrieve the time before you start processing a file.

3. Retrieve the file size.

4. After processing a file, take the time again and calculate the duration.

5. When processing a file, you'll probably going over your read rate. So after processing a file, calculate how long you have to wait to fall under the read rate again and then wait. The calculation is basically

wait_time = file_size / read_rate - duration

Use matching units such as seconds for wait_time and duration, bytes for file_size and bytes per second for the read_rate.

If the wait_time is negative, skip the waiting.