I am trying to understand how to implement Partitioner<T>.
My case:
We read from a file and we insert to the Azure table storage. We use tasks in order to speed up the process. The file has nearly 10,000,000 lines.
I tried to implement a more elegant solution using Parallel execution and partitioning. I would like to partition this large group of 9,000,000 into a group of partition sets and distribute them in 30-50 threads.
I would think that I need a dynamic implementation of Partitioner<List<List<TableEntity>>>. I have few examples, the one with the StringPartitioner, but it's not really helpful for my case as I need an IEnumerator for List<List<TableEntity>>.
If all this is confusing, please let me know so I can clarify.
The problem is how to implement a dynamic Partitioner in List<List<TableEntity>>.