I am working on a data mining C# project where I deal with 16 mb text files. I should process the files in runtime so after the user selects the files the program should load them one by one, search for keywords, split them into objects. So the program may have about 50 objects each of them is about 10 mb so about 500 mb of RAM.
I don't need all of these objects at once so I need to store them somewhere outside the RAM and then load the one I need.
What's the best way to deal with this situation?
IMHO I think you should save all your files in a database after the user select the files in case you want to keep them. Then you can deal with the processing problem. For example:
If you want to parse your files and extract information from them. Definitely you should consider to use ANTLR. To perform all operations above your AST(Abstract Syntax Tree). And if you know the structure of your text then you can define a grammar and load the entire AST from your file.
If you are trying to make just a simple search over the text, try to point to Knuth–Morris–Pratt algorithm or use a Trie
And for the performance, try to use a StringBuilder instead the regular string class