I need to store large number of integers. There can be duplicates in the input stream of integers, I just need to store distinct amongst them. I was using stl set initially but It went OutOfMem when input number of integers went too high. I am looking for some C++ container library which would allow me to store numbers with the said requirement possibly backed by file i.e container should not try to keep all numbers in-mem. I don't need to store this data persistently, I just need to find unique values amongst it.
Scalable stl set like container for C++
347 views Asked by Pqr At
4
There are 4 answers
1
On
Take a look at the STXXL; might be what you're looking for.
Edit: I haven't used it myself, but from the docs - you could use stream::runs_creator
to create sorted runs of your data (however much fits in memory), then stream::runs_merger
to merge the sorted streams, and finally use stream::unique
to filter uniques.
You should seriously at least try a database before concluding it is too slow. All you need is one of the lightweight key-value store ones. In the past I have used Berkeley DB, but here is a list of other ones.