Scalable stl set like container for C++

334 views Asked by At

I need to store large number of integers. There can be duplicates in the input stream of integers, I just need to store distinct amongst them. I was using stl set initially but It went OutOfMem when input number of integers went too high. I am looking for some C++ container library which would allow me to store numbers with the said requirement possibly backed by file i.e container should not try to keep all numbers in-mem. I don't need to store this data persistently, I just need to find unique values amongst it.

4

There are 4 answers

0
Richard Wolf On

You should seriously at least try a database before concluding it is too slow. All you need is one of the lightweight key-value store ones. In the past I have used Berkeley DB, but here is a list of other ones.

1
Jay On

Since you need larger than RAM allows you might look at memcached

1
tzaman On

Take a look at the STXXL; might be what you're looking for.

Edit: I haven't used it myself, but from the docs - you could use stream::runs_creator to create sorted runs of your data (however much fits in memory), then stream::runs_merger to merge the sorted streams, and finally use stream::unique to filter uniques.

1
Dmitry Yudakov On

Have you considered using DB (maybe SQLite)? Or it would be too slow?