All I need is a large persistent lookup table in Erlang and dets seems like just the thing though I need a definative answer to:
- just how big the total size of the binaries in the table can be.
- how big each entry can be
- what to do if the answer to the first question is less than 100G
One obvious approach, once it is thought of, is to hash dets entries over multiple dets files.
A linear hash should make it dynamically growable, by splitting buckets into newly created dets files when one file reaches an upper threshold.
There are also a number of port drivers that enable you to use sleepycat/berkely db, or tokyo tyrrant. Those databases have file limits that are much higher than 2Gb.