How big can Erlang DETS be and what to do if its too small?

1.5k views Asked by At

All I need is a large persistent lookup table in Erlang and dets seems like just the thing though I need a definative answer to:

  • just how big the total size of the binaries in the table can be.
  • how big each entry can be
  • what to do if the answer to the first question is less than 100G
3

There are 3 answers

1
Christian On BEST ANSWER

One obvious approach, once it is thought of, is to hash dets entries over multiple dets files.

A linear hash should make it dynamically growable, by splitting buckets into newly created dets files when one file reaches an upper threshold.

There are also a number of port drivers that enable you to use sleepycat/berkely db, or tokyo tyrrant. Those databases have file limits that are much higher than 2Gb.

2
Dustin On

This is kind of an RTFM question. As quoted directly in the second paragraph of the DETS manual:

The size of Dets files cannot exceed 2 GB. If larger tables are needed, Mnesia's table fragmentation can be used.

0
Arthur Ulfeldt On

using mnesia fragmented dic_copies can overcome these limits provided you know how many fragments to crate ahead of time
http://www.trapexit.org/Mnesia_Table_Fragmentation