Can I use boost geometry on local disk

Question

I am new to boost geometry. In my case, I need handle a large mount of data nodes, so they cannot be saved in memory. Is it possible to use boost geometry together with local file system?

Answer 1

A generic answer is: use a memory mapped file from Boost Interprocess (IPC) with (boost) containers that use the IPC allocators. ^[1]

This will make it possible to work with /virtually/ unlimited data sizes transparently (at least in 64bit processes).

However Paging Is Expensive.

Boost Geometry is likely not optimized for sequential access patterns, so you might need to play very tight with what algorithms work and in what order to apply them. Otherwise, scaling this kind of volume (I'm assuming >16Gb for simplicity) will in practice turn out unbearably slow due to paging.

In all usual circumstances, scaling to non-trivial volumes involves tuning the algorithms or even writing targeted ones for the purpose.

Without any knowledge of the actual task at hand you could try

• starting with memory mapped data allocation
• slowly start building the algorithmic steps, one by one at a time
• each step, incrementally grow your data set while keeping a close eye on the profiler

Your profiler will tell what step introduces a performance bottle-neck and at what volume it becomes discernible.

---

^[1] this gives you persistence for "free"; however, keep in mind you are responsible for transactions and fsync-ing at proper times. Also, contiguous/sequential containers work best.