Mnesia pagination with fragmented table

Question

I have a mnesia table configured as follow:

-record(space, {id, t, s, a, l}).

mnesia:create_table(space, [ {disc_only_copies, nodes()},
                                                {frag_properties, [ {n_fragments, 400}, {n_disc_copies, 1}]},
                                                {attributes, record_info(fields, space)}]),

I have at least 4 million records for test purposes on this table.

I have implemented something like this Pagination search in Erlang Mnesia

fetch_paged() ->
   MatchSpec = {'_',[],['$_']},
   {Record, Continuation} = mnesia:activity(async_dirty, fun mnesia:select/4, [space, [MatchSpec], 10000, read], mnesia_frag).

next_page(Cont) ->
   mnesia:activity(async_dirty, fun() -> mnesia:select(Cont) end, mnesia_frag).

When I execute the pagination methods it brings batch between 3000 and 8000 but never 10000.

What I have to do to bring me the batches consistently?

Answer 1

The problem is that you expect mnesia:select/4, which is documented as:

select(Tab, MatchSpec, NObjects, Lock) -> transaction abort | {[Object],Cont} | '$end_of_table'

to get you the NObjects limit, being NObjects in your example 10,000.

But the same documentation also says:

For efficiency the NObjects is a recommendation only and the result may contain anything from an empty list to all available results. 

and that's the reason you are not getting consistent batches of 10,000 records, because NObjects is not a limit but a recommendation batch size.

If you want to get your 10,000 records you won't have no other option that writing your own function, but select/4 is written in this way for optimization purposes, so most probably the code you will be written will be slower than the original code.

BTW, you can find the mnesia source code on https://github.com/erlang/otp/tree/master/lib/mnesia/src