Foxx arangodb runs into memory limit

Question

I have an simple query runs inside foxx

For u in collection
Filter u.someIndexedSparseFiler !=null
Return {_id:u._id}

This will return millions+ results. In the logs, arango have a message of limited memory heap reached and terminate the process.

reached heap-size limit of #3 interrupting V8 execution (heap size limit 3232954528, used 3060226424) during V8 internal collection

Even though I add the flag --javascript.v8-max-heap 3000 to the start-up. It still runs in the same error. What should I do? Is there a better approach than this

Answer 1

I'm not sure why you're getting out-of-memory errors, but it looks like the data you're returning is overflowing the V8 heap size. Another possibility is that something is causing the engine to miss/ignore the index, causing the engine to load every document before evaluating the someIndexedSparseFiler attribute.

Evaluating millions of documents (or lots of large documents) would not only cost a lot of disk/memory I/O, but could also require a lot of RAM. Try using the explain feature to return a query analysis - it should tell you what is going wrong.

For comparison, my query...

FOR u IN myCollection
    FILTER u.someIndexedSparseFiler != null
    RETURN u._id

...returns this when I click "explain":

Query String (82 chars, cacheable: true):
 FOR u IN myCollection
     FILTER u.someIndexedSparseFiler != null
     RETURN u._id

Execution plan:
 Id   NodeType          Est.   Comment
  1   SingletonNode        1   * ROOT
  7   IndexNode            5     - FOR u IN myCollection   /* persistent index scan, projections: `_id` */    
  5   CalculationNode      5       - LET #3 = u.`_id`   /* attribute expression */   /* collections used: u : myCollection */
  6   ReturnNode           5       - RETURN #3

Indexes used:
 By   Name                      Type         Collection     Unique   Sparse   Selectivity   Fields                         Ranges
  7   idx_1667363882689101824   persistent   myCollection   false    true        100.00 %   [ `someIndexedSparseFiler` ]   *

Optimization rules applied:
 Id   RuleName
  1   move-calculations-up
  2   move-filters-up
  3   move-calculations-up-2
  4   move-filters-up-2
  5   use-indexes
  6   remove-filter-covered-by-index
  7   remove-unnecessary-calculations-2
  8   reduce-extraction-to-projection

Note that it listss my sparse index under Indexes used:. Also, try changing the != to == and you will see that now it ignores the index! This is because the optimizer knows a sparse index will never have a null value, so it skips it.

If you aren't familiar with it, the "explain" functionality is extremely useful (indispensable, really) when tuning queries and creating indexes. Also, remember that indexes should match your query; in this case, the index should only have one attribute or the "selectivity" quotient may be too low and the engine will ignore it.