Adding 5834580 of node to Spatial Layer

Question

I am trying to create an R-TREE of 5834580 of nodes.
I found in this question a similair problem and i tried it's solution, so this is my code :

call apoc.periodic.commit("MATCH (pl:pickup_location) WITH collect(pl) AS pickup CALL spatial.addNodes('nyc',pickup) YIELD count RETURN count",{limit:1000})

however, since yesterday the computer didn't finish loading the result.

today, i tried the second answer with iterate :

CALL apoc.periodic.iterate(
"MATCH (pl:pickup_location) RETURN pl",
"CALL spatial.addNode('nyc', pl) YIELD node RETURN node",
{batchSize:10000, parallel:false, listIterate:true})

and i get this error :

Neo.ClientError.Procedure.ProcedureCallFailed: Failed to invoke procedure `apoc.periodic.iterate`: Caused by: java.lang.OutOfMemoryError: Java heap space

what is wrong ? what should i do ?

Answer 1

Your problem with the apoc.periodic.commit is that your query will always return the same nodes with your MATCH (pl:pickup_location). There is no condition to find only nodes that are not in the spatial layout.

I don't remember the model of the spatial plugin, but from what I remember, on your pickup_location nodes, you should have a specific relation to the R-Tree.

So you should transform your auery to something like that :

CALL apoc.periodic.commit("
  MATCH (pl:pickup_location)
  WHERE NOT (p1)-[:LINKS->(:spatialNode) // Change this according to the spatial model
  WITH p1 AS node LIMIT $limit
  WITH collect(node) AS pickup 
    CALL spatial.addNodes('nyc',pickup) YIELD count 
    RETURN count",
  {limit:1000}
)

For the problem on the apoc.periodic.iterate is just a memory problem, you don't have enought RAM to execute transactions.

You have two solutions :

• give more RAM to Neo4j by increasing the heap size of Neo4j (see the neo4j.conf file)
• decrease the size of the batch, 10000 is a little big, change it to 1000

Answer 2

As I don't have a rep of 50 I can't comment but logisima is spot on, just to add to their answer... the "where not" clause should be:

where not (p1)-[:RTREE_REFERENCE]-() 

RTREE_REFERENCE being the relationship created when adding a node to the spatial layer.