Indexing large objects into ElasticSearch

Question

I am trying to index large objects - objects with lots of relations (about 100'000) - into ElasticSearch index, but the objects don't fit into memory. Sometimes Postgres limit is exceeded, sometimes PHP. And I cannot increase the limits any more.

Records are fetched one-by-one, on-demand. Page size has been minimized to 1.

Framework is Symfony3, bundle is FOSElasticaBundle.

Any bright ideas? Write indexed object into disk?

Answer 1

well it's a know problem, indexing with php / ORM is slow. You should have a look at the Java JDBC elasticsearch importer which is far more faster.