What is best practice for batching Grakn insert queries?
from the docs:
"Keep the number of operations per transaction minimal. Although it is technically possible to commit a write transaction once after many operations, it is not recommended. To avoid lengthy rollbacks, running out of memory and conflicting operations, it is best to keep the number of queries per transaction minimal, ideally to one query per transaction."
On the other hand, I've heard some recommend 500-1000 queries per commit.
What are the possible gains, bottlenecks and risks?
In general, we would like to use many small lightweight transactions and keep this as the recommended advice where possible. However, in Grakn Versions < 2.0 there are still significant speed gains to be made by batching queries into a larger transaction, each of which can insert between 500-1000 concepts before committing.
In Grakn versions before 2.0, the Grakn's storage system has relatively expensive open and commit transaction operations, which should be much lighter in 2.0 and going forwards. By using small transactions, you also end up with smaller re-try operations for the cases where transactions fail due for any reason (for example, due to conflicts with other write transactions).