elasticsearch bulk indexing and redundant data in action part

Question

When indexing data using bulk API of elasticsearch here is the sample json from the site documentation

POST _bulk
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1" } }
{ "field1" : "value1" }
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "2" } }
{ "field1" : "value2" }
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "3" } }
{ "field1" : "value3" }

While "preparing" the data to be used by the bulk API, on first line I have to specify the operation and in next line I will provide data. Some redundant parts on each line might look obvious and pretty harmless but when I am indexing trillions of rows, doesn't it add up to latency? Is there is better way to push all the rows by specifying the index name and type only once at the header? Specially when I can use autogenerated id, I can avoid generating terabytes of data just to be prepended to every row for the same purpose again and again.

I believe I am missing something obvious here otherwise I am sure those guys at elastic are smart enough to have figured it out already and if they have done it this way, there should be some reason. But what?

Answer 1

Here you have shortcut:

POST /test/_doc/_bulk
{ "index": {} }
{ "field1" : "value1" }
{ "index": {} }
{ "field1" : "value2" }
{ "index": {} }
{ "field1" : "value3" }

Unfortunately you still need to repeat the { "index": {} } line but index name and document type you have specified in the path.

Please see more options in the Cheaper in Bulk article.

Answer 2

Well there's no better way in terms of preparing the data. Only thing you can do is to prepare the data programmatically.

You can simply write a code to construct the desired json and send it across using _bulk API.

Ideally it is best to have indexing done via a specific application called indexer which would actually wait until a batch of documents for e.g. 50 or 100 is collected and then execute the _bulk API programmatically.

Or instead of batch processing like that, you can have it document by document i.e. event based using messaging queues. (Best approach to minimize latency during indexing process)

Another option is to create an input file say data.json(purely batch processing) using a simple java program or any other programming language you use, append all documents you want programmatically and use CURL command to send the request as shown below:

$ curl -s -XPOST <host_name>:9200/_bulk --data-binary @data.json

So for this indexer application, you can add scheduling as well as mail notifications in such a way that you'd get to know the status of every job run and schedule time as when to run everyday/week depending on your requirement.

Otoh, you can make use of Logstash. Sorry, its not the best answer, but I hope it helps.

Answer 3

As was already told in Piotr Pradzynski's great answer, there's not much you can do, and the minimal footprint is the one Pyotr proposed. There are a couple of details that I believe deserve to be added.

How does bulk API help?

The main reason to consider bulk API is tuning for indexing speed. The improvements in performance here are largely due to saving on handling less HTTP connections on the Elasticsearch side. Practically speaking, your cluster will not be indexing the documents faster if you manage not to send those repetitive { "index": {} } parts.

What if network bandwidth is the bootle neck?

In this case I believe the best one can do is to send the data compressed, like this:

curl -v 'http://localhost:9200/my_index/doc/_bulk' \
    -H "Content-encoding: gzip"
    -H "content-type: application/json; charset=UTF-8"
    -X POST --data-binary @bulk_data.json.gz

To illustrate the idea I generated a file with random data that looks like this:

$ head bulk_data.json
{"index":{}}
{"request_id":"40485"}
{"index":{}}
{"request_id":"12417"}
{"index":{}}
{"request_id":"11945"}
{"index":{}}
{"request_id":"81722"}
{"index":{}}
{"request_id":"52613"}

The size of the file is 10 times smaller after compression with GZip:

$ ls -l
-rw-r--r--  1 vasiliev  staff  358836 Nov 16 20:09 bulk_data.json
-rw-r--r--  1 vasiliev  staff   35744 Nov 16 19:41 bulk_data.json.gz

This might help a lot in the case of limited bandwidth.

Compression is also available from client libraries, like elasticsearch-py library.

Hope that helps!