ElasticSearch error ""Message":"Request size exceeded 10485760 bytes"

Question

While uploading a 50 mb size huge JSON string in ElasticSearch using this method -

public static void postData(String json, String index, String type) {
    RestClient client = RestClient.builder(new HttpHost(testHostPreProd, 443, "https")).build();
    HttpEntity entity = new NStringEntity(json, ContentType.APPLICATION_JSON);

    Response response = client.performRequest("POST", "/" + index + "/" + type + "/_bulk",
            Collections.<String, String>emptyMap(), entity);

    client.close();
}

The error is

Exception in thread "main" org.elasticsearch.client.ResponseException: POST https:someURL/indexHere/typeHere/_bulk: HTTP/1.1 413 Request Entity Too Large
{"Message":"Request size exceeded 10485760 bytes"}

Putting a counter in the code that works send 10k batches might work but I'm not sure how to do that.

Any suggestions on how to handle this issue?

Answer 1

For downloading a big amount of records you can use Scroll API:

https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html

The mechanism is quite simple:

1. You create a query and give the chunk size, i.e. number of records for a single download – let's say 10.
2. It returns your [ten] records and a scroll_id which is the marker to the last document that you have downloaded.
3. If there are more records (20) it goes to step 2. If there are no records, you delete the scroll.

Something like this:

Request request = new Request("GET", "/" + index + "/_search?scroll=1m");
request.setEntity(new NStringEntity(jsonQuery, ContentType.APPLICATION_JSON));
Response response = client.performRequest(request);
String scrollId = jsonNode.path("_scroll_id").asText();

//Extract data you need

//Repeat the scroll, by passing scroll_id

Request scrollRequest = new Request("GET", "/_search/scroll");
scrollRequest.setEntity(new NStringEntity("{\"scroll_id\": \"" + scrollId + "\"}", ContentType.APPLICATION_JSON));
response = client.performRequest(scrollRequest);
responseBody = EntityUtils.toString(response.getEntity());
scrollId = jsonNode.path("_scroll_id").asText();