Elasticsearch java pagination not working

Question

I am trying to fetch all data from elastic index with scroll id.

SearchQuery searchQuery = new NativeSearchQueryBuilder()
                    .withIndices(ELASTIC_INDEX)
                    .withTypes(DOC_TYPE)
                    .withPageable(PageRequest.of(30500, BATCH_SIZE))
                    .withFields("id", "pid")
                    .build();
users = elasticsearchTemplate.startScroll(SCROLL_TIMEOUT, searchQuery, User.class);

But the pagination is not working. It always returns all the documents from page 0.

There are total 30500 documents in the index, so when i set .withPageable(PageRequest.of(30500, BATCH_SIZE)) with batch size 10, there should not be any data returned, but it still returns all data. Also while using scroll for page=304,size=1000 it should return only last 500 documents but it returns 1000 documents.

What i am doing wrong here?

Answer 1

According to javadoc of org.springframework.data.domain.PageRequest#of(int, int) static constructor:

Params: page – zero-based page index, must not be negative. size – the size of the page to be returned, must be greater than 0

Page index is not document index, so you have only documents/BATCH_SIZE = 30500/10 = 3050 indexed pages, and overflowing this param value may cause unexpected behavior