Elasticsearch SEARCH-API ignores some existing indices when searching with wildcards

Question

I want to retrieve information about all available indices in my elasticsearch db. For that I send a request to "<elasticsearch_endpoint>/logs-cfsyslog-*/_search/?format=json".

The body of the request is irrelevant for this problem. I'm simple filtering for a specifiy value for one field. I would expect that the api returns all indices of the last 30 days. However, I only receive some of the available archives. Some that are missing are: 3rd March, 11th-17th and 26th-27th February.

But when I retrieve all available indices with the "_CAT" API via "<elasticsearch_endpoint>/_cat/indices/logs-cfsyslogs-*" I can see ALL indices that I expect.

I can even specify the exact date that I'm looking for in the search API via: "<elasticsearch_endpoint>/logs-cfsyslog-2022.03.03/_search/?format=json" and the API will return the index that I specified.

So why or how does elasticsearch not return for example the index from 3rd March 2022 when I use the wildcard "*" in the search request?

Answer 1

it may be due to one of the below reson.

First, Default value of size is 10

Considering you are calling "<elasticsearch_endpoint>/logs-cfsyslog-*/_search/?format=json" this API and not passing size parameter so elastic search return max 10 documents in response. try below API and check how many result you are getting and from which index.

<elasticsearch_endpoint>/logs-cfsyslog-*/_search/?format=json&size=10000

Second, Due to filtering

I'm simple filtering for a specifiy value for one field.

As you mentioned in question, you are applying filter for one field on specific value so might be chances that filter condition is not matching with other indices.

Please check what value you are getting for hits.total in your response and based on that you can set value of size parameter. Please not that elasticsearch will return max 10,000 documents.

"hits" : {
    "total" : {
      "value" : 5,
      "relation" : "eq"
    }
}