We started using elasticsearch high level client recently and we use scroll API to fetch large set of data from ES. We see a pattern in high CPU utilization as follows:
It's pattern repeating every 30 minutes. No clue what's going on. We see exception in elasticsearch too -
[2021-05-12T04:19:29,516][DEBUG][o.e.a.s.TransportSearchScrollAction] [node-2] [93486247] Failed to execute query phase org.elasticsearch.transport.RemoteTransportException: [node-3][10.160.86.222:7550][indices:data/read/search[phase/query/scroll]] Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [93486247] at org.elasticsearch.search.SearchService.getExecutor(SearchService.java:496) ~[elasticsearch-6.8.9.jar:6.8.9] at org.elasticsearch.search.SearchService.runAsync(SearchService.java:373) ~[elasticsearch-6.8.9.jar:6.8.9] at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:435) ~[elasticsearch-6.8.9.jar:6.8.9] at org.elasticsearch.action.search.SearchTransportService$8.messageReceived(SearchTransportService.java:376) ~[elasticsearch-6.8.9.jar:6.8.9] at org.elasticsearch.action.search.SearchTransportService$8.messageReceived(SearchTransportService.java:373) ~[elasticsearch-6.8.9.jar:6.8.9] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.9.jar:6.8.9]
The high level client code being used is the usual code given in the official documentation-
final Scroll scroll = new Scroll(TimeValue.timeValueMinutes(1L));
SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
searchRequest.scroll(scroll);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
if (StringUtils.isNotBlank(keyword)) {
LOG.info("Searching for keyword: {}", keyword);
boolQueryBuilder.must(QueryBuilders.multiMatchQuery(keyword, INDEXED_FIELDS));
}
if(StringUtils.isNotBlank(param1)) {
boolQueryBuilder.filter(QueryBuilders.termQuery("param1", param1));
}
if(Objects.nonNull(param1)) {
boolQueryBuilder.filter(QueryBuilders.termsQuery("param1", param1));
}
if(Objects.nonNull(param1)) {
boolQueryBuilder.filter(QueryBuilders.termsQuery("param1", param1));
}
if(Objects.nonNull(param1)) {
boolQueryBuilder.filter(QueryBuilders.termsQuery("param1", param1));
}
if(Objects.nonNull(param1)) {
boolQueryBuilder.filter(QueryBuilders.termsQuery("param1", param1));
}
searchSourceBuilder.query(boolQueryBuilder);
searchRequest.source(searchSourceBuilder);
List<Object1> statuses = new ArrayList<>();
try {
SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
String scrollId = searchResponse.getScrollId();
SearchHit[] searchHits = searchResponse.getHits().getHits();
while (searchHits != null && searchHits.length > 0) {
for (SearchHit hit : searchHits) {
Object1 agent = JsonUtil.parseJson(hit.getSourceAsString(),
Object1.class);
statuses.add(agent);
}
SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
scrollRequest.scroll(scroll);
searchResponse = client.scroll(scrollRequest, RequestOptions.DEFAULT);
scrollId = searchResponse.getScrollId();
searchHits = searchResponse.getHits().getHits();
}
ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
clearScrollRequest.addScrollId(scrollId);
ClearScrollResponse clearScrollResponse = client.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
boolean succeeded = clearScrollResponse.isSucceeded();