Elasticsearch: Pattern in high CPU usage after we started using elasticsearch high level client

746 views Asked by At

We started using elasticsearch high level client recently and we use scroll API to fetch large set of data from ES. We see a pattern in high CPU utilization as follows:

enter image description here

It's pattern repeating every 30 minutes. No clue what's going on. We see exception in elasticsearch too -

[2021-05-12T04:19:29,516][DEBUG][o.e.a.s.TransportSearchScrollAction] [node-2] [93486247] Failed to execute query phase org.elasticsearch.transport.RemoteTransportException: [node-3][10.160.86.222:7550][indices:data/read/search[phase/query/scroll]] Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [93486247] at org.elasticsearch.search.SearchService.getExecutor(SearchService.java:496) ~[elasticsearch-6.8.9.jar:6.8.9] at org.elasticsearch.search.SearchService.runAsync(SearchService.java:373) ~[elasticsearch-6.8.9.jar:6.8.9] at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:435) ~[elasticsearch-6.8.9.jar:6.8.9] at org.elasticsearch.action.search.SearchTransportService$8.messageReceived(SearchTransportService.java:376) ~[elasticsearch-6.8.9.jar:6.8.9] at org.elasticsearch.action.search.SearchTransportService$8.messageReceived(SearchTransportService.java:373) ~[elasticsearch-6.8.9.jar:6.8.9] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:250) ~[?:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.9.jar:6.8.9]

The high level client code being used is the usual code given in the official documentation-

final Scroll scroll = new Scroll(TimeValue.timeValueMinutes(1L));
        SearchRequest searchRequest = new SearchRequest(INDEX_NAME);
        searchRequest.scroll(scroll);
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

        BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();

        if (StringUtils.isNotBlank(keyword)) {
            LOG.info("Searching for keyword: {}", keyword);
            boolQueryBuilder.must(QueryBuilders.multiMatchQuery(keyword, INDEXED_FIELDS));
        }

        if(StringUtils.isNotBlank(param1)) {
            boolQueryBuilder.filter(QueryBuilders.termQuery("param1", param1));
        }

        if(Objects.nonNull(param1)) {
            boolQueryBuilder.filter(QueryBuilders.termsQuery("param1", param1));
        }

        if(Objects.nonNull(param1)) {
            boolQueryBuilder.filter(QueryBuilders.termsQuery("param1", param1));
        }

        if(Objects.nonNull(param1)) {
            boolQueryBuilder.filter(QueryBuilders.termsQuery("param1", param1));
        }

        if(Objects.nonNull(param1)) {
            boolQueryBuilder.filter(QueryBuilders.termsQuery("param1", param1));
        }

        searchSourceBuilder.query(boolQueryBuilder);

        searchRequest.source(searchSourceBuilder);

        List<Object1> statuses = new ArrayList<>();
        try {
            SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
            String scrollId = searchResponse.getScrollId();
            SearchHit[] searchHits = searchResponse.getHits().getHits();

            while (searchHits != null && searchHits.length > 0) {
                for (SearchHit hit : searchHits) {
                    Object1 agent = JsonUtil.parseJson(hit.getSourceAsString(),
                    Object1.class);
                    statuses.add(agent);
                }
                SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId);
                scrollRequest.scroll(scroll);   
                searchResponse = client.scroll(scrollRequest, RequestOptions.DEFAULT);
                scrollId = searchResponse.getScrollId();
                searchHits = searchResponse.getHits().getHits();
            }

            ClearScrollRequest clearScrollRequest = new ClearScrollRequest();
            clearScrollRequest.addScrollId(scrollId);
            ClearScrollResponse clearScrollResponse = client.clearScroll(clearScrollRequest, RequestOptions.DEFAULT);
            boolean succeeded = clearScrollResponse.isSucceeded();
0

There are 0 answers