I'm troubleshooting an issue with our application hosted in K8s pod, which tries to query ES cluster through ES Jest Client. Most of the queries were successful, but randomly a call fails with read timeout. Any help will be appreciated.
Jest Client version:6.3.1
ES cluster is hosted in AWS
Below i'm providing apache http debug logs.
o.a.h.c.protocol.RequestAuthCache - Auth cache not set in the context o.a.h.i.c.BasicHttpClientConnectionManager - Get connection for route {}->http://dev-app-elasticsearch-lb.xyz.com:9200 o.a.h.impl.execchain.MainClientExec - Executing request POST /_search?search_type=dfs_query_then_fetch&preference=_primary_first HTTP/1.1 o.a.h.impl.execchain.MainClientExec - Target auth state: UNCHALLENGED o.a.h.impl.execchain.MainClientExec - Proxy auth state: UNCHALLENGED org.apache.http.headers - http-outgoing-16 >> POST /_search?search_type=dfs_query_then_fetch&preference=_primary_first HTTP/1.1 org.apache.http.headers - http-outgoing-16 >> Content-Length: 2531 org.apache.http.headers - http-outgoing-16 >> Content-Type: application/json; charset=UTF-8 org.apache.http.headers - http-outgoing-16 >> Host: dev-app-elasticsearch-lb.xyz.com:9200 org.apache.http.headers - http-outgoing-16 >> Connection: Keep-Alive org.apache.http.headers - http-outgoing-16 >> User-Agent: Apache-HttpClient/4.5 (Java/1.8.0_181) org.apache.http.headers - http-outgoing-16 >> Accept-Encoding: gzip,deflate org.apache.http.wire - http-outgoing-16 >> "POST /_search?search_type=dfs_query_then_fetch&preference=_primary_first HTTP/1.1[\r][\n]" org.apache.http.wire - http-outgoing-16 >> "Content-Length: 2531[\r][\n]" org.apache.http.wire - http-outgoing-16 >> "Content-Type: application/json; charset=UTF-8[\r][\n]" org.apache.http.wire - http-outgoing-16 >> "Host: dev-app-elasticsearch-lb.xyz.com:9200[\r][\n]" org.apache.http.wire - http-outgoing-16 >> "Connection: Keep-Alive[\r][\n]" org.apache.http.wire - http-outgoing-16 >> "User-Agent: Apache-HttpClient/4.5 (Java/1.8.0_181)[\r][\n]" org.apache.http.wire - http-outgoing-16 >> "Accept-Encoding: gzip,deflate[\r][\n]" org.apache.http.wire - http-outgoing-16 >> "[\r][\n]" org.apache.http.wire - http-outgoing-16 >> "{[\n]" ....<<outgoing query details>> o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections idle longer than 60000 MILLISECONDS o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections idle longer than 60000 MILLISECONDS o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections idle longer than 60000 MILLISECONDS o.a.h.i.c.PoolingHttpClientConnectionManager - Closing connections idle longer than 60000 MILLISECONDS c.t.c.util.concurrent.DatabaseLock - Cleaning up expired locks from DS 0 o.a.h.i.c.BasicHttpClientConnectionManager - Closing connection o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-6: Close connection o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Closing connections idle longer than 15 MINUTES o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Closing connections idle longer than 15 MINUTES o.a.h.i.n.c.PoolingNHttpClientConnectionManager - Closing connections idle longer than 15 MINUTES```
Any help would be greatly appreciated.
Thanks Naresh