I received an error org.apache.hc.core5.http.NoHttpResponseException: server failed to respond when i call API via quotaguard proxy in SpringBoot 3

Question

We are using QuotaGuard proxy to connect with external API.
Recently we did Spring Boot version update.
Technical configuration of our application.
Before Spring Boot version update.
Java 8
Spring Boot 2.7
Htttclient 4.5
After Spring Boot version update.
Java 17
Spring Boot 3.0
Htttclient5 5.2.1
However after migration we cannot connect to the API. We get the following error message.

Nov 29 02:57:08  app/web.1 org.springframework.web.client.ResourceAccessException: I/O error on POST request for "[https://test.com/postInfo]": [test.com:443]  failed to respond
Nov 29 02:57:08  app/web.1  at org.springframework.web.client.RestTemplate.createResourceAccessException(RestTemplate.java:888)
Nov 29 02:57:08  app/web.1  at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:868)
Nov 29 02:57:08  app/web.1  at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:764)
Nov 29 02:57:08  app/web.1  at org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:512)
Nov 29 02:57:08  app/web.1  at jp.co.demo.service.WinnerExportServiceImpl$3.doWithRetry(WinnerExportServiceImpl.java:381)
Nov 29 02:57:08  app/web.1  at jp.co.demo.service.WinnerExportServiceImpl$3.doWithRetry(WinnerExportServiceImpl.java:376)
Nov 29 02:57:08  app/web.1  at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286)
Nov 29 02:57:08  app/web.1  at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:179)
Nov 29 02:57:08  app/web.1  at jp.co.demo.service.WinnerExportServiceImpl.execute(WinnerExportServiceImpl.java:376)
Nov 29 02:57:08  app/web.1  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Nov 29 02:57:08  app/web.1  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
Nov 29 02:57:08  app/web.1  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Nov 29 02:57:08  app/web.1  at java.base/java.lang.reflect.Method.invoke(Method.java:568)
Nov 29 02:57:08  app/web.1  at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
Nov 29 02:57:08  app/web.1  at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
Nov 29 02:57:08  app/web.1  at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
Nov 29 02:57:08  app/web.1  at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
Nov 29 02:57:08  app/web.1  at org.springframework.util.concurrent.FutureUtils.lambda$toSupplier$0(FutureUtils.java:74)
Nov 29 02:57:08  app/web.1  at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
Nov 29 02:57:08  app/web.1  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
Nov 29 02:57:08  app/web.1  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
Nov 29 02:57:08  app/web.1  at java.base/java.lang.Thread.run(Thread.java:840)
Nov 29 02:57:08  app/web.1 Caused by: org.apache.hc.core5.http.NoHttpResponseException: [test.com:443] failed to respond
Nov 29 02:57:08  app/web.1  at org.apache.hc.core5.http.impl.io.DefaultHttpResponseParser.createConnectionClosedException(DefaultHttpResponseParser.java:87)
Nov 29 02:57:08  app/web.1  at org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:243)
Nov 29 02:57:08  app/web.1  at org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:53)
Nov 29 02:57:08  app/web.1  at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:298)
Nov 29 02:57:08  app/web.1  at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:175)
Nov 29 02:57:08  app/web.1  at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:218)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:712)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:216)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.ConnectExec.createTunnelToTarget(ConnectExec.java:233)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:151)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:96)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:115)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:106)
Nov 29 02:57:08  app/web.1  at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:55)
Nov 29 02:57:08  app/web.1  at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:93)
Nov 29 02:57:08  app/web.1  at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
Nov 29 02:57:08  app/web.1  at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)
Nov 29 02:57:08  app/web.1  at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:862)
Nov 29 02:57:08  app/web.1  ... 20 more

Here are the sample code we are using,

URL proxy;
proxy = new URL(proxyUrl);
String userInfo = proxy.getUserInfo();
String user = userInfo.substring(0, userInfo.indexOf(':'));
String password = userInfo.substring(userInfo.indexOf(':') + 1);

final CredentialsProvider credsProvider = new BasicCredentialsProvider();
Credentials credentials = new UsernamePasswordCredentials(user, password.toCharArray());
((BasicCredentialsProvider) credsProvider).setCredentials(new AuthScope(proxy.getHost(), proxy.getPort()), credentials);

HttpHost p = new HttpHost(proxy.getHost(), proxy.getPort());
HttpClient httpClient = HttpClients.custom().setProxy(p).setDefaultCredentialsProvider(credsProvider).build();
final HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory();
factory.setHttpClient(httpClient);

RestTemplate restTemplate = new RestTemplate(factory);
ResponseEntity<ResponseData> response = restTemplate.postForEntity(APIURL, req, ResponseData.class);
ResponseData responseBody = response.getBody();

However if we request API using quotaguard proxy via CURL on the command line, it passes successfully.
Also if we request API using Postman, it passes successfully.
Did anyone faced problem to connect API using proxyserver recently after updating SpringBoot 3.0 and Java 17 version?
Can anyone give any guideline how to approach or fix this problem? We have tried several method and example code none of them worked.

Answer 1

So far i found two solutions. Both are working for my scenario.

Solution 1: Telling the system to not keep alive the connection by setting setConnectionReuseStrategy.

final CloseableHttpClient httpClient = Httpclients.custom()
        .setProxy(p)
        .setDefaultCredentialsProvider(credsProvider)
        .setConnectionReuseStrategy(new ConnectionReuseStrategy(){
            @Override
            public boolean keepAlive(HttpRequest request, HttpResponse response, HttpContext context){
                return false;
            }
        }).build();
    

Solution 2: Implementing CustomRetryStrategy which implements RetryStrategy. Then Set CustomRetryStrategy in setRetryStrategy.

final CloseableHttpClient httpClient = Httpclients.custom()
        .setProxy(p)
        .setDefaultCredentialsProvider(credsProvider)
        .setRetryStrategy(new CustomRetryStrategy())
        }).build();     
        
private static class CustomRetryStrategy implements RetryStrategy {
    @Override
    public boolean retryRequest(HttpRequest request, IOException exception, int execCount, HttpContext context) {
        //Set execCount according to business requirement.
        return execCount <= 3 ;
    }

    @Override
    public boolean retryRequest(HttpResponse response, int execCount, HttpContext context) {
        //Set execCount according to business requirement.
        return execCount <= 3 ;
    }


    @Override
    public TimeValue getRetryInterval(HttpResponse response, int execCount, RetryExec exec) {
        return TimeValue.ofSeconds(3);
    }
}

Answer 2

Looking at the stacktrace, it seems that it cannot connect to the endpoint.

I've seen you're using proxy too.

Maybe there's some api changes that need rewriting since going from httpclient4 to httpclient5 (I've already spotted that some methods were kept from v4 to v5 but were no-op in v5)

The setProxy seems deprecated also.

Try setting the proxy this way :

HttpHost proxy = new HttpHost("localhost", 8090);
DefaultProxyRoutePlanner routePlanner = new DefaultProxyRoutePlanner(proxy);
HttpClient httpclient = HttpClients.custom()
  .setRoutePlanner(routePlanner)
  .build();

You can also look at full example with auth info here

If not working (as suggested in the comment), and not all your call to the API are failing, the real root-cause should be that your endpoint or proxy is ending the connection without noticing your client it has, so the RestTemplate is using a already closed connection (as suggested by the stacktrace location (createConnectionClosedException) :

Subclasses must override this method to generate an appropriate exception in case of unexpected connection termination by the peer endpoint.

Maybe there's some changes in the default properties of HttpClient as pointed out in their migration guide

Major differences are related to connection management configuration, SSL/TLS and timeout settings when building HttpClient instances.

They also suggest the following :

Set finite socket and connect timeouts.

Set finite connection total time to live (TTL).

Since the connection is closed by the remote endpoint, setting a low TTL for connections could allow the connection manager to close connections that are not more usable.

Another option would be to call the following methods while creating your HttpClient

Makes this instance of HttpClient proactively evict expired connections from the connection pool using a background thread : org.apache.hc.client5.http.impl.classic.HttpClientBuilder.evictExpiredConnections()

Makes this instance of HttpClient proactively evict idle connections from the connection pool using a background thread : org.apache.hc.client5.http.impl.classic.HttpClientBuilder.evictIdleConnections(TimeValue)

Doing so, your ConnectionManager will check if connection are expired or idle for too long and close them proactively, so you won't be using them when doing another call to the API.