How to retry HttpClient request on TunnelRefusedException?

Question

I would like HttpClient 5 to retry requests when facing a TunnelRefusedException (it happens randomly that the proxy refuses the initial connection, but it usually works ok when retrying).

I have tried extending DefaultHttpRequestRetryStrategy but it can only capture exceptions inheriting from IOException:

public class MyRetryStrategy extends DefaultHttpRequestRetryStrategy {

    public MyRetryStrategy() {
        super(2,
                TimeValue.ofSeconds(1L),
                Arrays.asList(InterruptedIOException.class, UnknownHostException.class, ConnectException.class, ConnectionClosedException.class, NoRouteToHostException.class, SSLException.class,
                        TunnelRefusedException.class  // not valid, because it is not an IOException !
                ),
                Arrays.asList(429, 503)
        );
    }
}

So, maybe HttpRequestRetryStrategy interface should support any kind of Exception? Or maybe TunnelRefusedException should be an IOException?

Any ideas about how to do this?

Sample stacktrace of the error that I would like to retry:

   org.apache.hc.client5.http.ClientProtocolException: CONNECT refused by proxy: HTTP/1.1 500 Internal Server Error
        at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:173)
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:245)
        at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188)
[...]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.apache.hc.client5.http.impl.TunnelRefusedException: CONNECT refused by proxy: HTTP/1.1 500 Internal Server Error
        at org.apache.hc.client5.http.impl.classic.ConnectExec.createTunnelToTarget(ConnectExec.java:284)
        at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:151)
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
        at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
        at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:96)
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
        at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
        at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:115)
        at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
        at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
        ... 19 more

Answer 1

DefaultHttpRequestRetryStrategy has excluded exception list in the definition see them here

Arrays.asList( InterruptedIOException.class, UnknownHostException.class, ConnectException.class, ConnectionClosedException.class, NoRouteToHostException.class, SSLException.class)

Easiest way to go is to implement custom strategy and custom exec chain handler.

public class Main {
    public static void main(String[] args) throws IOException {
        successfulCall();
        unknownHostException();
    }

    private static void unknownHostException() throws IOException {
        final ClassicHttpRequest httpGet = ClassicRequestBuilder.get("https://ht11tpbin.org/status/500")
                .build();

        CloseableHttpClient httpClient = HttpClientBuilder
                .create()
                .addExecInterceptorFirst("mainHandler", new CustomExecChainHandler(DefaultHttpRequestRetryStrategy.INSTANCE))
                .setRetryStrategy(new CustomRetryStrategy(0, TimeValue.ofSeconds(2)))
                .build();

        httpClient.execute(httpGet, response -> {
            System.out.println(response.getCode() + " " + response.getReasonPhrase());
            return null;
        });
    }

    private static void successfulCall() throws IOException {
        final ClassicHttpRequest httpGet = ClassicRequestBuilder.get("https://httpbin.org/status/500")
                .build();

        CloseableHttpClient httpClient = HttpClientBuilder
                .create()
                .addExecInterceptorFirst("mainHandler", new CustomExecChainHandler(DefaultHttpRequestRetryStrategy.INSTANCE))
                .setRetryStrategy(new CustomRetryStrategy(1, TimeValue.ofSeconds(2)))
                .build();

        httpClient.execute(httpGet, response -> {
            System.out.println(response.getCode() + " " + response.getReasonPhrase());
            return null;
        });
    }

    @Slf4j
    static class CustomExecChainHandler implements ExecChainHandler {

        private final HttpRequestRetryStrategy retryStrategy;

        public CustomExecChainHandler(HttpRequestRetryStrategy retryStrategy) {
            this.retryStrategy = retryStrategy;
        }

        @Override
        public ClassicHttpResponse execute(
                final ClassicHttpRequest request,
                final ExecChain.Scope scope,
                final ExecChain chain) throws IOException, HttpException {
            Args.notNull(request, "request");
            Args.notNull(scope, "scope");
            final String exchangeId = scope.exchangeId;
            final HttpRoute route = scope.route;
            final HttpClientContext context = scope.clientContext;
            ClassicHttpRequest currentRequest = request;

            for (int execCount = 1; ; execCount++) {
                final ClassicHttpResponse response;
                try {
                    response = chain.proceed(currentRequest, scope);
                } catch (final Exception ex) {
                    if (scope.execRuntime.isExecutionAborted()) {
                        throw new RequestFailedException("Request aborted");
                    }
                    final HttpEntity requestEntity = request.getEntity();
                    if (requestEntity != null && !requestEntity.isRepeatable()) {
                        if (log.isDebugEnabled()) {
                            log.debug("{} cannot retry non-repeatable request", exchangeId);
                        }
                        throw ex;
                    }
                    if (retryStrategy.retryRequest(request, new IOException(ex), execCount, context)) {
                        if (log.isDebugEnabled()) {
                            log.debug("{} {}", exchangeId, ex.getMessage(), ex);
                        }
                        if (log.isInfoEnabled()) {
                            log.info("Recoverable I/O exception ({}) caught when processing request to {}",
                                    ex.getClass().getName(), route);
                        }
                        final TimeValue nextInterval = retryStrategy.getRetryInterval(request, new IOException(ex), execCount, context);
                        if (TimeValue.isPositive(nextInterval)) {
                            try {
                                if (log.isDebugEnabled()) {
                                    log.debug("{} wait for {}", exchangeId, nextInterval);
                                }
                                nextInterval.sleep();
                            } catch (final InterruptedException e) {
                                Thread.currentThread().interrupt();
                                throw new InterruptedIOException();
                            }
                        }
                        currentRequest = ClassicRequestBuilder.copy(scope.originalRequest).build();
                        continue;
                    } else {
                        if (ex instanceof NoHttpResponseException) {
                            final NoHttpResponseException updatedex = new NoHttpResponseException(
                                    route.getTargetHost().toHostString() + " failed to respond");
                            updatedex.setStackTrace(ex.getStackTrace());
                            throw updatedex;
                        }
                        //TODO instead of throwing the exception we can find something better.
                        throw ex;
                    }
                }

                try {
                    final HttpEntity entity = request.getEntity();
                    if (entity != null && !entity.isRepeatable()) {
                        if (log.isDebugEnabled()) {
                            log.debug("{} cannot retry non-repeatable request", exchangeId);
                        }
                        return response;
                    }
                    if (retryStrategy.retryRequest(response, execCount, context)) {
                        final TimeValue nextInterval = retryStrategy.getRetryInterval(response, execCount, context);
                        // Make sure the retry interval does not exceed the response timeout
                        if (TimeValue.isPositive(nextInterval)) {
                            final RequestConfig requestConfig = context.getRequestConfig();
                            final Timeout responseTimeout = requestConfig.getResponseTimeout();
                            if (responseTimeout != null && nextInterval.compareTo(responseTimeout) > 0) {
                                return response;
                            }
                        }
                        response.close();
                        if (TimeValue.isPositive(nextInterval)) {
                            try {
                                if (log.isDebugEnabled()) {
                                    log.debug("{} wait for {}", exchangeId, nextInterval);
                                }
                                nextInterval.sleep();
                            } catch (final InterruptedException e) {
                                Thread.currentThread().interrupt();
                                throw new InterruptedIOException();
                            }
                        }
                        currentRequest = ClassicRequestBuilder.copy(scope.originalRequest).build();
                    } else {
                        return response;
                    }
                } catch (final RuntimeException ex) {
                    response.close();
                    throw ex;
                }
            }
        }

    }

    @Slf4j
    static class CustomRetryStrategy implements HttpRequestRetryStrategy {
        private final int maxRetries;
        private final TimeValue retryInterval;

        public CustomRetryStrategy(final int maxRetries, final TimeValue retryInterval) {
            this.maxRetries = maxRetries;
            this.retryInterval = retryInterval;
        }

        @Override
        public boolean retryRequest(
                final HttpRequest request,
                final IOException exception,
                final int execCount,
                final HttpContext context) {
            Args.notNull(request, "request");
            Args.notNull(exception, "exception");

            System.out.println(execCount + " - Exception happened retrying " + exception.getMessage());

            // Do not retry if over max retries
            return execCount <= this.maxRetries;
        }

        @Override
        public boolean retryRequest(
                final HttpResponse response,
                final int execCount,
                final HttpContext context) {
            Args.notNull(response, "response");

            return execCount <= this.maxRetries;
        }

        @Override
        public TimeValue getRetryInterval(HttpResponse response, int execCount, HttpContext context) {
            System.out.println("Retrying HTTP request after " + retryInterval.toString());
            return retryInterval;
        }
    }
}

My local tests output looks like below

18:07:30: Executing ':Main.main()'...

> Task :compileJava
> Task :processResources UP-TO-DATE
> Task :classes

> Task :Main.main() FAILED
18:07:30,946 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 1.4.14
18:07:30,947 |-INFO in ch.qos.logback.classic.util.ContextInitializer@704921a5 - No custom configurators were discovered as a service.
18:07:30,947 |-INFO in ch.qos.logback.classic.util.ContextInitializer@704921a5 - Trying to configure with ch.qos.logback.classic.joran.SerializedModelConfigurator
18:07:30,950 |-INFO in ch.qos.logback.classic.util.ContextInitializer@704921a5 - Constructed configurator of type class ch.qos.logback.classic.joran.SerializedModelConfigurator
18:07:30,954 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.scmo]
18:07:30,955 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.scmo]
18:07:30,963 |-INFO in ch.qos.logback.classic.util.ContextInitializer@704921a5 - ch.qos.logback.classic.joran.SerializedModelConfigurator.configure() call lasted 5 milliseconds. ExecutionStatus=INVOKE_NEXT_IF_ANY
18:07:30,963 |-INFO in ch.qos.logback.classic.util.ContextInitializer@704921a5 - Trying to configure with ch.qos.logback.classic.util.DefaultJoranConfigurator
18:07:30,963 |-INFO in ch.qos.logback.classic.util.ContextInitializer@704921a5 - Constructed configurator of type class ch.qos.logback.classic.util.DefaultJoranConfigurator
18:07:30,964 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
18:07:30,967 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/C:/Users/ozkan/projects/java-examlpes/httpclient5-retry/build/resources/main/logback.xml]
18:07:31,071 |-WARN in ch.qos.logback.core.model.processor.ImplicitModelHandler - Ignoring unknown property [Appenders] in [ch.qos.logback.classic.LoggerContext]
18:07:31,071 |-WARN in ch.qos.logback.core.model.processor.ImplicitModelHandler - Ignoring unknown property [Loggers] in [ch.qos.logback.classic.LoggerContext]
18:07:31,072 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@df27fae - End of configuration.
18:07:31,072 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@24a35978 - Registering current configuration as safe fallback point
18:07:31,072 |-INFO in ch.qos.logback.classic.util.ContextInitializer@704921a5 - ch.qos.logback.classic.util.DefaultJoranConfigurator.configure() call lasted 109 milliseconds. ExecutionStatus=DO_NOT_INVOKE_NEXT_IF_ANY

1 - Exception happened retrying No such host is known (ht11tpbin.org)
1 - Exception happened retrying ht11tpbin.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.4/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
3 actionable tasks: 2 executed, 1 up-to-date
Exception in thread "main" java.net.UnknownHostException: ht11tpbin.org
    at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
    at org.apache.hc.client5.http.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
    at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:144)
    at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:450)
    at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:162)
    at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172)
    at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:142)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at io.github.ozkanpakdil.Main$CustomExecChainHandler.execute(Main.java:86)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
    at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:245)
    at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:188)
    at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:162)
    at io.github.ozkanpakdil.Main.unknownHostException(Main.java:40)
    at io.github.ozkanpakdil.Main.main(Main.java:27)

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':Main.main()'.
> Process 'command 'C:\Users\ozkan\.jdks\temurin-17.0.10\bin\java.exe'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

BUILD FAILED in 1s
18:07:31: Execution finished ':Main.main()'.

check here for full working example project.