Failsafe with RetryPolicy and CircuitBreaker throws CircuitBreakerOpenException

Question

I am trying to combine a retry policy with the CircuitBreaker pattern with Failsafe but I get a CircuitBreakerOpenException exception when an attempt is made with the circuit open and it is interrupted.

https://github.com/jhalterman/failsafe

The problem is generated by setting a delay for retries less than the circuit's closing time.

How can I control this exception so that the retry policy is not interrupted? I want to do this because I can have several simultaneous instances launched requests to a rest service and that retries are not interrupted.

My code:

public class UnstableApplication {
    private final int MAX_FAILS = 4;
    private AtomicInteger failCount = new AtomicInteger(1);

    public String generateId() throws Exception {
        if (failCount.getAndIncrement() < MAX_FAILS) {
            System.err.printf("UnstableApplication throws SampleException at '%s'\n", ZonedDateTime.now());
            throw new Exception();
        }

        final String id = UUID.randomUUID().toString();
        System.out.printf("UnstableApplication: id '%s' generated at '%s'\n", id, ZonedDateTime.now());

        return id;
    }

}

public class FailsafeExample {

    public static void main(String[] args) throws Exception {

        UnstableApplication app = new UnstableApplication();

        RetryPolicy retryPolicy = new RetryPolicy()
                .retryOn(Exception.class)
                .withDelay(2, TimeUnit.SECONDS)
                .withMaxRetries(5);

        CircuitBreaker breaker = new CircuitBreaker();
        breaker.withFailureThreshold(2);
        breaker.withDelay(5, TimeUnit.SECONDS);
        breaker.withSuccessThreshold(3);
        breaker.onOpen(() -> {
            System.out.println("Circuit breaker is open");
        });

        breaker.onClose(() -> {
            System.out.println("Circuit breaker is close");
        });

        breaker.onHalfOpen(() -> {
            System.out.println("Circuit breaker is half-close");
        }); 

        Failsafe.with(retryPolicy)
        .with(breaker)
        .onFailedAttempt((a, b) -> {
            System.out.println(
                    String.format("Failed with exception: %s, at %s, circuit-breaker state is: %s", 
                            b, ZonedDateTime.now(), breaker.getState()));
        })
        .onSuccess(cxn -> {
            System.out.println("Succcess!");
        })
        .onFailure(cxn -> {
            System.out.println("Failed!");
        })
        .get(new Callable<String>() {
            @Override
            public String call() throws Exception {
                return app.generateId();
            }
        });
    }
}

My result:

UnstableApplication throws SampleException at '2019-05-31T16:30:09.214Z[Etc/UTC]'
Failed with exception: java.lang.Exception, at 2019-05-31T16:30:09.221Z[Etc/UTC], circuit-breaker state is: CLOSED
UnstableApplication throws SampleException at '2019-05-31T16:30:11.229Z[Etc/UTC]'
Circuit breaker is open
Failed with exception: java.lang.Exception, at 2019-05-31T16:30:11.230Z[Etc/UTC], circuit-breaker state is: OPEN
Exception in thread "main" net.jodah.failsafe.CircuitBreakerOpenException
    at net.jodah.failsafe.SyncFailsafe.call(SyncFailsafe.java:136)
    at net.jodah.failsafe.SyncFailsafe.get(SyncFailsafe.java:56)
    at com.kash.test.Foo.main(Foo.java:63)

Answer 1

This is the intended behavior. When the circuit is open, requests to that endpoint are stopped--if the dependency is broken or unreliable, of course your retries also have to be cut off, otherwise you'll experience a pileup of retries just as easily as if you didn't have a cb at all.

If you want to continue sending requests, either your trigger for breaking the circuit is too sensitive, or you're in a situation that doesn't call for circuit breaking. More than likely, the behavior that failsafe is enforcing is the correct approach.

If you really want to do this, put your retry logic inside the function you send to the circuitbreaker instead of using the RetryPolicy from failsafe. Or use an exponential backoff function in the retry policy.

Answer 2

Here is my understanding when I tried combining Circuit Breaker with Retry Policy in this example case:

CircuitBreaker breaker = new CircuitBreaker()
                .withFailureThreshold(3, 10)
                .withSuccessThreshold(5)
                .failOn(Exception.class)
                .withDelay(30, TimeUnit.SECONDS);

RetryPolicy retryPolicy = new RetryPolicy()
                .retryOn(ConnectException.class)
                .withDelay(3, TimeUnit.SECONDS)
                .withMaxRetries(5);

• First, it will Retry it 5 times, but since the threshold for Circuit Breaker is 3, thus at 3rd'continuous' retry attempt, it will break and open the circuit.
• Thus, once open, it 'might still retry' probably, but since the circuit is already open, a retry still under the time delay, would result a CircuitBreakerOpenException.