This is what I'm actually doing. table.get(getOp) makes a read call to Hbase which has a timeout set to 300ms.
return resilenceDecorator.execute(DIMENSION_READ_SERVICE, () -> {
Get getOp = getFetchOp(dimensionReadDBRequest);
try (Table table = getTable()) {
Result result = table.get(getOp);
return readResult(result, dimensionReadDBRequest);
} catch (IOException e) {
throw new RuntimeException(e);
}
}
This is the execute function where the supplier passed is specified in the above code block.
public <T> T execute(String serviceName, Supplier<T> supplier) throws DecoratorExecutionException {
com.core.config.resilience.TimeLimiterConfig timeLimiterConfig = getConfig(serviceName).getTimeLimiterConfig();
Supplier<T> decorateSupplier = Decorators.ofSupplier(supplier)
.withCircuitBreaker(getCircuitBreaker(serviceName))
.withBulkhead(getBulkhead(serviceName))
.withRateLimiter(getRateLimiter(serviceName))
.decorate();
}
try {
return CompletableFuture.supplyAsync(() -> Try.ofSupplier(decorateSupplier).get()).get(timeLimiterConfig.getTimeoutDuration(), TimeUnit.MILLISECONDS);
}
catch (Exception e) {
log.error("Exception caught during CompletableFuture completion", e);
throw new DecoratorExecutionException(e);
}
}
This specific code block is timing out intermittently. The timelimiter config has been set to 5s for debugging and its waiting for complete 5s even though Hbase calls would timeout within 300ms.
I have metrics setup inside and outside the resilience block. There is a very clear difference among them so Hbase most likely doesn't seem like an issue. I need help in figuring out what might be wrong in the execute method that is taking too long.
I was expecting the execute method to not wait for too long since Hbase has a way lesser timeout. I have verified the other circuitbreaker, ratelimiter and bulkhead configs and made sure they are not getting hit. This has been verified using both logs metrics and logs.
So I'm not sure why timelimiter is waiting for so long