I'm getting the below error on a set of microservices which are called in sequence, S1 is calling S2 which calls S3 in parallel using Mutiny.
"org.jboss.resteasy.spi.UnhandledException: jakarta.ws.rs.ProcessingException: RESTEASY004655: Unable to invoke request: org.apache.http.NoHttpResponseException: localhost:8090 failed to respond\r\n\tat org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:107)\r\n\tat org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:344)\r\n\tat org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:205)\r\n\tat org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:452)\r\n\tat org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:240)\r\n\tat org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:154)\r\n\tat org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:321)\r\n\tat org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:157)\r\n\tat org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:229)\r\n\tat io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:82)\r\n\tat io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:147)\r\n\tat io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:93)\r\n\tat io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:582)\r\n\tat org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)\r\n\tat org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)\r\n\tat org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)\r\n\tat org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)\r\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\r\n\tat java.base/java.lang.Thread.run(Thread.java:840)\r\nCaused by: jakarta.ws.rs.ProcessingException: RESTEASY004655: Unable to invoke request: org.apache.http.NoHttpResponseException: localhost:8090 failed to respond\r\n\tat org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.invoke(ManualClosingApacheHttpClient43Engine.java:361)\r\n\tat org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:427)\r\n\tat org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:134)\r\n\tat org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:103)\r\n\tat org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:61)\r\n\tat jdk.proxy26/jdk.proxy26.$Proxy237.validateTrades(Unknown Source)\r\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\r\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:568)\r\n\tat io.quarkus.restclient.runtime.QuarkusInvocationContextImpl.interceptorChainCompleted(QuarkusInvocationContextImpl.java:77)\r\n\tat io.quarkus.restclient.runtime.QuarkusInvocationContextImpl.proceed(QuarkusInvocationContextImpl.java:106)\r\n\tat io.smallrye.faulttolerance.FaultToleranceInterceptor.lambda$syncFlow$3(FaultToleranceInterceptor.java:253)\r\n\tat io.smallrye.faulttolerance.core.InvocationContext.call(InvocationContext.java:20)\r\n\tat io.smallrye.faulttolerance.core.Invocation.apply(Invocation.java:29)\r\n\tat io.smallrye.faulttolerance.core.timeout.Timeout.doApply(Timeout.java:55)\r\n\tat io.smallrye.faulttolerance.core.timeout.Timeout.apply(Timeout.java:30)\r\n\tat io.smallrye.faulttolerance.core.metrics.MetricsCollector.doApply(MetricsCollector.java:101)\r\n\tat io.smallrye.faulttolerance.core.metrics.MetricsCollector.apply(MetricsCollector.java:91)\r\n\tat io.smallrye.faulttolerance.FaultToleranceInterceptor.syncFlow(FaultToleranceInterceptor.java:255)\r\n\tat io.smallrye.faulttolerance.FaultToleranceInterceptor.intercept(FaultToleranceInterceptor.java:182)\r\n\tat io.smallrye.faulttolerance.FaultToleranceInterceptor_Bean.intercept(Unknown Source)\r\n\tat io.quarkus.restclient.runtime.QuarkusInvocationContextImpl$InterceptorInvocation.invoke(QuarkusInvocationContextImpl.java:195)\r\n\tat io.quarkus.restclient.runtime.QuarkusInvocationContextImpl.invokeNext(QuarkusInvocationContextImpl.java:68)\r\n\tat io.quarkus.restclient.runtime.QuarkusInvocationContextImpl.proceed(QuarkusInvocationContextImpl.java:104)\r\n\tat io.quarkus.restclient.runtime.QuarkusProxyInvocationHandler.invoke(QuarkusProxyInvocationHandler.java:162)\r\n\tat jdk.proxy26/jdk.proxy26.$Proxy238.validateTrades(Unknown Source)\r\n\tat se.sebank.portfoliomgt.compassweb.bff.client.tradingservice.TradeClient_xs7g2lhxeB6gTml8DI7qYjJuWlc_Synthetic_ClientProxy.validateTrades(Unknown Source)\r\n\tat se.sebank.portfoliomgt.compassweb.bff.view.order.management.domain.OrderService.validateOrders(OrderService.java:140)\r\n\tat se.sebank.portfoliomgt.compassweb.bff.view.order.management.domain.OrderService_ClientProxy.validateOrders(Unknown Source)\r\n\tat se.sebank.portfoliomgt.compassweb.bff.view.order.management.boundary.OrderResource.validateOrder(OrderResource.java:55)\r\n\tat se.sebank.portfoliomgt.compassweb.bff.view.order.management.boundary.OrderResource_Subclass.validateOrder$$superforward(Unknown Source)\r\n\tat se.sebank.portfoliomgt.compassweb.bff.view.order.management.boundary.OrderResource_Subclass$$function$$45.apply(Unknown Source)\r\n\tat io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)\r\n\tat io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)\r\n\tat io.quarkus.resteasy.runtime.QuarkusRestPathTemplateInterceptor.restMethodInvoke(QuarkusRestPathTemplateInterceptor.java:39)\r\n\tat io.quarkus.resteasy.runtime.QuarkusRestPathTemplateInterceptor_Bean.intercept(Unknown Source)\r\n\tat io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)\r\n\tat io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)\r\n\tat io.quarkus.arc.impl.AroundInvokeInvocationContext$NextAroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:97)\r\n\tat io.quarkus.security.runtime.interceptor.SecurityHandler.handle(SecurityHandler.java:27)\r\n\tat io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor.intercept(RolesAllowedInterceptor.java:29)\r\n\tat io.quarkus.security.runtime.interceptor.RolesAllowedInterceptor_Bean.intercept(Unknown Source)\r\n\tat io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)\r\n\tat io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:70)\r\n\tat io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)\r\n\tat io.quarkus.resteasy.runtime.StandardSecurityCheckInterceptor.intercept(StandardSecurityCheckInterceptor.java:45)\r\n\tat io.quarkus.resteasy.runtime.StandardSecurityCheckInterceptor_RolesAllowedInterceptor_Bean.intercept(Unknown Source)\r\n\tat io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)\r\n\tat io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)\r\n\tat io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)\r\n\tat se.sebank.portfoliomgt.compassweb.bff.view.order.management.boundary.OrderResource_Subclass.validateOrder(Unknown Source)\r\n\tat se.sebank.portfoliomgt.compassweb.bff.view.order.management.boundary.OrderResource_ClientProxy.validateOrder(Unknown Source)\r\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)\r\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:568)\r\n\tat org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:154)\r\n\tat org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:118)\r\n\tat org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:560)\r\n\tat org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:452)\r\n\tat org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:413)\r\n\tat org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:321)\r\n\tat org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:415)\r\n\tat org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:378)\r\n\tat org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:356)\r\n\tat org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:70)\r\n\tat org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:429)\r\n\t... 15 more\r\nCaused by: org.apache.http.NoHttpResponseException: localhost:8090 failed to respond\r\n\tat org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)\r\n\tat org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)\r\n\tat org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)\r\n\tat org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)\r\n\tat org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)\r\n\tat org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)\r\n\tat org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)\r\n\tat org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)\r\n\tat org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)\r\n\tat org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)\r\n\tat org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)\r\n\tat org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)\r\n\tat org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)\r\n\tat org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)\r\n\tat org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.invoke(ManualClosingApacheHttpClient43Engine.java:344)\r\n\t... 82 more"
S1 is only waiting roughly 10-14 seconds for S2 to respond, we've tried adding the faul-tolerance tag below to the rest-client in S1:
@Timeout(value = 30, unit = ChronoUnit.SECONDS)
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
public interface TradeClient {
S2 runs calls to S3 in parallell as follows:
public List<Trade> validateTrades(final List<TradeRequest<TradeRequestDetails>> sendTrades) {
final List<Uni<Trade>> unis = sendTrades.stream().map(trade ->
Uni.createFrom().item(() -> {
String callerThreadName = Thread.currentThread().getName();
LocalDateTime creationTime = LocalDateTime.now();
log.infof("Dispatching call from thread: " + callerThreadName + " at " + creationTime);
// Make the call and build the DTO within the lambda to ensure the logging happens just before the call
return validateTrade(trade);
})
.runSubscriptionOn(managedExecutor)).toList();
final var responses = Uni.join().all(unis)
.andCollectFailures()
.onFailure().retry().atMost(2)
.onFailure().transform((e) -> handleFailure(e))
.runSubscriptionOn(managedExecutor).await()
.atMost(Duration.ofMinutes(2));
return responses;
}