We have configured Spring cloud Gateway to forward requests to downstream APIs. However, we have noticed that some requests takes around 5sec or more to send the requests to the downstream APIs, and this delay in sending requests is causing other services to not work properly as they have a timeout of 5 sec and if response not received in this timeout the service will respond with error.
We checked the servers and there are no loads on the resources. We tried increasing the -Xms and -Xmx values to improve performance but that also did not help. We also tried only to send POST requests for routing but we still see some GET request being sent. and not sure how we can stop that.
In the below logs there is a delay of around 10 seconds and at the end we recieved **java.io.IOException: An existing connection was forcibly closed by the remote host **
2024-03-12 01:21:12.174 DEBUG 7104 --- [tor-http-nio-13] reactor.netty.http.server.HttpServer : [5b39460c-1, L:/Local IP:8058 - R:/10.216.130.23:14454] Handler is being applied: org.springframework.http.server.reactive.ReactorHttpHandlerAdapter@2bd948e4
2024-03-12 01:21:12.174 DEBUG 7104 --- [tor-http-nio-13] o.s.w.s.adapter.HttpWebHandlerAdapter : [5b39460c-2530] HTTP POST "/wallet/v1/wallet/qr/payment/auth"
2024-03-12 01:21:12.174 INFO 7104 --- [tor-http-nio-13] c.d.epiCloudproxy.LoggingFilter : Routing Incoming request orginal url Local API Endpoint route to url: api endpoint,
2024-03-12 01:21:12.174 DEBUG 7104 --- [ctor-http-nio-1] r.n.resources.PooledConnectionProvider : [c5448fc1, L:/Local IP:54766 - R:Remote IP:9480] Channel acquired, now: 1 active connections, 0 inactive connections and 0 pending acquire requests.
2024-03-12 01:21:12.174 DEBUG 7104 --- [ctor-http-nio-1] r.netty.http.client.HttpClientConnect : [c5448fc1-2, L:/Local IP:54766 - R:Remote IP:9480] Handler is being applied: {uri=api endpoint, method=POST}
2024-03-12 01:21:12.174 DEBUG 7104 --- [ctor-http-nio-1] r.n.r.DefaultPooledConnectionProvider : [c5448fc1-2, L:/Local IP:54766 - R:Remote IP:9480] onStateChange(POST{uri=/v1/wallet/qr/payment/auth, connection=PooledConnection{channel=[id: 0xc5448fc1, L:/Local IP:54766 - R:Remote IP:9480]}}, [request_prepared])
2024-03-12 01:21:12.174 DEBUG 7104 --- [tor-http-nio-13] reactor.netty.channel.FluxReceive : [5b39460c-1, L:/Local IP:8058 - R:/10.216.130.23:14454] [terminated=true, cancelled=false, pending=1, error=null]: subscribing inbound receiver
2024-03-12 01:21:12.174 DEBUG 7104 --- [ctor-http-nio-1] r.n.r.DefaultPooledConnectionProvider : [c5448fc1-2, L:/Local IP:54766 - R:Remote IP:9480] onStateChange(POST{uri=/v1/wallet/qr/payment/auth, connection=PooledConnection{channel=[id: 0xc5448fc1, L:/Local IP:54766 - R:Remote IP:9480]}}, [request_sent])
2024-03-12 01:21:21.940 DEBUG 7104 --- [tor-http-nio-14] reactor.netty.tcp.SslProvider : [0510b3b7, L:/Local IP:8058 - R:/10.216.130.22:55058] SSL enabled using engine 410137f5[SSLEngine[hostname=null port=-1] SSL_NULL_WITH_NULL_NULL]
2024-03-12 01:21:21.940 DEBUG 7104 --- [tor-http-nio-14] r.n.http.server.HttpServerOperations : [0510b3b7, L:/Local IP:8058 - R:/10.216.130.22:55058] New http connection, requesting read
2024-03-12 01:21:21.940 DEBUG 7104 --- [tor-http-nio-14] reactor.netty.transport.TransportConfig : [0510b3b7, L:/Local IP:8058 - R:/10.216.130.22:55058] Initialized pipeline DefaultChannelPipeline{(reactor.left.sslHandler = io.netty.handler.ssl.SslHandler), (reactor.left.sslReader = reactor.netty.tcp.SslProvider$SslReadHandler), (reactor.left.httpCodec = io.netty.handler.codec.http.HttpServerCodec), (reactor.left.httpTrafficHandler = reactor.netty.http.server.HttpTrafficHandler), (reactor.right.reactiveBridge = reactor.netty.channel.ChannelOperationsHandler)}
2024-03-12 01:21:21.940 DEBUG 7104 --- [tor-http-nio-14] r.n.http.server.HttpServerOperations : [0510b3b7, L:/Local IP:8058 - R:/10.216.130.22:55058] Increasing pending responses, now 1
2024-03-12 01:21:21.940 DEBUG 7104 --- [tor-http-nio-14] reactor.netty.http.server.HttpServer : [0510b3b7-1, L:/Local IP:8058 - R:/10.216.130.22:55058] Handler is being applied: org.springframework.http.server.reactive.ReactorHttpHandlerAdapter@2bd948e4
2024-03-12 01:21:21.940 DEBUG 7104 --- [tor-http-nio-14] o.s.w.s.adapter.HttpWebHandlerAdapter : [0510b3b7-2531] HTTP GET "/"
2024-03-12 01:21:21.940 DEBUG 7104 --- [tor-http-nio-14] o.s.w.r.handler.SimpleUrlHandlerMapping : [0510b3b7-2531] Mapped to ResourceWebHandler [classpath [META-INF/resources/], classpath [resources/], classpath [static/], classpath [public/]]
2024-03-12 01:21:21.940 DEBUG 7104 --- [tor-http-nio-14] o.s.w.r.resource.ResourceWebHandler : [0510b3b7-2531] Resource not found
2024-03-12 01:21:21.940 DEBUG 7104 --- [tor-http-nio-14] org.springframework.web.HttpLogging : [0510b3b7-2531] Resolved [ResponseStatusException: 404 NOT_FOUND] for HTTP GET /
2024-03-12 01:21:21.940 DEBUG 7104 --- [tor-http-nio-14] org.springframework.web.HttpLogging : [0510b3b7-2531] Encoding [{timestamp=Tue Mar 12 01:21:21 GST 2024, path=/, status=404, error=Not Found, message=null, requestI (truncated)...]
2024-03-12 01:21:21.940 DEBUG 7104 --- [tor-http-nio-14] r.n.http.server.HttpServerOperations : [0510b3b7-1, L:/Local IP:8058 - R:/10.216.130.22:55058] Detected non persistent http connection, preparing to close
2024-03-12 01:21:21.940 DEBUG 7104 --- [tor-http-nio-14] r.n.http.server.HttpServerOperations : [0510b3b7-1, L:/Local IP:8058 - R:/10.216.130.22:55058] Last HTTP packet was sent, terminating the channel
2024-03-12 01:21:21.940 DEBUG 7104 --- [tor-http-nio-14] reactor.netty.channel.ChannelOperations : [0510b3b7-1, L:/Local IP:8058 - R:/10.216.130.22:55058] [HttpServer] Channel inbound receiver cancelled (operation cancelled).
2024-03-12 01:21:21.940 DEBUG 7104 --- [tor-http-nio-14] o.s.w.s.adapter.HttpWebHandlerAdapter : [0510b3b7-2531] Completed 404 NOT_FOUND
2024-03-12 01:21:31.128 WARN 7104 --- [ctor-http-nio-1] r.netty.http.client.HttpClientConnect : [c5448fc1-2, L:/Local IP:54766 - R:Remote IP:9480] The connection observed an error
EDIT: Application.yml configuration below
server:
ssl:
key-store: cert.p12
key-store-password: pw
key-alias: alias
key-store-type: PKCS12
port: 8058
logging:
level:
reactor:
netty: DEBUG
org:
springframework:
web: DEBUG
spring:
cloud:
gateway:
httpclient:
proxy:
port: 9480
host: cloudproxy
routes:
- id: test-route
uri: Remote_uri
predicates:
- Path=/reqbin/**
filters:
- RewritePath=/reqbin/(?<path>.*), /$\{path}
- id: fill_go_routes
uri: Remote_uri
predicates:
- Path=/fill-go/**
filters:
- RewritePath=/fill-go/(?<path>.*), /$\{path}
- id: qr_wallet_routes
uri: Remote_uri
predicates:
- Path=/wallet/**
filters:
- RewritePath=/wallet/(?<path>.*), /$\{path}