I face an issue where the SendTimeout value is not respected. The application have 2 main components:
- HTTP server which is deployed as regular verticle
- IO verticle that is deployed as worker verticle
The HTTP server verticle send event on the event bus that is consumed by the IO verticle. as part of the event emission we define SendTimeout to 200ms The IO verticle is consuming the events and for each event perform async http request to specific endpoint
Under heavy load (~18K requests per second) I see degradation which impact the HTTP server response time. Since I set the SendTimeout to 200ms I expected to see maximum response time of 200ms but this is not the case there are many requests that took over a second to get completed
Hardware & configurations:
- Machine with 16 vCPU
- HTTP server verticle is deployed 10 times
- IO verticle is deployed 16 times with a dedicated thread pool of 16 threads
I tried to replace the IO operation that was done by the IO verticle with sleep and in that case it seems like the timeout is repected. I also tried many different configurations of number of verticles / number of threads in the IO verticle thread pool - non of them helped
I'm expecting the sendTimeout to be respected and the maximum response time of the HTTP server should not exceed the defined sendTimeout value (200ms)