We have a very simple application where we have one spring integration inbound jdbc poller which polls data from db table and put those records in direct channel. From that channel we have service activator which takes those records and put it in rabbitmq queue. Poller is configured to use user defined ThreadPoolTaskExecutor poll with min and max 1 thread and this is what we wanted.
This app runs fine for few days where it polls with fixed delay of 5 seconds. Also monitoring this thread looks like finishing its work in 200ms.
Then all of a sudden this polling thread stops polling for few minutes and this gap is sometimes 1min and sometimes 20 min. It is not consistent. If we restart the app then as soon as it restart it polls once and then again it goes back to incosistent poll.
We took multiple thread dumps via spring boot actuator and if we analyse this dumps then we are not seeing any deaklock. All the polling thread is showing on all thread dumps are WAITING state.
We use spring boot 2.2.6 with spring integration and app is deployed in PCF.
We have seen such behavior since spring boot upgrade from 2.1.4 to 2.2.6 but I cannot say this confidently.
Can someone provide some hints that what might be causing this strange behavior?
Here was the root cause. Spring boot adds single thread pool when you add @EnableScheduling annotation in your app. We also had spring-cloud-config-client as dependency and since we were using vault to store our secrets we had to renew the token, so we provided that token as property. This will add another auto configured bean to renew vault token. Now since there was single thread in the pool and we had network/vault perf issue, that thread was keep getting stuck to renew vault token and it was hardly getting freed up to do db polling. We were also using default renewal rate for vault which was adding to this problem. So two things helped the problem. We increased the ttl for vault token to 24 hrs and our system admin fixed network/vault perf issue.