I have a Spring Boot 3 application making use of @Scheduled jobs for polling messages from SQS. I want to configure a graceful shutdown (when I send a SIGTERM via kill pid) so the current process messages can be finished before the POD is tore down.
I set the following properties in application.properties file:
server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=60s
spring.task.execution.shutdown.await-termination=true
spring.task.execution.shutdown.await-termination-period=60s
spring.task.scheduling.shutdown.await-termination=true
spring.task.scheduling.shutdown.await-termination-period=60s
And I have jobs scheduled like so:
@Scheduled(fixedDelay = 1L, timeUnit = TimeUnit.SECONDS, initialDelay = 3L)
public void pollMessages() {
final var messages = sqsClient.receiveMessage(ReceiveMessageRequest.builder()
.queueUrl(queueUrl)
.waitTimeSeconds(maxWaitTime)
.maxNumberOfMessages(numberOfMessages)
.messageAttributeNames(".*")
.build());
// process messages....
}
However, I'm getting errors from AWS SDK when shutting down the application if there's any pending job polling for messages. It seems the shutdown process is interrupting Threads that are waiting a response from AWS SQS service.
Unexpected error occurred in scheduled task
s.a.a.c.e.SdkInterruptedException: null
at s.a.a.c.i.h.InterruptMonitor.checkInterrupted(InterruptMonitor.java:55)
at s.a.a.c.i.h.p.s.AfterTransmissionExecutionInterceptorsStage.execute(AfterTransmissionExecutionInterceptorsStage.java:34)
at s.a.a.c.i.h.p.s.AfterTransmissionExecutionInterceptorsStage.execute(AfterTransmissionExecutionInterceptorsStage.java:28)
at s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72)
... 54 common frames omitted
Wrapped by: s.a.a.c.e.AbortedException: Thread was interrupted
at s.a.a.c.e.AbortedException$BuilderImpl.build(AbortedException.java:93)
at s.a.a.c.e.AbortedException.create(AbortedException.java:38)
at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.handleInterruptedException(ApiCallAttemptTimeoutTrackingStage.java:141)
at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.translatePipelineException(ApiCallAttemptTimeoutTrackingStage.java:105)
at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:89)
at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
at s.a.a.c.i.h.p.s.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
at s.a.a.c.i.h.p.s.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
at s.a.a.c.i.h.p.s.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:52)
at s.a.a.c.i.h.p.s.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:37)
at s.a.a.c.i.h.p.s.RetryableStage.execute(RetryableStage.java:81)
at s.a.a.c.i.h.p.s.RetryableStage.execute(RetryableStag...
The same happens if I have a scheduled job with a Thread.sleep(), the graceful shutdown interrupts the sleeping thread. For example:
@Scheduled(fixedDelay = 1L, timeUnit = TimeUnit.SECONDS, initialDelay = 3L)
public void pollMessages() throws Exception {
Thread.sleep(10000); // Graceful shutdown interrupts this thread during sleep
// Extra processing steps omitted
}
Any ideas on how to fix this to make the Graceful Shutdown not to interrupt sleeping threads on Scheduled tasks?
Edit ------
I made a standalone Java application that does pretty much what I would expect spring to do here: https://github.com/schmittjoaopedro/schmittjoaopedro.github.io/blob/gh-pages/assets/other/SOQuestionGracefulShutdown.java