Spring graceful shutdown not waiting for scheduled tasks with thread sleep

651 views Asked by At

I have a Spring Boot 3 application making use of @Scheduled jobs for polling messages from SQS. I want to configure a graceful shutdown (when I send a SIGTERM via kill pid) so the current process messages can be finished before the POD is tore down.

I set the following properties in application.properties file:

server.shutdown=graceful
spring.lifecycle.timeout-per-shutdown-phase=60s
spring.task.execution.shutdown.await-termination=true
spring.task.execution.shutdown.await-termination-period=60s
spring.task.scheduling.shutdown.await-termination=true
spring.task.scheduling.shutdown.await-termination-period=60s

And I have jobs scheduled like so:

@Scheduled(fixedDelay = 1L, timeUnit = TimeUnit.SECONDS, initialDelay = 3L)
public void pollMessages() {
    final var messages = sqsClient.receiveMessage(ReceiveMessageRequest.builder()
                .queueUrl(queueUrl)
                .waitTimeSeconds(maxWaitTime)
                .maxNumberOfMessages(numberOfMessages)
                .messageAttributeNames(".*")
                .build());
    // process messages....
}

However, I'm getting errors from AWS SDK when shutting down the application if there's any pending job polling for messages. It seems the shutdown process is interrupting Threads that are waiting a response from AWS SQS service.

Unexpected error occurred in scheduled task
s.a.a.c.e.SdkInterruptedException: null
    at s.a.a.c.i.h.InterruptMonitor.checkInterrupted(InterruptMonitor.java:55)
    at s.a.a.c.i.h.p.s.AfterTransmissionExecutionInterceptorsStage.execute(AfterTransmissionExecutionInterceptorsStage.java:34)
    at s.a.a.c.i.h.p.s.AfterTransmissionExecutionInterceptorsStage.execute(AfterTransmissionExecutionInterceptorsStage.java:28)
    at s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72)
    ... 54 common frames omitted
Wrapped by: s.a.a.c.e.AbortedException: Thread was interrupted
    at s.a.a.c.e.AbortedException$BuilderImpl.build(AbortedException.java:93)
    at s.a.a.c.e.AbortedException.create(AbortedException.java:38)
    at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.handleInterruptedException(ApiCallAttemptTimeoutTrackingStage.java:141)
    at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.translatePipelineException(ApiCallAttemptTimeoutTrackingStage.java:105)
    at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:89)
    at s.a.a.c.i.h.p.s.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
    at s.a.a.c.i.h.p.s.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
    at s.a.a.c.i.h.p.s.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
    at s.a.a.c.i.h.p.s.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:52)
    at s.a.a.c.i.h.p.s.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:37)
    at s.a.a.c.i.h.p.s.RetryableStage.execute(RetryableStage.java:81)
    at s.a.a.c.i.h.p.s.RetryableStage.execute(RetryableStag...

The same happens if I have a scheduled job with a Thread.sleep(), the graceful shutdown interrupts the sleeping thread. For example:

@Scheduled(fixedDelay = 1L, timeUnit = TimeUnit.SECONDS, initialDelay = 3L)
public void pollMessages() throws Exception {
    Thread.sleep(10000); // Graceful shutdown interrupts this thread during sleep
    // Extra processing steps omitted
}

Any ideas on how to fix this to make the Graceful Shutdown not to interrupt sleeping threads on Scheduled tasks?

Edit ------

I made a standalone Java application that does pretty much what I would expect spring to do here: https://github.com/schmittjoaopedro/schmittjoaopedro.github.io/blob/gh-pages/assets/other/SOQuestionGracefulShutdown.java

0

There are 0 answers