Spring Quartz scheduled task stops running after some time

Question

I have a Quartz timer task running in a webapp. The task checks the database every 10 seconds for new data in a particular table.

<bean id="triggerCheckEvents" class="org.springframework.scheduling.quartz.SimpleTriggerBean">
<property name="jobDetail">
    <bean id="jobCheckEvents" class="org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean">
        <property name="targetObject" ref="checkEvents" />
        <property name="targetMethod" value="execute" />
        <property name="concurrent" value="false" />
    </bean>
</property>
<property name="startDelay" value="${checkEvent.startDelay}" />
<property name="repeatInterval" value="${checkEvent.repeatDelay}" />

About every second day or so, the timer stops running. I am somewhat sure that it gets stuck while doing something in the database. However, restarting the database does not seem to have effect on restoring the normal functioning of the timer. No exceptions get thrown. So there is not stack trace.

While I am trying to find the root cause. I think, I/O problems are bound to occur in a long running process. So the proper solution to this problem is to have some kind of watchdog that would, say, kill the thread after a timeout, spawn a new tread and continue the timer as per schedule.

Is there a way to terminate a scheduled task and create a new instance when the task fails to return within a specified time frame?

I am not trying to fix this particular error with the Database. Because tomorrow some other error may block the thread. I am looking for a high level solution that would just terminate the tread or do whatever is needed to free up resources blocked by the hung timer thread and create a fresh instance. Unexpected problems are inevitable in a long running process. The right solution is not to study and fix numerous individual problems as and when they show up, but to have a strategy to restore normal function even when errors occur.

Answer 1

I had a similar problem where a task used to get blocked. One approach is: at each invocation of the task, do actual stuff in new thread and make task's thread to wait for new thread to complete (till max allowed time is exceeded). This way task will never get blocked.

    public void work(JobExecutionContext ctx) {
    Thread t = new Thread(new Runnable() {
        public void run() {
            doSomethingInNewThread();
        }
    }, "ThreadName-" + System.currentTimeMillis());
    t.start();
    try {
        t.join(maxAllowedTime);
        if (t.isAlive()) {
            logger.warn("Tasks child thread "
                    + t.getName()
                    + "is still alive, potential thread block.");
            //TBD - if needed, add logic to terminate child thread
        }
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}