Activity workflow threw ActivityTaskTimedOut but process is still running

Question

Hi,  
  
We currently have long running processes/jobs that are being executed via workflows. A workflow implementation calls the jobs to run. The call and the job runs asynchronously. By default, we have set the activity workflow to time-out after an hour.   
  
We recently saw an issue where if a job has been running for more than an hour, SWF throws an ActivityTaskTimedOutException in our workflow implementation. We expected that this should also cause the long running job to stop, but what happens is that only the workflow implementation catches this exception but job still continued running in the background.  
  
My question is, why did this not terminate the long running process? Is this an issue on how swf times-out processes, or do we need to have an implementation on our end that checks these time-out exceptions and terminate the process? If it is the latter, is there any guidance on how we should implement this?  
  
We badly need some help on this as we have been searching for a solution on this for some time now.  
  
Thank you!

Accepted Answer

Java doesn't provide a method to terminate a thread that executes an activity. So the actual activity implementation is not stopped on the timeout. It is the responsibility of the implementation to ensure that timeouts are not exceeded.   
  
It is recommended to heartbeat from a long running activity to ensure timely timeout if the worker goes down. It also helps with the activity cleanup as the heartbeat method throws an exception if an activity is already timed out or was canceled. See "Activity Heartbeat" section from the   .

Answer

It is a Java language issue: https://www.baeldung.com/java-thread-stop

Answer

Hi Maxim, thanks for the response! Looks like we will have to do some implementation to fix this.   
  
I'm curious, by "Java doesn't provide a method to terminate..." did you mean that the underlying issue is within how Java language manages threads, or that the Java version for swf currently does not support terminating threads that executes an activity. Because if it is the latter, we might just have to wait and see if swf will have an update in the future that would support this

Answer

I see, thanks for this!

Activity workflow threw ActivityTaskTimedOut but process is still running

Relevant content