Questions tagged with Amazon Simple Workflow (Amazon SWF)

1)I have 150 ETL jobs need to be moved from on-premise to AWS cloud. What is the best way to schedule them in AWS? AWS batch or SWF or STEP function or any other service? Which tool/service has all the capability of a full time job scheduler? 2) As per AWS Batch documentation, it says that it supports all jobs that are Docker container. Does it mean that AWS batch is only meant for Docker container jobs? In my case all jobs are either bash/ksh scripts and normal ETL jobs that read from files from EC2 and load data into RDS DB.

I know there's a major outage going on in us-east-1 right now, per https://status.aws.amazon.com/. The notice at the top of the page says "Services impacted include: EC2, Connect, DynamoDB, Glue, Athena, Timestream, and Chime and other AWS Services in US-EAST-1." Per that page's status table (and this RSS feed: https://status.aws.amazon.com/rss/swf-us-east-1.rss ), SWF is currently working. However, we're seeing tons of 500s in our app logs, and none of our processes that use SWF seem to be making progress. Is SWF one of those "other impacted services"? I don't know of any successful requests going through, so I think it is, but an official confirmation would be nice. An edited example error from our PHP log follows: ``` Error: start_workflow_execution: exception 'AwsSwfExceptionSwfException' with message 'Error executing "StartWorkflowExecution" on "https://swf.us-east-1.amazonaws.com"; AWS HTTP error: Server error: `POST https://swf.us-east-1.amazonaws.com` resulted in a `500 Internal Server Error` response: {"__type":"com.amazon.coral.service#InternalFailure","message":"All attempts to invoke operation: StartWorkflowExecution (truncated...) ```

Hi, We currently have long running processes/jobs that are being executed via workflows. A workflow implementation calls the jobs to run. The call and the job runs asynchronously. By default, we have set the activity workflow to time-out after an hour. We recently saw an issue where if a job has been running for more than an hour, SWF throws an ActivityTaskTimedOutException in our workflow implementation. We expected that this should also cause the long running job to stop, but what happens is that only the workflow implementation catches this exception but job still continued running in the background. My question is, why did this not terminate the long running process? Is this an issue on how swf times-out processes, or do we need to have an implementation on our end that checks these time-out exceptions and terminate the process? If it is the latter, is there any guidance on how we should implement this? We badly need some help on this as we have been searching for a solution on this for some time now. Thank you!

Hello, We have hit the limit "Maximum workflow and activity types" Although we can still register activities in other domains, switching domains is too complicated in out case because the domain name is used in many places outside SWF :-) We use Java framework and use same version across all WFs and activities. So whenever we change activities, we bump up the version number that leads to hundreds of new activities to be registered. While we are waiting on the limit increase, what would be the recommended practice to minimize the number of activities registered? Thank you

Hello, I would like to wait on a list of promises, but the list is dynamically updated (items are added to) by a signal handler. I tried two approaches - @Wait in the function and waiting on AndPromise. In case of AndPromise - it is a member variable of WF class and gets recreated when a new promise is added to the list by signal handler. In both cases the function starts execution as soon as the first promise in the list is ready. How to fix it?

Hello We are designing a long-running workflow. We have a "parent" workflow that only does setting up and cleanup, the majority of work is done in child WF. Complete task may run for up to a year, so we thinking to implement child workflow to continue as continuous WF (https://docs.aws.amazon.com/amazonswf/latest/awsflowguide/continuous.html) continuing as new after completing each "chunk", but keep the parent WF running (setting timeout to one year as max allowed) We have tested the approach and it seems to work. The parent WF seems to be unaware of child restarting (which is exactly what we need) and history stays small. Is there any drawback to this approach? Anything we should consider? Thank you, Akbar

As there is 32K limit on activity result I see activities just timing out when result is larger while it is OK for normal cases, in case when activity is throwing an exception I cannot see the error reason. Specifically we are using SWF Java Flow Framework. Besides we use Hibernate with Spring. When data access layer throws exception, its text (including all the stack) is longer than 32K. In this case we see no activity failure, but just timeout. We could resolve it using custom serializer, but I would prefer a solution using fewer custom components. Is there such a solution?