AWS Glue Workflow triggers, randomly triggering jobs

Hello tmavrikis, is this still an issue? If so, hopefully some of the following recommendations and links can help answer your question. I'll break it down into the various questions and challenges you mentioned.

Workflow Support: I can assure you that AWS Glue workflows are fully supported by AWS. They are an integral part of AWS Glue and are designed to manage complex ETL activities. Recent Changes: It's important to note that even if you haven't made any changes, updates to the AWS Glue service itself could potentially impact workflow behavior.

To address the issues you're facing, let's consider a few potential causes and solutions:

Concurrency Limits:
    Review your workflow's concurrency settings. Ensure that the maximum number of concurrent workflow runs is set appropriately.
    Check the concurrency limits for individual jobs within the workflow.
    Consider adjusting these limits based on your expected workload.

Trigger Configuration:
    Review the configuration of your ANY and ALL triggers. Ensure they are set up correctly and are targeting the intended jobs.
    Check if there are any duplicate or conflicting trigger configurations.

Event Processing:
    If your workflows are triggered by events, ensure that there isn't a backlog of events causing multiple trigger activations.
    Review your event sources to see if there's been an unexpected increase in event frequency.

Logging and Monitoring:
    Enable detailed logging for your workflows and jobs.
    Use AWS CloudTrail to review API calls related to your Glue workflows.
    Set up CloudWatch alarms to alert you of unusual activity or exceeded thresholds.

Version Control:
    If you're using any form of version control or CI/CD for your Glue resources, check if there have been any recent deployments or changes.

Next Steps:

1. Implement the above checks and adjustments as appropriate. If the issue persists, I recommend opening a case with AWS Support. They can provide more in-depth, account-specific assistance and escalate to the Glue service team if necessary.

2. To verify if the problem has been resolved: Monitor your workflows closely after making any changes. Run test workflows with controlled inputs to ensure they behave as expected. Check that jobs are triggering in the correct order and only running once as intended.

3. If you're still unsure about the root cause, consider the following additional steps: Temporarily simplify your workflows to isolate the issue. Create a test workflow that mimics the problematic behavior to see if you can reproduce it consistently. Review any recent changes in your data sources or job scripts that might be impacting workflow execution.

Remember, when making any changes to your AWS Glue resources, always test in a non-production environment first to ensure the changes have the desired effect without disrupting your production workloads. If you need more specific guidance or if these steps don't resolve the issue, please open a case with AWS Support.

Resources: https://docs.aws.amazon.com/glue/latest/webapi/API_StartWorkflowRun.html

https://docs.aws.amazon.com/glue/latest/webapi/API_ResumeWorkflowRun.html

https://repost.aws/questions/QUSR3S3B-OSReZsp9xSPPuhg/glue-queue-max-concurrent-runs-exceeded