Skip to content

Invoking AWS Glue Workflows from Step Functions

0

Invoking a Glue Workflow from Step Functions got the following error when deploying Cloudformation:

Resource handler returned message: "Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: The resource provided arn:aws:states:::glue:startWorkflowRun is not recognized. 
The value is not a valid resource ARN, or the resource is not available in this region.
at /States/StartGlueWorkflow/Resource' (Service: AWSStepFunctions; Status Code: 400; Error Code: InvalidDefinition;..., HandlerErrorCode: InvalidRequest)

State machine task for starting Glue Workflow.

            "StartGlueWorkflow": {
              "Type": "Task",
              "Resource": "arn:aws:states:::glue:startWorkflowRun",
              "Parameters": {
                "Name": "glue-test"
              },
              "End": true
            },

Glue workflow ("glue-test") exists in the same region, and has a trigger on-demand.

Role for Step Functions state machine with Glue Workflow policies:

  RoleStmaMain:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: !Sub 'states.${AWS::Region}.amazonaws.com'
            Action: 'sts:AssumeRole'
      Policies:
        - PolicyName: role-stma-policy
          PolicyDocument:
            Statement:
              ...
              ...
              - Effect: Allow
                Action:
                  - "glue:StartWorkflowRun"
                  - "glue:GetWorkflowRun"
                  - "glue:GetWorkflow"
                  - "glue:BatchGetJobs"
                  - "glue:BatchGetCrawlers"
                Resource:
                  - "*"

Notes: - Step Functions: Standard - Region: eu-central-1

Can someone shine a light? Is the ARN correct (arn:aws:states:::glue:startWorkflowRun)?

Does Step Functions allow invocation of Glue Workflows? Most examples show Step Functions integration with a Glue Job and not a Glue Workflow.

2 Answers
2
Accepted Answer

The arn you are using for the resource type itself looks slightly wrong. I believe it should be arn:aws:states:::aws-sdk:glue:startWorkflowRun notice "aws-sdk".

AWS
answered a year ago
EXPERT
reviewed 10 months ago
  • Awesome! I confirm it works by adding the aws-sdk, very good, thanks!

0

The error suggests that the integration between Glue Workflows and Step Functions may not be available in eu-central-1 region. Try in a different region.

AWS
EXPERT
answered a year ago
  • I had the wrong ARN, it seems it requires aws-sdk, thanks

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.