Skip to content

Inconsistent JSONata variable scoping in Distributed Map states

0

As the docs explain, states inside a Distributed Map do not inherit JSONata variables from the parent scope containing that Dist Map. This is easy enough to understand, even if cumbersome to work around (must pass global info as part of Map input items).

However, trying to reassign a variable name inside the Dist Map triggers the (unjustified, IMHO) error "The variable name '...' was already defined in a parent scope".

This seems to be a contradiction in scoping rules. If a variable from the parent scope cannot be accessed in a child scope, then redefinition of its name should be allowed in that child scope. The way it is now contradicts common sense.

Example code below:
  • as is, produces the error "The variable name 's3_bucket ' was already defined in a parent scope"
  • If the line marked "HERE1" is removed, it will fail because the parent $s3_bucket cannot be accessed.
  • If in "HERE1" and "HERE2" we replace "s3_bucket" with, for example, "s3_bucket_inner", everything works.
QueryLanguage: JSONata
StartAt: Init
States:
  Init:
    Type: Pass
    Assign:  # parent scope variables
      s3_bucket: my-s3-bucket
    Output: ['my/prefix_1/', 'my/prefix_2/', 'my/prefix_3/']  # items for Map
    Next: DistMap

  DistMap:
    Type: Map
    ItemSelector:
      s3_bucket: '{% $s3_bucket %}'    # read parent scope variable and pass it as input to inner scope
      s3_prefix: '{% $states.context.Map.Item.Value %}'
    ItemProcessor:
      ProcessorConfig: {Mode: DISTRIBUTED, ExecutionType: EXPRESS}
      StartAt: InnerInit
      States:
        InnerInit:
          Type: Pass
          Assign:   # make item data available to all inner states, without having to pass it in state I/O
            s3_bucket: '{% $states.input.s3_bucket %}'      # define child scope variable  <--- HERE1
            s3_prefix: '{% $states.input.s3_prefix %}'
          Output: null
          Next: ListObjects

        ListObjects:
          Comment: example, one of many states that use "s3_bucket" and "s3_prefix"
          Type: Task
          Resource: arn:aws:states:::aws-sdk:s3:listObjectsV2
          Arguments:
            Bucket: '{% $s3_bucket %}'    # use child scope variable   <--- HERE2
            Prefix: '{% $s3_prefix %}'
          End: true
    End: true

asked 2 years ago639 views

2 Answers
0
Accepted Answer

The documentation says: Exception: Distributed Map states cannot currently reference variables in outer scopes.

I am guessing that we plan to support out scope variables in the future also for distributed map state, and for that, to be future proof, we don;t allow the same names also now.

AWS
EXPERT

answered 2 years ago

EXPERT

reviewed 2 years ago

  • Thanks @Uri So if I understand correctly, the future intent is:

    • outer scope variables will be readable in the inner scope
    • variable names that exist in the outer scope cannot be reassigned in the inner scope (i.e. no "local shadowing") is this correct?
0

The documentation mentions that inner scope variables cannot use the same name as one defined in the outer scope: https://docs.aws.amazon.com/step-functions/latest/dg/workflow-variables.html#variable-scope

To help avoid common errors, a variable assigned in an inner scope cannot have the same name as one assigned in an outer scope

And in Map states, you do have access to outer scope variables:

Parallel and Map states have their own scope, but can access variables in outer scopes.~~

Update: Please see the update from Uri regarding the exception for Distributed Map. And please note that you can currently use ItemSelector to pass additional data from the parent execution into the input of the child executions

AWS
EXPERT

answered 2 years ago

EXPERT

reviewed 2 years ago

  • Thanks @ben-from-aws My bad for not reading the docs thoroughly enough.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.