StepFunctions update list of objects into DynamoDB

0

I have a DynamoUpdateItem taskState in my state-machine in which I want to update a list of objects into the dynamodb table. The input to the step looks something like:

{
    "myList": [
        {
            "varA": "aaa",
            "varB": "bbb"
        },
        {
            "varA": "aaaa",
            "varB": "bbbb"
        },
        ...
    ]
}

I want to update myList as is into my DDB entry. Note that the size of myList for each execution is not static.

So far, i've tried using DynamoAttributeValue.listFromJsonPath but that doesn't work as SFN does not recognize the individual objects (map) within the list provided. Is there a way this requirement can be achieved without having to write a custom lambda function?

asked 5 months ago592 views
2 Answers
0

Yes, it is possible to update a list of objects in DynamoDB using Step Functions without writing a custom Lambda function.

The key is to use DynamoDB's list_append update expression to add the new list to the existing list attribute.

Here is an example state machine definition:

{
  "StartAt": "UpdateDynamoDB",
  "States": {
    "UpdateDynamoDB": {
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:updateItem",
      "Parameters": {
        "TableName": "myTable",
        "Key": {
          "id": { "S": "123" } 
        },
        "UpdateExpression": "SET myList = list_append(myList, :newList)",
        "ExpressionAttributeValues": {
          ":newList": {
            "L": [
              {
                "M": {
                  "varA": { "S": "aaa" },
                  "varB": { "S": "bbb" }
                }
              },
              {
                "M": {
                  "varA": { "S": "aaaa"}, 
                  "varB": { "S": "bbbb" }
                }
              }
            ]  
          }
        }
      },
      "End": true
    }
  }
}

The key points:

  • Use list_append to add new list to existing list
  • Pass new list as an ExpressionAttributeValue
  • Use DynamoDB's list data type for the attribute value

This will handle lists of any length.

AWS
Saad
answered 5 months ago
  • Thanks for the answer! As mentioned above, myList is of dynamic size (not always 2). Want to know if that can be achieved.

0

To accomplish this , you use a map state and do updates for any single item as an iteration

{
    "Type": "Map",
    "ItemsPath": "$.myList"
    "ItemProcessor": {
        "ProcessorConfig": {
            "Mode": "INLINE"
        },
        "StartAt": "Update",
        "States": {
            "Update": {
                "Type": "Task",
                 "Resource": "arn:aws:states:::dynamodb:updateItem",
                 "Parameters": {
                       "TableName": "TransferDataRecords-DDBTable-3I41R5L5EAGT",
                       "Key": {
                             "key": {"S.$": "$. varA"},
                         },
                        "UpdateExpression": "....",
                       ....
                 },
                "End": true
            }
        }
    },
    "End": true,
}

Being aware of Map Execution limits can be useful , if you have a lot of items so probably you get the event history error reaching 25K limit, in this case you can choose the distributed map to by pas that limit as well optimise your process latency with the help of native parallelism power.

Another option can be to using a distributed map and for any child workflow execution use a map this way you have a Paralelized operation and yet you benefit from Map Iterative functionality.

To summerize I recommend you to have a look at following references Map State: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-map-process-modes.html

Service Integrations: https://docs.aws.amazon.com/step-functions/latest/dg/connect-ddb.html

Hope this could be helpful

profile picture
answered 5 months ago
  • Thanks for the answer! Gave this a go and since i'm appending to a list in the database, this is leading to consistency issues where not all items are being updated in some cases. And this happens with all modes of map-run.

    Due to this, decided to go ahead with updating a list of string (serializing the objects).

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions