Coordinating Step functions from App Flow -> Event Bridge -> DynamoDB

You did not show how your code looks like, but the item should be in the following format "AttributeName": {"BOOL": False|True}. Is that what you are doing?

I cannot post the source-code. But in pseudocode, here is what it is doing:

                // deserialize objects
                var jsonReader = new JsonTextReader(new StringReader(payload))
                {
                    SupportMultipleContent = true
                };

                JsonSerializerSettings jss = new JsonSerializerSettings();
                jss.ReferenceLoopHandling = ReferenceLoopHandling.Ignore;

               var jsonSerializer = new JsonSerializer();
               // MyObject is the model
               List<MyObject> objects = new List<MyObject>
                while (jsonReader.Read())
                {
                    var item = jsonSerializer.Deserialize<MyObjecct>(jsonReader);
                    objects.Add(doc);
                }
               return new Response() {
                   Data = objects
               };

The response gets serialized as JSON automatically in the step function. I tried changing the model to a DynamoDB Document type (a Dictionary<string,object>) which is what the DynamoDB SDK does, but that ends up returning the value of each item in the dictionary as an empty object when the Step Function serializer serializes the return value.

You need to use a Lambda to transform the output of the S3 GetItem because the Map step doesn't seem to understand how to deserialize JSONL, which the GetItem outputs.

It seems the only way to avoid problems with the Step Function serializer on the output of the Lambda is return a string that is the already JSON serialized object that has the DynamoDB type on each key in the JSON object (so it doesn't get broken by AWS's serializer), then deserialize it in the next step by using StringToJson. But that seems way too much overhead for something that should be straight-forward. It's less work to just have a lambda that directly writes to Dynamo using the AWS SDK, than to try and use the built-in DynamoDB PutItem step. This feels like it is defeating the purpose of step functions.

On top of that, trying to find documentation on what to do when you receive that error message is very difficult. Almost all AWS documentation for DynamoDB is on using the SDK directly, which abstracts the JSON format that DynamoDB expects.

I would also like to point out this seems to be a bug. If you have only strings, you don't need to add the type information - all strings work.

As soon as you have a number or a boolean, you need to add type information. However, the JSON format is limited to types allowed:

• Object
• Array
• String
• Number
• Boolean

I would expect that given the very limited allowed types in the JSON specification, you would be able to get away without providing type qualifiers to the JSON for flat JSON objects. The deserializer would be trivial:

{ "name": "foo", "count": 1, "isValid" : true } // could be easily tranformed to the following by a deserializer

{
  "name": {
     "S": "foo"
  },
  "count": {
    "N": "1"
  },
  "isValid": {
    "BOOL": true
  }
}

But I cannot change the deserializer, I'm stuck with either having everything a string, or jumping through hoops

Just a side observation, away from the main issue:

2. Event Bridge watching the S3 bucket

AppFlow is actually integrated with EventBridge, and each flow execution will emit event into EB in the default bus. More info: here.

I'd recommend you rather subscribe to this events, and kick-start the step function that way, than watching for S3 PutObject events.

AppFlow may produce several objects as part of a the ingest (subject to size of the data), and in this case, if you watch for PutObject on S3, you will invoke Step Function multiple times (for each object).

With the recommended approach, you can run a single SF execution and loop through all S3 objects (ListObjects) stored as part of the flow execution.

I hope this make sense.

Kind regards, Kamen

That is how it is configured - App Flow -> Event Bridge -> Step function invocation that gets passed the S3 objects.

The issue isn't how the objects are passed to the step functions, or how many there are. The problem is how do you take these objects and write them to DynamoDB. You must convert the JSONL that is the result of the Event Bridge event. The objects are passed to the Step Functions as JSONL (stream of JSON objects with no separator). In order to process those objects, you need to:

1. Convert JSONL to JSON Array (Can't use Dynamo batch write, since it is limited to 25 items per invocation)
2. Add Dynamo Type Information to each field in each object (because if you have any field that's not a String, it won't write).
3. Pass to Map step with a DynamoDB PutItem step inside

There is no "No Code" option for this flow. This is why I say it seems to defeat the purpose of the Step Function