How to use Step Functions to forward messages to SQS with an unknown number of message attributes

Question

I need to be able to forward messages from an existing SQS queue to a new SQS queue using an Event Bridge Pipe and Step Functions.  Each message in the new queue must look identical to the message in the original queue, including the attributes.  This is a temporary configuration.  The messages will eventually go to lambdas instead of the new queue.

My problem is that the messages don't have consistent attributes.  They may have 0 - 9 attributes, in any combination.

The examples I found for including the attributes when sending a message from Step Functions look something like this:
```
     "Parameters": {
       "QueueUrl": "https://sqs..amazonaws.com/",
       "MessageBody.$": "$.body",
       "MessageAttributes": {
         "someAttribute": {
           "DataType.$": "$.messageAttributes.someAttribute.dataType",
           "StringValue.$": "$.messageAttributes.someAttribute.stringValue"
         },
         "someAttribute2": {
           "DataType.$": "$.messageAttributes.someAttribute2.dataType",
           "StringValue.$": "$.messageAttributes.someAttribute2.stringValue"
         }
```
This works, but if I specify all potential attributes and an incoming message is missing some, I get an error like this: The JSONPath... specified for the field... could not be found in the input

If I try to pass the attributes using the jsonPath like this:
```
"MessageAttributes.$":  "$.messageAttributes"
```
I get: An error occurred while executing the state 'SendMessage'. The Parameters... could not be used to start the Task: [The field "stringValue" is not supported by Step Functions]

I considered using Choice, but there are too many possible combinations of attributes.

Is there a simple way to include all of the attributes from the original json input when using SQS Send Message in Step Functions without spelling them out one at a time?

Mikel Del Tio · Accepted Answer

Hi,

The way you set the parameters via JSONPath looks fine (I attach an example below), so `MessageAttributes` should be equals to `messageAttributes`. Can you verify in the execution logs that this is the case?

**UPDATED**: 
After reading your response, I think it's because the attributes are case sensitive and you're sending `stringValue` instead of `StringValue`. Anyway, I have coded a Step Function that reads from queue A and leaves the messages as is in queue B. Hope this helps you.

**Step Function definition**
```
{
  "Comment": "Example",
  "StartAt": "Receive Message From Queue A",
  "States": {
    "Receive Message From Queue A": {
      "Type": "Task",
      "Parameters": {
        "QueueUrl": "REPLACE_WITH_YOUR_SQS_URL",
        "MessageAttributeNames": [
          "All"
        ]
      },
      "Resource": "arn:aws:states:::aws-sdk:sqs:receiveMessage",
      "Next": "Map"
    },
    "Map": {
      "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "INLINE"
        },
        "StartAt": "Send Message To Queue B",
        "States": {
          "Send Message To Queue B": {
            "Type": "Task",
            "Resource": "arn:aws:states:::sqs:sendMessage",
            "Parameters": {
              "QueueUrl": "REPLACE_WITH_YOUR_SQS_URL",
                "MessageBody.$": "$.Body",
                "MessageAttributes.$": "$.MessageAttributes"
            },
            "End": true
          }
        }
      },
      "ItemsPath": "$.Messages",
      "End": true
    }
  }
}
```

How to use Step Functions to forward messages to SQS with an unknown number of message attributes

Add your answer

Relevant content