Unable to correctly configure InputTemplate json for Eventbridge PutTarget operation

0

Hi there,

I'm working on dynamically setting ECS container task environment variables by extracting values from the event that triggers the task via EventBridge.

I'm able to extract individual values from the Event detail attribute and pass that to the container, but I keep getting errors when trying to pass along the entire detail object.

I've set up a deadletter queue to help debug the eventbridge rule and target execution, but haven't yet found anything actionable. I've also tried this question (https://repost.aws/questions/QUYyKFM5vWQMmIECUXsx11Jw/unable-to-pass-aws-events-event-json-in-input-transformer-template) and this documentation (https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-transform-target-input.html#eb-transform-input-issues) to verify what I'm trying to accomplish but have had no luck as of yet.

I'm using the Python Boto3 library to add the target for my eventbridge rule.

template = '{{"containerOverrides": [{{"name": "{}", "environment": [{{"name": "PAYLOAD", "value": "TEST VALUE"}}, {{"name": "CLIENT", "value": <client>}}, {{"name": "DETAIL", "value": <detail>}}]}}]}}'.format(container_name)

config = {
    'Id': target_id,
    'Arn': ecs_cluster_arn,
    'RoleArn': eventbridge_role_arn,
    'InputTransformer': {
        'InputPathsMap': {
            'client': '$.detail.CLIENT',
            'detail': '$.detail'
        },
        'InputTemplate': template
    },
    ... # commented out for brevity
}

response = self.__client.put_targets(
    Rule=rule_name,
    Targets=[config]
)

I don't run into any issues when creating the rule target; however, when attempting to test the rule with a PutEvents command from the AWS CLI, the task fails to be invoked and the event goes directly to the deadletter queue.

When debugging my failed task invocation via the dead-letter queue, I see the detail object in my event, yet am getting INVALID JSON errors when the json actually looks okay to me.

{'Records': [{'messageId': '17aa13e7-e3da-4e7f-808b-657493faccd2', 'receiptHandle': 'AQEBmUP+EPXVt6dMdreKBgrl24o1+e6hnGB+Ijq88q/AjB8NIgaGb5bq0fKH1U4zV8grzW4vIXHPwRudQYMzIGUVlJScSYv+E0pFFTZfoL1gMCRArjXynabxEvirS2WkMzJlwvG7fV9vtFtG1qu340Jenujy8QBoAD4JXj4pm6VJ9LEFTNcHY5X6koSgIJpROki+2pzpS5zo8cNP2PICDP8lBpIPUTGnzt7X/KRe/+mKUvA7L9r6FWgo9z3lDQchR7FVLTQ+dFj6J0bVYlHd4ff1lvsC9hIJ35JBp/Vi2M0FPK2VBZD3zLZf7uTOChHK40ucNAPad5SIyubcRePNars2lN9pJzRLGv/SuoGZDxP7tMU8B1VcWkyTd8ukoLRTUb1l8moyoML+s/KlM3p9jEnF73nsetfaPJeOEA2I9zSgOxo=', 'body': '{"containerOverrides": [{"name": "dataplatform-job-1000", "environment": [{"name": "PAYLOAD", "value": "TEST VALUE"}, {"name": "CLIENT", "value": "MERCER"}, {"name": "DETAIL", "value": {"JOB_NAME":"RUN_MODEL_CHANGE_TRACKING","CLIENT":"MERCER"}}]}]}', 'attributes': {'ApproximateReceiveCount': '1', 'SentTimestamp': '1704214692999', 'SenderId': 'AIDAIE6VAXUQU2DKHA7DK', 'ApproximateFirstReceiveTimestamp': '1704214693001'}, 'messageAttributes': {'RULE_ARN': {'stringValue': 'arn:aws:events:us-west-2:871137929234:rule/dataplatform-job-1000-custom', 'stringListValues': [], 'binaryListValues': [], 'dataType': 'String'}, 'TARGET_ARN': {'stringValue': 'arn:aws:ecs:us-west-2:871137929234:cluster/dataplatform-dev', 'stringListValues': [], 'binaryListValues': [], 'dataType': 'String'}, 'ERROR_MESSAGE': {'stringValue': 'Invalid input for target.', 'stringListValues': [], 'binaryListValues': [], 'dataType': 'String'}, 'ERROR_CODE': {'stringValue': 'INVALID_JSON', 'stringListValues': [], 'binaryListValues': [], 'dataType': 'String'}}, 'md5OfMessageAttributes': 'dfb37a1907310a2f33a09ec242208633', 'md5OfBody': 'e59eb72c7e4eb650639245425e62496f', 'eventSource': 'aws:sqs', 'eventSourceARN': 'arn:aws:sqs:us-west-2:871137929234:dataplatform-deadletter-queue-dev', 'awsRegion': 'us-west-2'}]}

I feel like there needs to be some escaping done but I can't identify where.

Any help is appreciated!


As a follow-up:

According to ECS run task docs, the ContainerOverrides -> Environment object should be comprised of KeyValue pairs (https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerOverride.html) and the values should be a string type. Is there a way to force objects (like the $.detail object above) to be escaped strings instead of passed as an object? I'm wondering if that may be an underlying issue.

Chris
asked 4 months ago234 views
1 Answer
0

Hello.

I think it is necessary to escape double quotes when JSON is stored as a string within JSON.
So, escape the JSON double quotes in 'body' and try running the test event with this.

{'Records': [{'messageId': '17aa13e7-e3da-4e7f-808b-657493faccd2', 'receiptHandle': 'AQEBmUP+EPXVt6dMdreKBgrl24o1+e6hnGB+Ijq88q/AjB8NIgaGb5bq0fKH1U4zV8grzW4vIXHPwRudQYMzIGUVlJScSYv+E0pFFTZfoL1gMCRArjXynabxEvirS2WkMzJlwvG7fV9vtFtG1qu340Jenujy8QBoAD4JXj4pm6VJ9LEFTNcHY5X6koSgIJpROki+2pzpS5zo8cNP2PICDP8lBpIPUTGnzt7X/KRe/+mKUvA7L9r6FWgo9z3lDQchR7FVLTQ+dFj6J0bVYlHd4ff1lvsC9hIJ35JBp/Vi2M0FPK2VBZD3zLZf7uTOChHK40ucNAPad5SIyubcRePNars2lN9pJzRLGv/SuoGZDxP7tMU8B1VcWkyTd8ukoLRTUb1l8moyoML+s/KlM3p9jEnF73nsetfaPJeOEA2I9zSgOxo=', 'body': '{\"containerOverrides\": [{\"name\": \"dataplatform-job-1000\", \"environment\": [{\"name\": \"PAYLOAD\", \"value\": \"TEST VALUE\"}, {\"name\": \"CLIENT\", \"value\": \"MERCER\"}, {\"name\": \"DETAIL\", \"value\": {\"JOB_NAME\":\"RUN_MODEL_CHANGE_TRACKING\",\"CLIENT\":\"MERCER\"}}]}]}', 'attributes': {'ApproximateReceiveCount': '1', 'SentTimestamp': '1704214692999', 'SenderId': 'AIDAIE6VAXUQU2DKHA7DK', 'ApproximateFirstReceiveTimestamp': '1704214693001'}, 'messageAttributes': {'RULE_ARN': {'stringValue': 'arn:aws:events:us-west-2:871137929234:rule/dataplatform-job-1000-custom', 'stringListValues': [], 'binaryListValues': [], 'dataType': 'String'}, 'TARGET_ARN': {'stringValue': 'arn:aws:ecs:us-west-2:871137929234:cluster/dataplatform-dev', 'stringListValues': [], 'binaryListValues': [], 'dataType': 'String'}, 'ERROR_MESSAGE': {'stringValue': 'Invalid input for target.', 'stringListValues': [], 'binaryListValues': [], 'dataType': 'String'}, 'ERROR_CODE': {'stringValue': 'INVALID_JSON', 'stringListValues': [], 'binaryListValues': [], 'dataType': 'String'}}, 'md5OfMessageAttributes': 'dfb37a1907310a2f33a09ec242208633', 'md5OfBody': 'e59eb72c7e4eb650639245425e62496f', 'eventSource': 'aws:sqs', 'eventSourceARN': 'arn:aws:sqs:us-west-2:871137929234:dataplatform-deadletter-queue-dev', 'awsRegion': 'us-west-2'}]}
profile picture
EXPERT
answered 4 months ago
  • I agree that there needs to be some escaping done. But how can I accomplish that using the InputTransformer field (InputTemplate) as shown in my example above?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions