Hi there,
I'm working on dynamically setting ECS container task environment variables by extracting values from the event that triggers the task via EventBridge.
I'm able to extract individual values from the Event detail
attribute and pass that to the container, but I keep getting errors when trying to pass along the entire detail
object.
I've set up a deadletter queue to help debug the eventbridge rule and target execution, but haven't yet found anything actionable. I've also tried this question (https://repost.aws/questions/QUYyKFM5vWQMmIECUXsx11Jw/unable-to-pass-aws-events-event-json-in-input-transformer-template) and this documentation (https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-transform-target-input.html#eb-transform-input-issues) to verify what I'm trying to accomplish but have had no luck as of yet.
I'm using the Python Boto3 library to add the target for my eventbridge rule.
template = '{{"containerOverrides": [{{"name": "{}", "environment": [{{"name": "PAYLOAD", "value": "TEST VALUE"}}, {{"name": "CLIENT", "value": <client>}}, {{"name": "DETAIL", "value": <detail>}}]}}]}}'.format(container_name)
config = {
'Id': target_id,
'Arn': ecs_cluster_arn,
'RoleArn': eventbridge_role_arn,
'InputTransformer': {
'InputPathsMap': {
'client': '$.detail.CLIENT',
'detail': '$.detail'
},
'InputTemplate': template
},
... # commented out for brevity
}
response = self.__client.put_targets(
Rule=rule_name,
Targets=[config]
)
I don't run into any issues when creating the rule target; however, when attempting to test the rule with a PutEvents command from the AWS CLI, the task fails to be invoked and the event goes directly to the deadletter queue.
When debugging my failed task invocation via the dead-letter queue, I see the detail
object in my event, yet am getting INVALID JSON
errors when the json actually looks okay to me.
{'Records': [{'messageId': '17aa13e7-e3da-4e7f-808b-657493faccd2', 'receiptHandle': 'AQEBmUP+EPXVt6dMdreKBgrl24o1+e6hnGB+Ijq88q/AjB8NIgaGb5bq0fKH1U4zV8grzW4vIXHPwRudQYMzIGUVlJScSYv+E0pFFTZfoL1gMCRArjXynabxEvirS2WkMzJlwvG7fV9vtFtG1qu340Jenujy8QBoAD4JXj4pm6VJ9LEFTNcHY5X6koSgIJpROki+2pzpS5zo8cNP2PICDP8lBpIPUTGnzt7X/KRe/+mKUvA7L9r6FWgo9z3lDQchR7FVLTQ+dFj6J0bVYlHd4ff1lvsC9hIJ35JBp/Vi2M0FPK2VBZD3zLZf7uTOChHK40ucNAPad5SIyubcRePNars2lN9pJzRLGv/SuoGZDxP7tMU8B1VcWkyTd8ukoLRTUb1l8moyoML+s/KlM3p9jEnF73nsetfaPJeOEA2I9zSgOxo=', 'body': '{"containerOverrides": [{"name": "dataplatform-job-1000", "environment": [{"name": "PAYLOAD", "value": "TEST VALUE"}, {"name": "CLIENT", "value": "MERCER"}, {"name": "DETAIL", "value": {"JOB_NAME":"RUN_MODEL_CHANGE_TRACKING","CLIENT":"MERCER"}}]}]}', 'attributes': {'ApproximateReceiveCount': '1', 'SentTimestamp': '1704214692999', 'SenderId': 'AIDAIE6VAXUQU2DKHA7DK', 'ApproximateFirstReceiveTimestamp': '1704214693001'}, 'messageAttributes': {'RULE_ARN': {'stringValue': 'arn:aws:events:us-west-2:871137929234:rule/dataplatform-job-1000-custom', 'stringListValues': [], 'binaryListValues': [], 'dataType': 'String'}, 'TARGET_ARN': {'stringValue': 'arn:aws:ecs:us-west-2:871137929234:cluster/dataplatform-dev', 'stringListValues': [], 'binaryListValues': [], 'dataType': 'String'}, 'ERROR_MESSAGE': {'stringValue': 'Invalid input for target.', 'stringListValues': [], 'binaryListValues': [], 'dataType': 'String'}, 'ERROR_CODE': {'stringValue': 'INVALID_JSON', 'stringListValues': [], 'binaryListValues': [], 'dataType': 'String'}}, 'md5OfMessageAttributes': 'dfb37a1907310a2f33a09ec242208633', 'md5OfBody': 'e59eb72c7e4eb650639245425e62496f', 'eventSource': 'aws:sqs', 'eventSourceARN': 'arn:aws:sqs:us-west-2:871137929234:dataplatform-deadletter-queue-dev', 'awsRegion': 'us-west-2'}]}
I feel like there needs to be some escaping done but I can't identify where.
Any help is appreciated!
As a follow-up:
According to ECS run task docs, the ContainerOverrides -> Environment object should be comprised of KeyValue pairs (https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerOverride.html) and the values should be a string
type. Is there a way to force objects (like the $.detail
object above) to be escaped strings instead of passed as an object? I'm wondering if that may be an underlying issue.
I agree that there needs to be some escaping done. But how can I accomplish that using the InputTransformer field (InputTemplate) as shown in my example above?