Skip to content

Unable to get Lifecycle Hooks to function properly as a lambda in ECS Blue/Green deployments

0

Following both the guidance of this blog post https://aws.amazon.com/blogs/aws/accelerate-safe-software-releases-with-new-built-in-blue-green-deployments-in-amazon-ecs/ and the documentation here https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-lifecycle-hooks.html I have yet to get a lifecycle hook to pass. The errors are usually in the form of

Service deployment rolled back because POST_TEST_TRAFFIC_SHIFT lifecycle hook(s) failed. ECS was unable to parse the response from arn:aws:lambda:us-east-1:940900242534:function:post_traffic_hook due to: HookStatus must not be null

Simplest repro code is as follows (python)

def lambda_handler(event, context):
    try:
        time.sleep(120)
        return {'hookStatus': 'SUCCEEDED'}
    
    except Exception as error:
        logger.error(f"File upload test failed: {str(error)}")
        return { "hookStatus": "FAILED" }

I've also tried json string-ifying it, http styled return bodies, casing etc. Any idea on why it is claiming null and what the return structure should be?

asked 10 months ago322 views
2 Answers
0

The issue with your lifecycle hook Lambda function is related to the response format. When using lifecycle hooks with Amazon ECS blue/green deployments, the Lambda function must return a specific response structure.

Based on the error message "HookStatus must not be null," it appears that ECS is not correctly parsing the response from your Lambda function. The correct response format should be a JSON object with a hookStatus key that has one of these values: SUCCEEDED, FAILED, or IN_PROGRESS.

Your code is almost correct, but there are a few potential issues:

  1. Make sure your Lambda function has enough execution time. Since you're sleeping for 120 seconds, ensure your Lambda timeout is set to more than 2 minutes.

  2. The response format should be exactly as follows:

{
"hookStatus": "SUCCEEDED"
}
  1. Unlike CodeDeploy hooks which require calling PutLifecycleEventHookExecutionStatus(), ECS blue/green deployments use the direct Lambda response to determine the hook status.

  2. If you want to implement a polling mechanism, you can return "hookStatus": "IN_PROGRESS" along with an optional callBackDelay to specify when ECS should invoke your function again (default is 30 seconds if not specified).

Also, ensure that your ECS service has the proper IAM permissions to invoke the Lambda function. The IAM role needs the lambda:InvokeFunction permission for your Lambda function.

If you're still experiencing issues, try simplifying your Lambda function to just return the success status without any sleep or try/catch blocks to isolate the problem:

def lambda_handler(event, context):
return {"hookStatus": "SUCCEEDED"}

Sources
Lifecycle hooks for Amazon ECS service deployments - Amazon Elastic Container Service
Permissions required for Lambda functions in Amazon ECS blue/green deployments - Amazon Elastic Container Service
Migrate CodeDeploy blue/green deployments to Amazon ECS blue/green deployments - Amazon Elastic Container Service

answered 10 months ago
0

Lambda timeout was set to less than my execution time, hence getting stopped early and no return code.

answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.