Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

boto3 - api operation with elb describe target health timeout

0

Here is an interesting problem,

I have a lambda running in a vpc - in private subnet, with a nlb in same private subnets, gave the correct role to the lambda for elb describe_target_health. The thing is that the operation timeout itself, and it never receives a value; i have no idea how to troubleshoot this.

The lambda functions is to check if the ip of an rds private link is the same as the one in target group and then update the target group if it is not matching.

Points:

  • NLB deployed in same private subnet and vpc as lambda
  • correct permissions for role given to lambda for describe_health_target
  • passed in the correct arn for the target group for the operation boto3
  • python runtime 3.9|3.10
  • even increasing the lambda duration to 1 mins will still make it timeout on the operation
Topics
ServerlessComputeNetworking & Content Delivery
Tags
Elastic Load BalancingAWS Lambda
Language
English
asked 3 years ago760 views
1 Answer
2
Accepted Answer

If I'm guessing correctly, the private subnet is completely private - it has no access to the internet. The Lambda function is trying to query the status of the load balancer using the AWS API endpoint which is on the internet, therefore it isn't working.

You can test this by (temporarily) removing the connection to the VPC for the Lambda function. If it works when not connected to the VPC then that's definitely the issue.

Two solutions:

  • Add a private endpoint to your VPC for the Elastic Load Balancing service.
  • Add a NAT Gateway to your VPC so that the Lambda function can access the internet. This will allow it to access other internet-based resources as well so the first option is probably better.
AWS
EXPERT
answered 3 years ago
  • Vincent Chu
    3 years ago

    everything is in private subnet, the nlb is in private subnet. both the nlb and lambda are in same private subnet so they should be able to talk

  • Brettski-AWS EXPERT
    3 years ago

    Having those things in the private subnet does not mean that the Lambda function can access the ELB control plane to call DescribeTargetHealth. You need the private endpoint for that because it's not the load balancer that responds - it's the ELB service.

  • Vincent Chu
    3 years ago

    ok, let say i deployed the lambda in the public subnets with internet gateways, theorically it should be able to consume the ELB service ? or is it possible i messed up the route tables for the public subnets that it cant reach the ELB service via boto3 ?

  • Brettski-AWS EXPERT
    3 years ago

    You'd deploy the Lambda function in a private subnet and then either use a private endpoint or use a NAT Gateway that is in the public subnet. Bear in mind there are two things here: "Reaching" the load balancer means being able to communicate with the targets that are registered with the load balancer - that doesn't require anything except the Lambda to be in the private subnet. Calling the load balancer API and using DescribeTargetHealth means accessing the AWS endpoint which requires the private endpoint or NAT Gateway. Given that you're using boto3 you're doing the second of these.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content