Lambda Execution Function Issue For RDS Reboot

0

Greetings, I created a simple function taking as reference the basic Lambda in Python to start/stop RDS from here:

https://aws.amazon.com/es/blogs/database/schedule-amazon-rds-stop-and-start-using-aws-lambda/

But I changed it for reboot purposes, so my Python code is the following:

    # Lambda for RDS reboot given a REGION, KEY and VALUE
    import boto3
    import os
    import sys
    import time
    from datetime import datetime, timezone
    from time import gmtime, strftime

    # REGION: the rds region
    # KEY - VALUE: the KEY and VALUE from RDS tag
    def reboot_rds():
    region = os.environ["REGION"]
    key = os.environ["KEY"]
    value = os.environ["VALUE"]
    client = boto3.client("rds", region_name=region)
    response = client.describe_db_instances()
    v_readReplica = []
    for i in response["DBInstances"]:
        readReplica = i["ReadReplicaDBInstanceIdentifiers"]
    v_readReplica.extend(readReplica)

    for i in response["DBInstances"]:
        # Check if the RDS is Aurora
        if i["Engine"] not in ["aurora-mysql", "aurora-postgresql"]:
            # Check if RDS is a replica instance
            if (
                i["DBInstanceIdentifier"] not in v_readReplica
                and len(i["ReadReplicaDBInstanceIdentifiers"]) == 0
            ):
                arn = i["DBInstanceArn"]
                resp2 = client.list_tags_for_resource(ResourceName=arn)
    # Check tag
    if 0 == len(resp2["TagList"]):
        print("Instance {0} tag value is not correct".format(i["DBInstanceIdentifier"]))
    else:
        for tag in resp2["TagList"]:
            # if tag values match
            if tag["Key"] == key and tag["Value"] == value:
                if i["DBInstanceStatus"] == "available":
                    client.reboot_db_instance(
                        DBInstanceIdentifier=i["DBInstanceIdentifier"],
                        ForceFailover=False,
                    )
                    print("Rebooting RDS {0}".format(i["DBInstanceIdentifier"]))
                elif i["DBInstanceStatus"] == "rebooting":
                    print(
                        "Instance RDS {0} is already rebooting".format(
                            i["DBInstanceIdentifier"]
                        )
                    )
                elif i["DBInstanceStatus"] == "creating":
                    print(
                        "Instance RDS {0} is on creation, try later".format(
                            i["DBInstanceIdentifier"]
                        )
                    )
                elif i["DBInstanceStatus"] == "modifying":
                    print(
                        "Instance RDS {0} {0} is modifying, try later".format(
                            i["DBInstanceIdentifier"]
                        )
                    )
                elif i["DBInstanceStatus"] == "stopped":
                    print(
                        "Cannot reboot RDS {0} it is already stopped".format(
                            i["DBInstanceIdentifier"]
                        )
                    )
                elif i["DBInstanceStatus"] == "starting":
                    print(
                        "Instance RDS {0} is starting, try later".format(
                            i["DBInstanceIdentifier"]
                        )
                    )
                elif i["DBInstanceStatus"] == "stopping":
                    print(
                        "Instance RDS {0} is stopping, try later.".format(
                            i["DBInstanceIdentifier"]
                        )
                    )
                elif tag["Key"] != key and tag["Value"] != value:
                    print(
                        "Tag values {0} doesn't match".format(i["DBInstanceIdentifier"])
                    )
                elif len(tag["Key"]) == 0 or len(tag["Value"]) == 0:
                    print("Error {0}".format(i["DBInstanceIdentifier"]))
                else:
                    print(
                        "Instance RDS {0} is on a different state, check the RDS monitor for more info".format(
                            i["DBInstanceIdentifier"]
                        )
                    )
    def lambda_handler(event, context):
         reboot_rds()

My environment variables:

KeyValue
KEYtmptest
REGIONus-east-1e
VALUEreboot

And finally my event named 'Test'

{ "key1": "tmptest", "key2": "us-east-1e", "key3": "reboot" }

I checked the indentation of my code before execute it and its fine, but in execution of my test event I got the following output:

{ "errorMessage": "2022-01-14T14:50:22.245Z b8d0dc59-714d-4543-8651-b5a2532dfe8e Task timed out after 1.00 seconds" }

START RequestId: b8d0dc59-714d-4543-8651-b5a2532dfe8e Version: $LATEST
END RequestId: b8d0dc59-714d-4543-8651-b5a2532dfe8e
REPORT RequestId: b8d0dc59-714d-4543-8651-b5a2532dfe8e	
Duration: 1000.76 ms	
Billed Duration: 1000 ms	
Memory Size: 128 MB	
Max Memory Used: 65 MB	
Init Duration: 243.69 ms	
2022-01-14T14:50:22.245Z b8d0dc59-714d-4543-8651-b5a2532dfe8e Task timed out after 1.00 seconds

Also my test RDS has the correct tag values in order to get the reboot action but nothing, until now I cannot reboot my instance with my Lambda function.

Any clue what's wrong with my code?

Maybe some additional configuration issue or something in my code is not correct, I don't know. I'd appreciate if someone can give a hand with this.

UPDATE 2022/01/15

As suggestion of Brettski@AWS I increased the time from 1 second to 10 then I got the following error message:

{
  "errorMessage": "Could not connect to the endpoint URL: \"https://rds.us-east-1e.amazonaws.com/\"",
  "errorType": "EndpointConnectionError",
  "requestId": "b2bb3840-42a2-4220-84b4-642d17d7a9e6",
  "stackTrace": [
    "  File \"/var/task/lambda_function.py\", line 103, in lambda_handler\n    reboot_rds\n",
    "  File \"/var/task/lambda_function.py\", line 16, in reboot_rds\n    response = client.describe_db_instances()\n",
    "  File \"/var/runtime/botocore/client.py\", line 386, in _api_call\n    return self._make_api_call(operation_name, kwargs)\n",
    "  File \"/var/runtime/botocore/client.py\", line 691, in _make_api_call\n    http, parsed_response = self._make_request(\n",
    "  File \"/var/runtime/botocore/client.py\", line 711, in _make_request\n    return self._endpoint.make_request(operation_model, request_dict)\n",
    "  File \"/var/runtime/botocore/endpoint.py\", line 102, in make_request\n    return self._send_request(request_dict, operation_model)\n",
    "  File \"/var/runtime/botocore/endpoint.py\", line 136, in _send_request\n    while self._needs_retry(attempts, operation_model, request_dict,\n",
    "  File \"/var/runtime/botocore/endpoint.py\", line 253, in _needs_retry\n    responses = self._event_emitter.emit(\n",
    "  File \"/var/runtime/botocore/hooks.py\", line 357, in emit\n    return self._emitter.emit(aliased_event_name, **kwargs)\n",
    "  File \"/var/runtime/botocore/hooks.py\", line 228, in emit\n    return self._emit(event_name, kwargs)\n",
    "  File \"/var/runtime/botocore/hooks.py\", line 211, in _emit\n    response = handler(**kwargs)\n",
    "  File \"/var/runtime/botocore/retryhandler.py\", line 183, in __call__\n    if self._checker(attempts, response, caught_exception):\n",
    "  File \"/var/runtime/botocore/retryhandler.py\", line 250, in __call__\n    should_retry = self._should_retry(attempt_number, response,\n",
    "  File \"/var/runtime/botocore/retryhandler.py\", line 277, in _should_retry\n    return self._checker(attempt_number, response, caught_exception)\n",
    "  File \"/var/runtime/botocore/retryhandler.py\", line 316, in __call__\n    checker_response = checker(attempt_number, response,\n",
    "  File \"/var/runtime/botocore/retryhandler.py\", line 222, in __call__\n    return self._check_caught_exception(\n",
    "  File \"/var/runtime/botocore/retryhandler.py\", line 359, in _check_caught_exception\n    raise caught_exception\n",
    "  File \"/var/runtime/botocore/endpoint.py\", line 200, in _do_get_response\n    http_response = self._send(request)\n",
    "  File \"/var/runtime/botocore/endpoint.py\", line 269, in _send\n    return self.http_session.send(request)\n",
    "  File \"/var/runtime/botocore/httpsession.py\", line 373, in send\n    raise EndpointConnectionError(endpoint_url=request.url, error=e)\n"
  ]
}

It's strange because my VPC configuration is fine, it's the same VPC of my RDS, its zone and the same security group. What else have I to consider in order to make my code work properly?

3 Answers
2

Hi, not sure if Lamba has access to stop/reboot your RDS instance, if not - it's the reason

Check this article, it has all you need, described step by step

answered 2 years ago
  • Ok, I'll check it. Thanks

1

Definitely check the access for the Lambda function as per the other answer.

However, your error message is saying that the Lambda function is timing out after one second. This means that the maximum duration that you've set for this function is one second - if there is a delay in RDS service communicating with your function you could easily exceed the one second time and the Lambda function will be forcibly terminated even though things might be working normally (albeit slowly).

Try increasing the maximum runtime for the function to (say) 5 or 10 seconds. You'll still only be billed for the time that the function runs; but it allows some breathing room in case there are unexpected delays during execution.

profile pictureAWS
EXPERT
answered 2 years ago
  • I did but now I got a new error related with the connection but my VPC connection is correct. I updated the question if you want to know more about it.

1

It's strange because my VPC configuration is fine, it's the same VPC of my RDS, its zone and the same security group. What else have I to consider in order to make my code work properly?

  1. Two resources being in the same Security Group does not actually confer permission for those resources to communicate with each other unless the security group inbound rules include that same security group as a source. This is not actually relevant here since your problem is reaching the RDS endpoint, and not the instance itself. It would be relevant if the Lambda were attempting data-plane operations like read and write.
  2. If your Lambda is connected to a VPC, then you need to configure your Lambda networking so you can reach the RDS endpoint. You can do this over the internet using a NAT GW, or your can configure an AWS Private Link. In either case you will need to ensure that the SG configuration allows the Lambda to contact the NGW or VPC endpoint respectively. If you Lambda has no reason to be connected to a VPC, then it should not be, and you can avoid this entire step.
  3. The IAM Role used by the Lambda must have a policy that includes RebootDBInstance
profile pictureAWS
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions