Strange behaviour from urllib in lambda

Question

Hello,

I have created a Lambda which is responsible for receiving certificate files from a frontend and uploading them to Transfer Family. In order to obtain the certificate chain, we begin by making a request like such:

```
with urllib.request.urlopen(issuer_cert_url) as response:
                                issuer_cert_data = response.read()
```

Previously, we attempted to close the connection manually by doing response.close(). This worked for the first 10 or so invocations, but then these requests would begin timing out. When we switched to using `as response`, this seemingly resolved itself for the most part, but we are still experiencing intermittent issues with these requests timing out. Cloudwatch indicates that the timeout always occurs at this point in the code. The only other connections we are making are to AWS services and our database.

These timeouts do not begin as a result of something like a redeployment or a VPC change, they seem to happen at random. The specific errors we are receiving are as follows:` [Errno 97] Address family not supported by protocol [Errno 110] Connection timed out`. Although it would appear that there is some issue with the address protocol, we can eliminate this as a possibility for two reasons.

1) When our VPC was previously misconfigured to disallow traffic, we got errors identical to this, and it was not affected by which one was submitted (and processed) first. The same is true for when we attempted manually closing the connection.
2) These same certificates are test cases that we have used repeatedly, and only time out occasionally.

We have also checked to ensure that the lambda execution time does not exceed our API gateway timeout limit, which is set to 29s, where our lambda typically takes about 10 seconds to run. Any advice is appreciated!

re:Post Agent · Answer

The issue you're experiencing with intermittent timeouts in your Lambda function when using urllib to make HTTP requests is likely related to how Lambda handles network connections in a VPC environment. Here are a few potential causes and solutions to consider:

1. VPC Configuration: Even though you've ruled out a misconfigured VPC, it's worth double-checking that your Lambda function has proper internet access. When a Lambda function is connected to a VPC, all outbound requests go through the VPC. To ensure internet connectivity, configure your VPC to send outbound traffic from the function's subnet to a NAT gateway in a public subnet.

2. Connection Reuse: Lambda functions may reuse execution environments across invocations, which can lead to stale or exhausted network connections. While using the `with` statement is a good practice, it might not fully solve the issue. Consider implementing a connection pooling mechanism or using a library like `requests` that handles connection pooling more efficiently.

3. DNS Resolution: The "[Errno 97] Address family not supported by protocol" error can sometimes be related to DNS resolution issues. Ensure that your VPC is configured with the AWS-provided DNS server and that DNS hostnames are enabled.

4. Timeouts and Retries: Implement proper timeout and retry logic in your code. Instead of relying solely on the default urllib behavior, set explicit timeouts and implement a retry mechanism with exponential backoff.

5. Lambda Concurrency: If your function is experiencing high concurrency, it might be hitting limits on the number of network interfaces Lambda can create in your VPC. Check your CloudWatch metrics for any throttling events and consider adjusting your concurrency settings if necessary.

6. Memory and CPU: Increase the memory allocation for your Lambda function. This not only provides more memory but also allocates more CPU power, which can help with network-related operations.

7. Downstream Service Issues: Monitor the availability and response times of the services you're connecting to. Implement logging or use AWS X-Ray to track the performance of these external calls.

To improve the reliability of your function, consider the following steps:

1. Implement robust error handling and retry logic.
2. Use a more feature-rich HTTP client library like `requests`.
3. Implement connection pooling if making multiple requests.
4. Set explicit timeouts on your HTTP requests.
5. Use AWS X-Ray to trace and analyze the function's execution, including external HTTP calls.
6. Monitor your function's performance using CloudWatch metrics and logs.
7. Consider using AWS SDK for Python (Boto3) for interacting with AWS services, as it's optimized for use within Lambda.

By implementing these suggestions, you should be able to improve the reliability and consistency of your Lambda function's network operations.  
**Sources**  
[Troubleshoot networking issues in Lambda - AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/troubleshooting-networking.html)  
[Troubleshooting integration errors - AWS Lambda](https://docs.aws.amazon.com/lambda/latest/operatorguide/integration-errors.html)  
[Troubleshooting Lambda configurations - AWS Lambda](https://docs.aws.amazon.com/lambda/latest/operatorguide/configurations.html)  
[Troubleshoot execution issues in Lambda - AWS Lambda](https://docs.aws.amazon.com/lambda/latest/dg/troubleshooting-execution.html)

Strange behaviour from urllib in lambda

Add your answer

Relevant content