Optimizing AWS Lambda for Google Cloud Platform Service Checks

Question

I'm currently using AWS Lambda in Python to write code that checks services on the Google Cloud Platform (using the googleclientapis library). I invoke Lambda functions through an API Gateway. However, I'm facing an issue where my Lambda function takes longer than the 29-second timeout limit enforced by the API Gateway, resulting in a 504 Timeout error on the API Gateway.

The code needs to check all regions for a specific GCP service, which requires looping through each region and making API calls to retrieve the resources for each region. As a result, it takes a bit longer to run (approximately 55 seconds).

I've tried several approaches:
1. Using asynchronous techniques in Python 
- It doesn't work; the list of regions is still called sequentially, and it doesn't run in parallel.
2. Using multithreading 
- Similarly, when making calls to the GCP API, it still performs sequentially.
3. Deploying multiple Lambda functions to handle different groups of regions
- This approach is also not efficient because the next Lambda function is only invoked after one completes and returns results.

Can anyone suggest a way to accomplish this in under 30 seconds?

Uri · Accepted Answer

I do not know what in the GCP APIs prevents the parallel access. If there is really something on their side, I would assume it has to do with an API Key or credentials. If this is the case, you will need to obtain multiple API keys/credentials so that you can run in parallel.

Assuming GCP is not the limiting factor, or that you can obtain multiple API Keys/credentials to run in parallel, you have a few options:
1. Run multiple threads in your function. There is no reason the threads can't make a call to an API at the same time.
2. Create an Express state machine in Step Functions and use the Map state to run a function in a loop, giving it high concurrency. Each invocation will check the status of one region. The total running time should not be much longer than the slowest region.

Optimizing AWS Lambda for Google Cloud Platform Service Checks

Relevant content