New to AWS: What will be the best service to use to repeatedly ping an API endpoint? (And ideally stay in free tier)

Question

As I mentioned, I'm new to AWS and trying to plan out a personal project. I have an API that I want to get near-real-time updates from, but there is no notification service for that API (yet). My goal is to ping it at the rate limit to detect updates, and when certain updates come through, I want to ingest the data from those updates and store them. I have two current thoughts:

EC2 instance to ping endpoint -> SQS message when I find the certain updates I need  -> lambda for data transformation (and another API call) -> store in dynamodb for later access
Limitations: 
* If updates to this API come more often than the request rate limit, then I will need to make requests from multiple instances/different IP addresses
* To stay up to date, this would require 24/7 EC2 uptime, which I imagine will be expensive and I would like to not spend much on this personal project

Alternate Idea would be the same process, but all API calls would come from lambda functions (called from the EC2 instance) to get more API requests and more data in the same time. Getting around the API rate limit to stay up to date on the data I'm looking for
Limitations: 
* 24/7 EC2 uptime plus significantly more Lambda calls. This would be even more expensive

I would appreciate advice from anyone who has suggestions for other services or practices I could use to keep this as close to free-tier as possible!

Carlos Domingues · Accepted Answer

Depending on some numbers using a Lambda might be worth it. Assumptions:

1. A month has 2629800 seconds.
2. You can use the cheapest Lambda for checks (128 MB), which costs 0.000001666675 per second + 0.0000002 per call in the cheapest regions ([official docs](https://aws.amazon.com/lambda/pricing/)). 
3. `Tu` is the time you take to detect updates.
4. `Tc` is the time between checks (lambda executions).

Your monthly cost for using Lambdas will be `Tu * 0.000001666675 * (2629800 / Tc) +  (2629800 / Tc) *  0.0000002` =  `(4.383Tu + 0.526)/Tc`

You don't have much control over `Tu`, although you can optimize the function to be quicker. Let's assume you're using something like Python and each check takes 200 ms. Now you have: Cost = `1.4/Tc`

If you accept making checks every second, we are talking about 1.4 USD / month, which is cheaper than any on demand EC2 instance in the cheapest regions. If you need quicker checks, having a very small EC2 instance such as the `t4g.nano` turned on 24/7 costs about 3 USD per month, again, in the cheapest regions. So it all depends on your definition of near-real-time updates.

zafar_khan · Answer

You may want to use the Amazon EventBridge service to create a rule to schedule the execution of the Lambda function, which pings the API to fetch the data. Amazon EventBridge is a managed service for building event-driven applications. It costs only $1.00/million invocations per month after the initial 14 million free invocations per month. So it is very cheap than running and maintaining an EC2 instance.

You can also use AWS Step Functions to orchestrate the Lambda functions to build a workflow.

For more information, refer https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-run-lambda-schedule.html

You can also use AWS Step Functions to orchestrate the Lambda functions to build a workflow.

For more information, refer https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-run-lambda-schedule.html

New to AWS: What will be the best service to use to repeatedly ping an API endpoint? (And ideally stay in free tier)

Relevant content