DynamoDB Read Requests went from 80million to 1.2billion. Why / How?

0

I have an application hosted on AWS which stores data in DynamoDB. I then read this data on a daily basis.

Every month prior to October 2022 I was hitting below 100million read requests.

For October it says I hit 1.2billion requests. Id like to know if this is some sort of error because I havnt done anything different from my side so I dont see how this 1.2 billion requests is even possible.

Is there a way for me to see how this happened?

Thanks in advance!

jcom09
asked a month ago35 views
2 Answers
1

First of all, do you know what region these requests are happening in? If not, then Cost Explorer (assuming this was enabled already) can tell you which region had the highest DynamoDB costs (filter by DynamoDB and group by region).

Then, to see where the requests are coming from, you can look at CloudWatch. This should be able to tell you which table was seeing elevated requests. The default metric storage may only be 2 weeks, so if it was a constant increase, then this should still show you where to look. And you should be able to drill down to see when the requests were happening. These metrics should give you an indication of what is happening and open up for you where to look next.

profile picture
answered a month ago
  • Thank you for the advice! I was able to find the day all these requests happened. I stupidly dont have a long enough retention time on my logs so cant get any other info other than the day it happened. It seems this might have been some sort of an attach though... I dont know why someone would want to do this but it happened early hours of the morning when the system wasnt being used. I wonder if the billing department will cut me some slack here :(

    Would there be some way to prevent this happening again? What to implement to trigger a stop or I dont know something to prevent this...

  • (1) Log retention is a tricky, you want them long enough to be useful, but not long enough to cost too much. (2) You can open a support case for billing asking about the extra large cost. There are no guarantees a refund will be approved, but you can ask. (3) Regarding prevention, this will really depend on your application and architecture. If you have developer or higher support, you can ask about this in a technical support case. If you want to be notified about spikes in cost, you should look at Cost Anomaly Detector and/or AWS Budgets.

1

I would highly recommend reaching out to Customer Services for this type of issue:

https://aws.amazon.com/contact-us/

Customer Services can work with internal support teams to understand the reason why your consumption increased and if in-fact an attack had happened. Unfortunately I am not in a position to state if CS can cut you some slack in regards to the costs, something you will need to discuss with them.

Going forward, I would strongly advise that you do a number of things:

  • Set realistic scaling limits on your tables, ensuring you cannot exceed a given amount will prevent this issue in future
  • Ensure that your application is secured by IAM and only authenticated users can access it
  • If you use REST API's for access to your resources, impose throttling rules to prevent abuse
  • Set alarms in CloudWatch that you get notified (email/slack/other) when your table exceeds capacity for x amount of time.
  • You can also set alarms on costs, to notify you if costs surpass a given value for a given month.
profile picture
answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions