Sagemaker instances keep awakening and charge the credit

0

I have tried Data Wrangler in Sagemaker last month and close the service. A few weeks later I have noticed the credit was charge $1 every hour and just realized that the Data Wranger auto-save the flow every minute. So, I deleted the unsaved flow and shut down all the services and instances according to advice on these two links :

Then, I left the Sagemaker untouched for the whole month of May, and just got back to the console yesterday. This is what I found out for May's bill:

Amazon SageMaker RunInstance $531.74

DetailUsageTotal
$0.00 for Host:ml.m5.xlarge per hour under monthly free tier125.000 Hrs$0.00
$0.00 for Notebk:ml.t2.medium per hour under monthly free tier107.056 Hrs$0.00
$0.00 per Data Wrangler Interactive ml.m5.4xlarge hour under monthly free tier25.000 Hrs$0.00
$0.23 per Hosting ml.m5.xlarge hour in US East (N. Virginia)88.997 Hrs$20.47
$0.922 per Data Wrangler Interactive ml.m5.4xlarge hour in US East (N. Virginia)554.521 Hrs$511.27

So, with another attempt, I installed an extension to automatically shut down idle kernels and set the limit to 10 min from advice here: https://aws.amazon.com/blogs/machine-learning/save-costs-by-automatically-shutting-down-idle-resources-within-amazon-sagemaker-studio/ Checked the cost in usage report, it turns out that the service was shut down after installing the extension but then it revoked itself after 5 hours later (during my sleep time). There's still cost from Studio although with less charge than previous one.

ServiceOperationUsageTypeStartTimeEndTimeUsageValue
AmazonSageMakerRunInstanceUSE1-Studio_DW:KernelGateway-ml.m5.4xlarge5/24/2022 23:005/25/2022 0:001
AmazonSageMakerRunInstanceUSE1-Studio_DW:KernelGateway-ml.m5.4xlarge5/25/2022 0:005/25/2022 1:001
AmazonSageMakerRunInstanceUSE1-Studio_DW:KernelGateway-ml.m5.4xlarge5/25/2022 1:005/25/2022 2:001
AmazonSageMakerRunInstanceUSE1-Studio_DW:KernelGateway-ml.m5.4xlarge5/25/2022 2:005/25/2022 3:000.76484417
AmazonSageMakerRunInstanceUSE1-Studio_DW:KernelGateway-ml.m5.4xlarge5/25/2022 8:005/25/2022 9:000.36636722
AmazonSageMakerRunInstanceUSE1-Studio_DW:KernelGateway-ml.m5.4xlarge5/25/2022 9:005/25/2022 10:000.38959556

During this time, I'm sure that there're no running instances, running apps, kernel sessions or terminal sessions. I even deleted the user profile. Last thing I haven't tried is to set up scheduled shutdown coz I think the services should not cause difficulty to our life that much. Any advice for any effective action to completely shutdown the Sagemaker instance? Thanks.

1 Answer
0

Hi, you can shut down SageMaker Studio resources per the documentation here - https://docs.aws.amazon.com/sagemaker/latest/dg/notebooks-run-and-manage-shut-down.html (you've linked to Studio Lab documentation, so making sure). In addition, I see you've been charged for hosting and you've since deleted the endpoints, and don't see the hosting charges after deleting them.

For data wrangler, once you have the flow saved, you need to shut down the app (closing the window does not automatically shut down the app). Note that if you open the DW flow later, it does start a compute instance, which you'll then have to shut down.

If you've deleted the user profile (and associated apps), you shouldn't be seeing any more Studio charges for that user profile. If you still see the DW charges (and there's no other user profile), please cut a ticket to support for further investigation.

AWS
Durga_S
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions