Unexpected SageMaker costs

Question

Hello,
I have been learning AWS as a part of a Udemy course, and I have been following the steps to create instances, roles, etc. I used SageMaker AI and tried running a project as a requirement for that course, but it didn't go through, so I stopped running that project.
I am fairly new to AWS so I didn't realise that using SageMaker would result in such high costs. Please help me with it!

re:Post Agent · Accepted Answer

When you're experiencing unexpected SageMaker charges even after stopping your project, there are several hidden components that might still be running. Here's what you should check and clean up:

1. **MLflow Tracking Servers**: These can continue running in the background and appear as "RunTrackingServer" charges on your bill. You may need to use the AWS CLI or API to list and delete these.

2. **Inference endpoints**: These continue to run and incur charges even if you're not actively using them.

3. **Canvas workspaces**: Even if you logged out of Canvas, there might be "Workspace Instance Session-Hour" charges running that aren't visible in the Console.

4. **EBS volumes and ML storage**: Simply stopping a project doesn't automatically remove associated storage volumes. The charge of $0.112 per GB-Mo for provisioned ML storage can continue.

5. **Check all regions**: SageMaker resources are region-specific, so verify all AWS regions to ensure you haven't missed any active resources.

6. **Hidden resources**: Some SageMaker components like HyperPod clusters, Pipelines, EMR clusters, RStudio resources, and TensorBoard applications might not be automatically terminated when you stop a project.

7. **VPC resources**: Check for any lingering VPC resources associated with SageMaker.

To identify the exact source of charges:
- Use AWS Cost Explorer to pinpoint which specific SageMaker resources are causing the costs
- Filter by SageMaker in the Service dropdown and view by API Operations to see which API calls are costing you

If you're still unable to locate the source of the charges after checking these areas, contact AWS Support for a detailed breakdown and assistance in identifying and terminating any lingering resources.

For future use, consider setting up AWS Budgets to alert you before exceeding your desired spending limit.  
**Sources**  
[Unexpected SageMaker charges appearing on my bill | AWS re:Post](https://repost.aws/questions/QUrRzL1PaFQ3WdS0tZoZM2tA/unexpected-sagemaker-charges-appearing-on-my-bill)  
[.56 cents every month. | AWS re:Post](https://repost.aws/questions/QU6hmmeWMbSe-MCvL2Rwk_VA/56-cents-every-month)  
[Being charged for unused SageMaker: What beyond Instances and Domain can I shut down in SageMaker? | AWS re:Post](https://repost.aws/questions/QUgiBiPI6ARWasijuNHRbfSg/being-charged-for-unused-sagemaker-what-beyond-instances-and-domain-can-i-shut-down-in-sagemaker)  
[How to cancel SageMaker | AWS re:Post](https://repost.aws/questions/QUpWpNgsS0TQW4NmpByOie4Q/how-to-cancel-sagemaker)  
[Unable to turn off $0.112 per GB-Mo of provisioned ML storage for SageMaker | AWS re:Post](https://repost.aws/questions/QUhaoEzF8gSJSHXpdR0i910A/unable-to-turn-off-0-112-per-gb-mo-of-provisioned-ml-storage-for-sagemaker)

SHAJAM · Answer

Also, since you are new to AWS, you should raise a support ticket and they might help you and provide you a refund or lower the bill.

Unexpected SageMaker costs

Add your answer

Relevant content