Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Being charged for unused SageMaker: What beyond Instances and Domain can I shut down in SageMaker?

0

On Monday I restarted my SageMaker domain to run an Autopilot experiment in Canvas. That took about two hours, over which I was charged about $28 for active work. When I finished the work, I hit the "off" button for Canvas (which seems to have worked fine), and then checked my EC2 Global View -> Region Explorer tab to make sure that no additional instances were still running. 0 out of 0 Instances were (and are) running in any Region. I then walked away, thinking I'd done my required tidying up (short of lingering S3 data).

I woke up the next morning to a budget alert and find that I'm being charged $1.97/hour for something SageMaker. I thought maybe it was the existence of the Domain itself, regardless of EC2 activity, so I "stopped" it. I also thought maybe a notebook had been automatically spun up in the background, and checked for those, but there are no notebooks active. Even with the Domain stopped, I am still being charged $1.96-$1.97/hour for "SageMaker." What else do I need to shut down? I can't imagine that the S3 storage for the output and logging a brief Autopilot experiment can be costing this much.

Topics
Cloud Financial ManagementComputeMachine Learning & AIStorage
Tags
Amazon SageMaker CanvasAmazon SageMakerAmazon EC2AWS Cost and Usage Report
Language
English
asked a year ago905 views
4 Answers
1

Hi,

I'd suggest you to read the initial part of this blog post to identify your resources creating the costs in order to be able to remove them.

https://aws.amazon.com/blogs/machine-learning/part-1-analyze-amazon-sagemaker-spend-and-determine-cost-optimization-opportunities-based-on-usage-part-1/

Best,

Didier

EXPERT
answered a year ago
EXPERT
reviewed a year ago
  • CBio
    a year ago

    I have read that, Didier. We're pretty far past that work.

    The problem appears to be a Canvas Workspace instance that did not respond to the my log off, even though on my end I am logged off in the region of interest (and all other regions supporting Canvas). I am working with AWS support for them to shut this Canvas Workspace down on their end.

  • Didier Durand EXPERT
    a year ago

    Hi CBio, glad that you found the source of those costs. Didier

1

It appears that you've taken some important steps to shut down SageMaker resources, but there might be additional components still running that are causing the ongoing charges. Here are some additional areas to check and shut down:

  1. Inference endpoints: SageMaker endpoints can continue to run and incur charges even if you're not actively using them. Make sure to delete any endpoints that were created during your Autopilot experiment.

  2. MLflow resources: If you used MLflow for experiment tracking, there might be resources associated with it that need to be cleaned up.

  3. HyperPod clusters: If your Autopilot experiment used HyperPod, ensure that any clusters created have been deleted.

  4. Pipelines: If any SageMaker pipelines were created, make sure to stop them.

  5. EMR clusters: If your experiment involved EMR in Studio Classic, terminate any Amazon EMR clusters that might still be running.

  6. RStudio resources: If you used RStudio on SageMaker, clean up any associated image resources and shut down RStudio.

  7. TensorBoard applications: Delete any unused TensorBoard applications.

  8. VPC resources: Check if there are any VPC resources associated with your SageMaker setup that need to be deleted.

It's important to note that shutting down or stopping a domain doesn't automatically terminate all associated resources. You may need to manually delete or terminate these resources to stop incurring charges.

To get a comprehensive view of all active SageMaker resources, it's recommended to use AWS Cost Explorer. This will help you identify exactly which resources are still running and contributing to the charges.

If you're still unable to identify the source of the charges after checking these areas, it would be advisable to contact AWS Support for a detailed breakdown of the charges and assistance in identifying and terminating any lingering resources.
Sources
Where to shut down resources per SageMaker AI features - Amazon SageMaker AI
Shut down SageMaker AI resources in your domain - Amazon SageMaker AI
Shut down resources from Amazon SageMaker Studio Classic - Amazon SageMaker AI

answered a year ago
EXPERT
reviewed a year ago
0
Accepted Answer

If you log out of Canvas and it shows "logged out" on your end, but you're still being charged a couple of dollars indefinitely, get in touch with AWS Support to resolve the problem.

There is a "Workspace Instance Session-Hour" charge running (see AWS docs on sagemaker-ai canvas pricing structure) even though you are logged out of Canvas, and this won't be visible to you in the Console, so you won't be able to shut it down. It's not an EC2, a VPC, storage, or any of the other usual suspects. This issue is not covered in the AWS docs linked above this.

answered a year ago
0

I didn't spin up anything in the items 1-7. I have already checked cost explorer and have been carefully monitoring it for about 48 hours now. The only thing (I believe) that could be costing me extra for a SageMaker VPC is a NAT gateway at $0.045/hour, which (1) doesn't seem to exist and (2) would not come anywhere near to explaining this on-going cost of $1.96-$1.97/hour.

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content