Skip to content

propagate sagemaker domain tags to any underlying resource created by user

1

In sagemaker all underlying resources and user created resources do not get tags. One example is EFS volume for domain, s3 bucket create by domain, any user training job, experiment or model has tags optional and is not created by default.

I'm looking for a way to propagate sagemaker domain tags to any resource that would be created in underlying domain, not only underlying dependencies but also all users work/actions.

is it possible to add default tags thru policy or lifecycle?

asked 2 years ago657 views

1 Answer
0

For SageMaker resources like training jobs, endpoints, etc, my suggestion would be:

  1. Use a SageMaker Python SDK defaults configuration file (YAML) to set up default tags whenever users create jobs/etc through the SDK
    • Set up correctly, the tags specified in the config file should get automatically added without users having to specify them each time in their notebooks/Python scripts. Likewise other default settings you save here.
    • Note you can use a Lifecycle Configuration Script to automatically (re)-initialize this config file(s) each time each user starts Jupyter.
  2. Consider whether you'd like to restrict the permissions of Studio users' attached IAM roles to only allow creation of jobs/etc when tags are specified (e.g. the aws:TagKeys condition)
    • Be aware that users attempting to create jobs/etc without tags attached will see a pretty opaque "access denied" error: You'll want to share information about this change with the team to help everybody understand likely causes of seeing the error

You might also be interested to apply a similar strategy for other things like e.g. making sure resources get created within the scope of your AWS VPC.

For other resources in general (similar to other services from which users can log in to an instance and perform AWS actions), I believe tag propagation will need to be considered on a per-service/resource basis.

While SageMaker Studio Classic used to create a shared EFS filesystem for your domain and use that to store users' files, I'd strongly advise customers to be migrating towards new Studio (launched 2023-12 re:Invent) in which storage is EBS-based and I believe the volumes should be automatically tagged based on user tags. New Studio can still make use of EFS filesystems to share content between users' spaces, but attributing cost to individual users is not necessarily straightforward.

For Amazon S3, if you want your SageMaker notebook users to be able to create new buckets you could maybe deny them the direct IAM permission and give them access instead to a proxy service that requires specifying tag(s) up-front? I see the s3:CreateBucket action doesn't support a native aws:TagNames condition in IAM.

In summary, how to enforce & simplify tag attachment will vary depending on the kinds of AWS resources you want to allow your users to create - because there's such a wide range available across the whole AWS portfolio... But the most common resources in this context would be SageMaker jobs/endpoints/etc (which can be tackled nicely through a combination of SDK defaults setup and IAM) and Amazon S3 (in which the usual pattern is to carefully control which users have access to which project buckets/prefixes, and usually to restrict creation of brand new buckets so you always know what project/cost centre they're being created under).

AWS
EXPERT

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.