By using AWS re:Post, you agree to the Terms of Use
/NoCredentialsError: Unable to locate credentials when using dask_ml inside sagemaker/

NoCredentialsError: Unable to locate credentials when using dask_ml inside sagemaker

0

I'm trying to use dask in sagemaker because I have over 1B+ rows in a single dataset. Creating a dask.dataframe works fine, when I create a client through dask, it also works:

client = Client(n_workers=6, threads_per_worker=20)
client

However when I try to use dask_ml.preprocessing.Categorizer, I get the error NoCredentialsError: Unable to locate credentials. I understand the issue might be dask distributed client doesn't have authorization to sagemaker client. Its confusing but how do I have them connect somehow? Code below:

import dask.dataframe as dd
df3 = dd.read_parquet('s3://bucket/parquetfiles2/data_*.parquet',
                     storage_options={'token': 'anon'})
if __name__ == "__main__":
    obj_df = df3.select_dtypes(include=['object','datetime64'])
    num_df = df3.select_dtypes(exclude=['object','datetime64'])
    ce = Categorizer()
    obj_df_1 = ce.fit_transform(obj_df)
1 Answers
0

Hello,

Thank you for contacting us and for using Amazon Sagemaker.

I understand that you encountered a "NoCredentialsError: Unable to locate credentials" when trying to use dask_ml.preprocessing.Categorizer.

It looks like you're running the code locally on your machine. Please feel free to correct me if I have misunderstood anything here. The error is usually seen when you don’t have AWS credentials correctly configured on your machine.

However, I see you're using Dask Distributed client to use Amazon SageMaker Processing. Please have a look here : https://sagemaker-examples.readthedocs.io/en/latest/sagemaker_processing/feature_transformation_with_sagemaker_processing_dask/feature_transformation_with_sagemaker_processing_dask.html#Build-a-Dask-container-for-running-the-preprocessing-job where we have built Dask enabled containers for SageMaker Processing.

You might need to run aws configure so set up IAM Credentials (User) on your machine. 1 However, to be able to replicate and investigate into this further, we'd need your IAM role arn and other details. Hence, for further investigation on this issue, I recommend to cut a support case and provide more detail about your account information and script/config. Due to security reason, we cannot discuss account specific issue in the public posts.

Please open a support case with AWS using the link:

https://console.aws.amazon.com/support/home?#/case/create

Thank you

References :

1 https://aws.amazon.com/premiumsupport/knowledge-center/s3-locate-credentials-error/ 2 https://youtu.be/UMUQs2PojdE

answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions