Questions tagged with Amazon SageMaker

Hi Team, I have a fine-tuned BERT model which was trained using following libraries. torch == 1.8.1+cu111 transformers == 4.19.4 And not able to convert that fine-tuned BERT model into AWS neuron and getting following compilation errors. Could you please help me to resolve this issue? **Note:** Trying to compile BERT model on SageMaker notebook instance and with "conda_python3" conda environment. **Installation:** #### Set Pip repository to point to the Neuron repository !pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com #### Install Neuron PyTorch - Note: Tried both options below. "#!pip install torch-neuron==1.8.1.* neuron-cc[tensorflow] "protobuf<4" torchvision sagemaker>=2.79.0 transformers==4.17.0 --upgrade" !pip install --upgrade torch-neuron neuron-cc[tensorflow] "protobuf<4" torchvision --------------------------------------------------------------------------------------------------------------------------------------------------- **Model compilation:** ``` import os import tensorflow # to workaround a protobuf version conflict issue import torch import torch.neuron from transformers import AutoTokenizer, AutoModelForSequenceClassification model_path = 'model/' # Model artifacts are stored in 'model/' directory # load tokenizer and model tokenizer = AutoTokenizer.from_pretrained(model_path) model = AutoModelForSequenceClassification.from_pretrained(model_path, torchscript=True) # create dummy input for max length 128 dummy_input = "dummy input which will be padded later" max_length = 128 embeddings = tokenizer(dummy_input, max_length=max_length, padding="max_length", truncation=True, return_tensors="pt") neuron_inputs = tuple(embeddings.values()) # compile model with torch.neuron.trace and update config model_neuron = torch.neuron.trace(model, neuron_inputs) model.config.update({"traced_sequence_length": max_length}) # save tokenizer, neuron model and config for later use save_dir="tmpd" os.makedirs("tmpd",exist_ok=True) model_neuron.save(os.path.join(save_dir,"neuron_model.pt")) tokenizer.save_pretrained(save_dir) model.config.save_pretrained(save_dir) ``` --------------------------------------------------------------------------------------------------------------------------------------------------- **Model artifacts:** We have got this model artifacts from multi-label topic classification model. config.json model.tar.gz pytorch_model.bin special_tokens_map.json tokenizer_config.json tokenizer.json --------------------------------------------------------------------------------------------------------------------------------------------------- **Error logs:** ``` INFO:Neuron:There are 3 ops of 1 different types in the TorchScript that are not compiled by neuron-cc: aten::embedding, (For more information see https://github.com/aws/aws-neuron-sdk/blob/master/release-notes/neuron-cc-ops/neuron-cc-ops-pytorch.md) INFO:Neuron:Number of arithmetic operators (pre-compilation) before = 565, fused = 548, percent fused = 96.99% INFO:Neuron:Number of neuron graph operations 1601 did not match traced graph 1323 - using heuristic matching of hierarchical information WARNING:tensorflow:From /home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch_neuron/ops/aten.py:2022: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where INFO:Neuron:Compiling function _NeuronGraph$698 with neuron-cc INFO:Neuron:Compiling with command line: '/home/ec2-user/anaconda3/envs/python3/bin/neuron-cc compile /tmp/tmpv4gg13ze/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpv4gg13ze/graph_def.neff --io-config {"inputs": {"0:0": [[1, 128, 768], "float32"], "1:0": [[1, 1, 1, 128], "float32"]}, "outputs": ["Linear_5/aten_linear/Add:0"]} --verbose 35' INFO:Neuron:Compile command returned: -9 WARNING:Neuron:torch.neuron.trace failed on _NeuronGraph$698; falling back to native python function call ERROR:Neuron:neuron-cc failed with the following command line call: /home/ec2-user/anaconda3/envs/python3/bin/neuron-cc compile /tmp/tmpv4gg13ze/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpv4gg13ze/graph_def.neff --io-config '{"inputs": {"0:0": [[1, 128, 768], "float32"], "1:0": [[1, 1, 1, 128], "float32"]}, "outputs": ["Linear_5/aten_linear/Add:0"]}' --verbose 35 Traceback (most recent call last): File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch_neuron/convert.py", line 382, in op_converter item, inputs, compiler_workdir=sg_workdir, **kwargs) File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/torch_neuron/decorators.py", line 220, in trace 'neuron-cc failed with the following command line call:\n{}'.format(command)) subprocess.SubprocessError: neuron-cc failed with the following command line call: /home/ec2-user/anaconda3/envs/python3/bin/neuron-cc compile /tmp/tmpv4gg13ze/graph_def.pb --framework TENSORFLOW --pipeline compile SaveTemps --output /tmp/tmpv4gg13ze/graph_def.neff --io-config '{"inputs": {"0:0": [[1, 128, 768], "float32"], "1:0": [[1, 1, 1, 128], "float32"]}, "outputs": ["Linear_5/aten_linear/Add:0"]}' --verbose 35 INFO:Neuron:Number of arithmetic operators (post-compilation) before = 565, compiled = 0, percent compiled = 0.0% INFO:Neuron:The neuron partitioner created 1 sub-graphs INFO:Neuron:Neuron successfully compiled 0 sub-graphs, Total fused subgraphs = 1, Percent of model sub-graphs successfully compiled = 0.0% INFO:Neuron:Compiled these operators (and operator counts) to Neuron: INFO:Neuron:Not compiled operators (and operator counts) to Neuron: INFO:Neuron: => aten::Int: 97 [supported] INFO:Neuron: => aten::add: 39 [supported] INFO:Neuron: => aten::contiguous: 12 [supported] INFO:Neuron: => aten::div: 12 [supported] INFO:Neuron: => aten::dropout: 38 [supported] INFO:Neuron: => aten::embedding: 3 [not supported] INFO:Neuron: => aten::gelu: 12 [supported] INFO:Neuron: => aten::layer_norm: 25 [supported] INFO:Neuron: => aten::linear: 74 [supported] INFO:Neuron: => aten::matmul: 24 [supported] INFO:Neuron: => aten::mul: 1 [supported] INFO:Neuron: => aten::permute: 48 [supported] INFO:Neuron: => aten::rsub: 1 [supported] INFO:Neuron: => aten::select: 1 [supported] INFO:Neuron: => aten::size: 97 [supported] INFO:Neuron: => aten::slice: 5 [supported] INFO:Neuron: => aten::softmax: 12 [supported] INFO:Neuron: => aten::tanh: 1 [supported] INFO:Neuron: => aten::to: 1 [supported] INFO:Neuron: => aten::transpose: 12 [supported] INFO:Neuron: => aten::unsqueeze: 2 [supported] INFO:Neuron: => aten::view: 48 [supported] --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-1-97bba321d013> in <module> 18 19 # compile model with torch.neuron.trace and update config ---> 20 model_neuron = torch.neuron.trace(model, neuron_inputs) 21 model.config.update({"traced_sequence_length": max_length}) 22 ~/anaconda3/envs/python3/lib/python3.6/site-packages/torch_neuron/convert.py in trace(func, example_inputs, fallback, op_whitelist, minimum_segment_size, subgraph_builder_function, subgraph_inputs_pruning, skip_compiler, debug_must_trace, allow_no_ops_on_neuron, compiler_workdir, dynamic_batch_size, compiler_timeout, _neuron_trace, compiler_args, optimizations, verbose, **kwargs) 182 logger.debug("skip_inference_context - trace with fallback at {}".format(get_file_and_line())) 183 neuron_graph = cu.compile_fused_operators(neuron_graph, **compile_kwargs) --> 184 cu.stats_post_compiler(neuron_graph) 185 186 # Wrap the compiled version of the model in a script module. Note that this is ~/anaconda3/envs/python3/lib/python3.6/site-packages/torch_neuron/convert.py in stats_post_compiler(self, neuron_graph) 491 if succesful_compilations == 0 and not self.allow_no_ops_on_neuron: 492 raise RuntimeError( --> 493 "No operations were successfully partitioned and compiled to neuron for this model - aborting trace!") 494 495 if percent_operations_compiled < 50.0: RuntimeError: No operations were successfully partitioned and compiled to neuron for this model - aborting trace! ``` --------------------------------------------------------------------------------------------------------------------------------------------------- Thanks a lot.

When I open the Model details in the Amazon SageMaker console, the details clearly show Tags that have been added to the model during it's creation. | Tags | | Key | | sagemaker:project-name | | aws:cloudformation:stack-name| | sagemaker:deployment-stage| | sagemaker:deployment-stage| But the Search API provided by Amazon SageMaker, [https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_Search.html]() mentions only the following resources can be searched for using Tags: Valid Values: TrainingJob | Experiment | ExperimentTrial | ExperimentTrialComponent | Endpoint | ModelPackage | ModelPackageGroup | Pipeline | PipelineExecution | FeatureGroup | Project | FeatureMetadata I wish to obtain Model details, not the ModelPackageGroup/ModelPackage details, using Tags so if there is a way to do that, please share. Also if there is no way to obtain it using Tags, like the Search API Documentation suggests, what is the purpose of the Tags still present in the Model details?

Hi folks, I'm trying to set the spark executor instances & memory, driver memory and switch of dynamic allocation. What is the correct way to do it?

We've just created a train/build/deploy template in AWS SageMaker which provides a deployment of an Abalone model. We're trying to test it via the Test Inference endpoint, but the JSON there is rejected with the following message: > Error invoking endpoint: Received client error (415) from model with message "application/json is not an accepted ContentType: csv, libsvm, parquet, recordio-protobuf, text/csv, text/libsvm, text/x-libsvm, application/x-parquet, application/x-recordio-protobuf.". See https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logEventViewer:group=/aws/sagemaker/Endpoints/USEngProbOfConversion-staging in account 607522716587 for more information. However the Test Inference endpoint only allows us to hit the endpoint with JSON - what can we do? Here's a screenshot (if this dropbox embed works):  but that's not working so here's the request dump: ``` { "body": { "s-x": "M", "length": 3, "diameter": 5, "height": 7, "whole_weight": 45, "shucked_weight": 34, "viscera_weight": 23, "shell_weight": 76 }, "contentType": "application/json", "endpointName": "USEngProbOfConversion-staging", "customURL": "", "customHeaders": [ { "Key": "sm_endpoint_name", "Value": "USEngProbOfConversion-staging" }, { "Key": "", "Value": "" } ] } ```

I have a sagemaker notebook that has no connections to internet or codecommit but has access to 1 s3 bucket. I would like to use that 1 s3 bucket as a place to house git repos, ideally I would like to be able to pull/push to repos in that bucket from other sagemaker notebooks or ec2 instances that have connections to that bucket. Has anyone tried this before?

Hi all, I have raised a ticket for multiple issues we've been having with SageMaker lately, the ticket was created more than 36 hours ago, and I have not had any response, in fact the ticket hasn't even been assigned yet. The case ID is 10300240931. I thought AWS guarantee a response under 12 hours for "system impaired" issues, does anyone know what I can do to accelerate this? thank you! Ruoy

After months of seamless work in SageMaker Studio, the JupyterServer App won't load for the last 4 days. The Control Panel shows that the JupyterServer is in "Pending" or "Failed" state after I try to launch the app. When clicking "Launch app", the screen shows that: * "The JupyterServer app default encountered a problem and was stopped." * The "Restart Now" button is visible, but pressing this results in the same behaviour. I created a new JupyterServer App and it experiences the same problem under that account. I use a different account for another project and the JupyterServer under that account works perfectly. I even mounted the EFS associated with the App on an EC2 instance and deleted some files to reduce the EFS volume but it did not help (it was 995 MB and as far as I know, 5GB is the default limit). I found a post stating the same problem from 2 years ago, but could not follow the advice to delete the app and create a new one, since the Delete option is not available in the Action dropdown ([https://repost.aws/questions/QUxoSA7eTzQbK-T4OWjJvSmQ/sage-maker-studio-will-not-load]()). All apps that I create is "default". Please help, how could I overcome this and access Jupyter Lab again? Thank you.

I've successfully connected a Sagemaker "notebook" to a private github repository, but wondering if it isn't possible for a studio instance? Failing that is there an easy way to get the remote codecommit git url for an existing "studio" instance so that code there can at least be pulled to my local machine?

Hi all, Since Jupyterlab 3.0 was finally released on SM Studio, we have been super happy with it, however, for reasons unknown to us, the jupterlab interface works very poorly with SM resources, the following phenomenon have been observes: 1. It takes FOREVER to load the page for SM pipelines, and half the time it reports error ("Error listing pipeline executions: Rate exceeded") 2. Changing instance type and size for a notebook is now super laggy, and do not work half the time Anyone knows if this is merely a lack of optimisation on the service team's part or is there something I can do to stop this behaviour? it's making our work very slow and unbearable, we don't want to revert back to 1.0 so any help would be greatly appreciated! Best, RUoy

I'm creating a data processing job in sagemaker notebook: ``` from sagemaker.sklearn.processing import SKLearnProcessor sklearn_processor = SKLearnProcessor(role=role, base_job_name='end-to-end-ml-sm-proc', instance_type='ml.m5.large', instance_count=1, framework_version='0.23-1') ``` my processing script uses : ``` from sagemaker_sklearn_extension.decomposition import RobustPCA ``` and I get an error during the job exectution: ``` Traceback (most recent call last): File "/opt/ml/processing/input/code/preprocessor.py", line 14, in <module> from sagemaker_sklearn_extension.decomposition import RobustPCA ModuleNotFoundError: No module named 'sagemaker_sklearn_extension' ``` as far as I understrand : framework_version='0.23-1' should make sagemaker create docker image based on image from that repo: https://github.com/aws/sagemaker-scikit-learn-container and the 0.23-1 branch handles extensions installation (if extenssion/Dockerfile.cpu file is executed), but I don't see how I can make Sagemaker run that script when creating the job. how can I use sagemaker_sklearn_extension in Sagemaker job?

I have created a multi-container model in SageMaker notebook and deployed it through an endpoint. But while attempting to do the same through a SageMaker Studio Project (build, train and deploy model template), I need to register the multi-container model through a 'sagemaker.workflow.step_collections.RegisterModel' step, which I am unable to do. What I understand till now is multi-container model is created through boto3 api call. I haven't found a way to create it using 'sagemaker.model.Model' and hence not being able to register it. Please help.

**Requirement:** Create SakeMaker GroundTruth labeling job with input/output location pointing to S3 bucket in another AWS account **High Level Steps Followed:** Lets say, *Account_A:* SageMaker GroundTruth labeling job and *Account_B*: S3 bucket 1. Create role *AmazonSageMaker-ExecutionRole* in *Account_A* with 3 policies attached: * AmazonSageMakerFullAccess * Account_B_S3_AccessPolicy: Policy with necessary S3 permissions to access S3 bucket in Account_B * AssumeRolePolicy: Assume role policy for *arn:aws:iam::Account_B:role/Cross-Account-S3-Access-Role* 2. Create role *Cross-Account-S3-Access-Role* in *Account_B* with 1 policy and 1 trust relationship attached: * S3_AccessPolicy: Policy with necessary S3 permissions to access S3 bucket in the this Account_B * TrustRelationship: For principal *arn:aws:iam::Account_A:role/AmazonSageMaker-ExecutionRole* **Error:** While trying to create SakeMaker GroundTruth labeling job with IAM role as *AmazonSageMaker-ExecutionRole*, it throws error *AccessDenied: Access Denied - The S3 bucket 'Account_B_S3_bucket_name' you entered in Input dataset location cannot be reached. Either the bucket does not exist, or you do not have permission to access it. If the bucket does not exist, update Input dataset location with a new S3 URI. If the bucket exists, give the IAM entity you are using to create this labeling job permission to read and write to this S3 bucket, and try your request again.*

I am attempting to create a native PDF annotation labeling job for use with Comprehend to identify entities within similar documents. I have around 20 pdf files, all of them around 100-300 pages long. I used the tools and followed the directions from [this blog post.](https://aws.amazon.com/blogs/machine-learning/custom-document-annotation-for-extracting-named-entities-in-documents-using-amazon-comprehend/) I've struggled a little with the tools but ultimately got everything working. My problem comes from the labeling job itself. When I open the labeling job in a private workforce that I've created, I find only a blank page. I did some research and found that there is something wrong with the input manifest as it seems AWS isn't able to parse it for some reason. I checked my manifest and found that it was generated as multiple objects. Each object was a single page from my PDFs. This seems normal, however the objects were not put into a list or 'top level' object, which does not fit JSON Lines guidelines. I attempted a quick fix of placing these objects all within a list (which satisfies JSON Lines) but it does not seem to help. Any suggestions or advice would be greatly appreciated.

When I try to download the pre-trained Hugging Face BERT models weights to the conda_pytorch_p36 kernel of my Amazon SageMaker Notebook instance using the following command, the kernel always restarts: ``` PRE_TRAINED_MODEL_NAME2='sshleifer/distilbart-cnn-12-6' model2 = BartForConditionalGeneration.from_pretrained(PRE_TRAINED_MODEL_NAME2, cache_dir='hf_cache_dir/') ``` Note I have installed following libraries using pip commands. ``` !pip install transformers==4.17.0 ``` The result is the same for Hugging Face "facebook/bart-large-cnn" models. Why is this happening, and how do I resolve the issue?

Hi, i want to deploy a serverless model using CloudFormation. I've created the model, the endpoint configuration, and when I try to create the endpoint the script times out, because it can't find a saved model to attach to the endpoint (because I never trained one for this instance). I've tried to look around for a trainingjob cloudformation API, but there doesn't seem to be one. How do I solve this issue?

I have the following model in a jupyter notebook: ```python import tensorflow as tf from tensorflow.keras.models import Model from tensorflow.keras.optimizers import Adam from tensorflow.keras import layers physical_devices = tf.config.list_physical_devices('GPU') tf.config.experimental.set_memory_growth(physical_devices[0], True) SIZE = (549, 549) SHUFFLE = False BATCH = 32 EPOCHS = 20 train_datagen = DataGenerator(train_files, batch_size=BATCH, dim=SIZE, n_channels=1, shuffle=SHUFFLE) test_datagen = DataGenerator(test_files, batch_size=BATCH, dim=SIZE, n_channels=1, shuffle=SHUFFLE) inp = layers.Input(shape=(*SIZE, 1)) x = layers.Conv2D(filters=549, kernel_size=(5,5), padding="same", activation="relu")(inp) x = layers.BatchNormalization()(x) x = layers.Conv2D(filters=549, kernel_size=(3, 3), padding="same", activation="relu")(x) x = layers.BatchNormalization()(x) x = layers.Conv2D(filters=549, kernel_size=(1, 1), padding="same", activation="relu")(x) x = layers.BatchNormalization()(x) x = layers.Conv2D(filters=549, kernel_size=(3, 3), padding="same", activation="sigmoid")(x) model = Model(inp, x) model.compile(loss=tf.keras.losses.binary_crossentropy, optimizer=Adam()) model.summary() ``` Sagemaker and EC2 are running tensorflow 2.7.1. The EC2 instance is p3.2xlarge with Deep Learning AMI GPU TensorFlow 2.7.0 (Amazon Linux 2) 20220607. The SageMaker notebook is using ml.p3.2xlarge and I am using the conda_tensorflow2_p38 kernel. The notebook is in an FSx Lustre file system that is mounted to both SageMaker and EC2 so it is definitely the same code running on both machines. nvidia-smi output on SageMaker: ``` +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla V100-SXM2... On | 00000000:00:1E.0 Off | 0 | | N/A 37C P0 24W / 300W | 0MiB / 16384MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ ``` nvidia-smi output on EC2: ``` +-----------------------------------------------------------------------------+ | NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla V100-SXM2... On | 00000000:00:1E.0 Off | 0 | | N/A 42C P0 51W / 300W | 2460MiB / 16384MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 11802 C /bin/python3.8 537MiB | | 0 N/A N/A 26391 C python3.8 1921MiB | +-----------------------------------------------------------------------------+ ``` The model.summary() output on SageMaker is: ```python Model: "model" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(None, 549, 549, 1)] 0 conv2d (Conv2D) (None, 549, 549, 1) 7535574 batch_normalization (BatchN (None, 549, 549, 1) 4 ormalization) conv2d_1 (Conv2D) (None, 549, 549, 1) 2713158 batch_normalization_1 (Batc (None, 549, 549, 1) 4 hNormalization) conv2d_2 (Conv2D) (None, 549, 549, 1) 301950 batch_normalization_2 (Batc (None, 549, 549, 1) 4 hNormalization) conv2d_3 (Conv2D) (None, 549, 549, 1) 2713158 ================================================================= Total params: 13,263,852 Trainable params: 13,263,846 Non-trainable params: 6 ``` The model.summary() output on EC2 is (notice the shape change): ```python Model: "model" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(None, 549, 549, 1)] 0 conv2d (Conv2D) (None, 549, 549, 549) 14274 batch_normalization (BatchN (None, 549, 549, 549) 2196 ormalization) conv2d_1 (Conv2D) (None, 549, 549, 549) 2713158 batch_normalization_1 (Batc (None, 549, 549, 549) 2196 hNormalization) conv2d_2 (Conv2D) (None, 549, 549, 549) 301950 batch_normalization_2 (Batc (None, 549, 549, 549) 2196 hNormalization) conv2d_3 (Conv2D) (None, 549, 549, 549) 2713158 ================================================================= Total params: 5,749,128 Trainable params: 5,745,834 Non-trainable params: 3,294 _________________________________________________________________ ``` One other thing that is interesting, if I change my model on the EC2 instance to: ```python inp = layers.Input(shape=(*SIZE, 1)) x = layers.Conv2D(filters=1, kernel_size=(5,5), padding="same", activation="relu")(inp) x = layers.BatchNormalization()(x) x = layers.Conv2D(filters=1, kernel_size=(3, 3), padding="same", activation="relu")(x) x = layers.BatchNormalization()(x) x = layers.Conv2D(filters=1, kernel_size=(1, 1), padding="same", activation="relu")(x) x = layers.BatchNormalization()(x) x = layers.Conv2D(filters=1, kernel_size=(3, 3), padding="same", activation="sigmoid")(x) model = Model(inp, x) model.compile(loss=tf.keras.losses.binary_crossentropy, optimizer=Adam()) ``` My model.summary() output becomes: ```python Model: "model_2" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_3 (InputLayer) [(None, 549, 549, 1)] 0 conv2d_8 (Conv2D) (None, 549, 549, 1) 26 batch_normalization_6 (Batc (None, 549, 549, 1) 4 hNormalization) conv2d_9 (Conv2D) (None, 549, 549, 1) 10 batch_normalization_7 (Batc (None, 549, 549, 1) 4 hNormalization) conv2d_10 (Conv2D) (None, 549, 549, 1) 2 batch_normalization_8 (Batc (None, 549, 549, 1) 4 hNormalization) conv2d_11 (Conv2D) (None, 549, 549, 1) 10 ================================================================= Total params: 60 Trainable params: 54 Non-trainable params: 6 _________________________________________________________________ ``` In the last model the shape is similar to SageMaker but the trainable parameters are very low. Any ideas as to why the output shape is different and why this is happening with the filters? When I run this model on my personal computer, the shape is the same as EC2. I think there might be an issue with SageMaker.

Hi Dears, Hope this message finds you well I have a sagemaker model, buit by on demand notebook. I have been used batch transform jobs using lambda function, It take input inference json from s3 to create batch transform job and have finally predictions. The question how can I make lambda to use last trained model automaticity ? model_name = 'forecasting-deepar-2022-05-20-22-23-20-225', Lambda code : if 'input_data_4' in file: def batch_transform(): transformer = Transformer( model_name = 'forecasting-deepar-2022-05-20-22-23-20-225', instance_count = 2, instance_type = 'ml.m5.xlarge', assemble_with = 'Line', output_path = output_data_path, base_transform_job_name ='daily-output-predictions-to-s3', #sagemaker_session = sagemaker.session.Session, accept = 'application/jsonlines') transformer.transform(data = input_data_path, content_type = 'application/jsonlines', split_type = 'Line', wait=False, logs=True) #Waits for the Pipeline Transform Job to finish. print('Batch Transform Job Created successfully!') batch_transform() Thanks Basem

Hi there, I have a SM pipeline which contains a lambda step. I have recently run into problems with the pipeline/lambda function, everytime I call the `pipeline.upsert()` method, I receive the following error message(s): ``` ResourceConflictException: An error occurred (ResourceConflictException) when calling the UpdateFunctionCode operation: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:eu-west-x:XXXXXXXXXXXX:function:evaluation-input-generator During handling of the above exception, another exception occurred: ``` ``` ValueError: {'Message': 'The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:eu-west-X:XXXXXXXXXXXX:function:evaluation-input-generator', 'Code': 'ResourceConflictException'} ``` For some reasons, the lambda seems to be permanently stuck in `An update is in progress for resource: ...`, however, I suspect that this is not the case, I am able to call the lambda helper's update method for the function and it will return status 200. this behaviour is observed in multiple SM pipelines, any help would be greatly appreciated! Best, Ruoy

We are submitting a Sagemaker Training job with Sagemaker SDK with a custom docker image. The job finishes successfully for EFS [FileSystemInput](https://sagemaker.readthedocs.io/en/stable/api/utility/inputs.html#sagemaker.inputs.FileSystemInput) or [TrainingInput](https://sagemaker.readthedocs.io/en/stable/api/utility/inputs.html#sagemaker.inputs.TrainingInput). Trying to use the FIleSystemInput with FSxLustre configuration leads to the training job dying during the `Preparing the instances for training` stage: ``` InternalServerError: We encountered an internal error. Please try again. ``` This error is persistent upon re-submission. What we figured out until now: - the job errors before the training image is downloaded. - specifying an invalid mount point leads to a proper error: ```ClientError: Unable to mount file system: xxx directory path: yyy. Incorrect mount path. Please ensure the mount path specified exists on the filesystem.``` - the job finishes successfully when running locally with docker-compose ([Estimator](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html#sagemaker.estimator.EstimatorBase) with `instance_type="local"`). - we can mount the FSx file system on an EC2 instance with the TrainingJob's VPC and security group. How can we narrow the problem down further and get more information about the failure reason? Can you suggest likely problems that could cause this behavior?

I am unclear as o how to determine the "right" instance type running Jupyter notebook in Sagemaker. When reading/writing a small size parquet file, no problem; but when I try to read/write a huge parquet file, the program stops and gives an error, "Job aborted due to stage failure: Task 21 in stage 33.0 failed 1 times, most recent failure: Lost task 21.0 in stage 33.0 (TID 1755, localhost, executor driver" I would appreciate any insights please... thanks.

I'm trying to deploy custom trained Yolov5 model in Sagemaker for inference. (Note : The model was not trained in sagemaker). Followed this doc for deploying the model and inference script - [Sagemaker docs](https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/using_pytorch.html#bring-your-own-model) ``` ModelError Traceback (most recent call last) <ipython-input-7-063ca701eab7> in <module> ----> 1 result1=predictor.predict("FILE0032.JPG") 2 print(result1) ~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/predictor.py in predict(self, data, initial_args, target_model, target_variant, inference_id) 159 data, initial_args, target_model, target_variant, inference_id 160 ) --> 161 response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args) 162 return self._handle_response(response) 163 ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs) 399 "%s() only accepts keyword arguments." % py_operation_name) 400 # The "self" in this scope is referring to the BaseClient. --> 401 return self._make_api_call(operation_name, kwargs) 402 403 _api_call.__name__ = str(py_operation_name) ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params) 729 error_code = parsed_response.get("Error", {}).get("Code") 730 error_class = self.exceptions.from_code(error_code) --> 731 raise error_class(parsed_response, operation_name) 732 else: 733 return parsed_response ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (0) from primary with message "Your invocation timed out while waiting for a response from container primary. Review the latency metrics for each container in Amazon CloudWatch, resolve the issue, and try again.". See https://ap-south-1.console.aws.amazon.com/cloudwatch/home?region=ap-south-1#logEventViewer:group=/aws/sagemaker/Endpoints/pytorch-inference-2022-06-14-11-58-04-086 in account 772044684908 for more information. ``` After researching about `InvokeEndpoint`, tried this ``` import boto3 sagemaker_runtime = boto3.client("sagemaker-runtime", region_name='ap-south-1') endpoint_name='pytorch-inference-2022-06-14-11-58-04-086' response = sagemaker_runtime.invoke_endpoint( EndpointName=endpoint_name, Body=bytes('{"features": ["This is great!"]}', 'utf-8') # Replace with your own data. ) print(response['Body'].read().decode('utf-8')) ``` But this didn't help as well, detailed output : ``` ReadTimeoutError Traceback (most recent call last) <ipython-input-8-b5ca204734c4> in <module> 12 response = sagemaker_runtime.invoke_endpoint( 13 EndpointName=endpoint_name, ---> 14 Body=bytes('{"features": ["This is great!"]}', 'utf-8') # Replace with your own data. 15 ) 16 ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs) 399 "%s() only accepts keyword arguments." % py_operation_name) 400 # The "self" in this scope is referring to the BaseClient. --> 401 return self._make_api_call(operation_name, kwargs) 402 403 _api_call.__name__ = str(py_operation_name) ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params) 716 apply_request_checksum(request_dict) 717 http, parsed_response = self._make_request( --> 718 operation_model, request_dict, request_context) 719 720 self.meta.events.emit( ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _make_request(self, operation_model, request_dict, request_context) 735 def _make_request(self, operation_model, request_dict, request_context): 736 try: --> 737 return self._endpoint.make_request(operation_model, request_dict) 738 except Exception as e: 739 self.meta.events.emit( ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/endpoint.py in make_request(self, operation_model, request_dict) 105 logger.debug("Making request for %s with params: %s", 106 operation_model, request_dict) --> 107 return self._send_request(request_dict, operation_model) 108 109 def create_request(self, params, operation_model=None): ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/endpoint.py in _send_request(self, request_dict, operation_model) 182 request, operation_model, context) 183 while self._needs_retry(attempts, operation_model, request_dict, --> 184 success_response, exception): 185 attempts += 1 186 self._update_retries_context( ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/endpoint.py in _needs_retry(self, attempts, operation_model, request_dict, response, caught_exception) 306 event_name, response=response, endpoint=self, 307 operation=operation_model, attempts=attempts, --> 308 caught_exception=caught_exception, request_dict=request_dict) 309 handler_response = first_non_none_response(responses) 310 if handler_response is None: ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/hooks.py in emit(self, event_name, **kwargs) 356 def emit(self, event_name, **kwargs): 357 aliased_event_name = self._alias_event_name(event_name) --> 358 return self._emitter.emit(aliased_event_name, **kwargs) 359 360 def emit_until_response(self, event_name, **kwargs): ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/hooks.py in emit(self, event_name, **kwargs) 227 handlers. 228 """ --> 229 return self._emit(event_name, kwargs) 230 231 def emit_until_response(self, event_name, **kwargs): ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/hooks.py in _emit(self, event_name, kwargs, stop_on_response) 210 for handler in handlers_to_call: 211 logger.debug('Event %s: calling handler %s', event_name, handler) --> 212 response = handler(**kwargs) 213 responses.append((handler, response)) 214 if stop_on_response and response is not None: ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/retryhandler.py in __call__(self, attempts, response, caught_exception, **kwargs) 192 checker_kwargs.update({'retries_context': retries_context}) 193 --> 194 if self._checker(**checker_kwargs): 195 result = self._action(attempts=attempts) 196 logger.debug("Retry needed, action of: %s", result) ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/retryhandler.py in __call__(self, attempt_number, response, caught_exception, retries_context) 266 267 should_retry = self._should_retry(attempt_number, response, --> 268 caught_exception) 269 if should_retry: 270 if attempt_number >= self._max_attempts: ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/retryhandler.py in _should_retry(self, attempt_number, response, caught_exception) 292 # If we've exceeded the max attempts we just let the exception 293 # propogate if one has occurred. --> 294 return self._checker(attempt_number, response, caught_exception) 295 296 ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/retryhandler.py in __call__(self, attempt_number, response, caught_exception) 332 for checker in self._checkers: 333 checker_response = checker(attempt_number, response, --> 334 caught_exception) 335 if checker_response: 336 return checker_response ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/retryhandler.py in __call__(self, attempt_number, response, caught_exception) 232 elif caught_exception is not None: 233 return self._check_caught_exception( --> 234 attempt_number, caught_exception) 235 else: 236 raise ValueError("Both response and caught_exception are None.") ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/retryhandler.py in _check_caught_exception(self, attempt_number, caught_exception) 374 # the MaxAttemptsDecorator is not interested in retrying the exception 375 # then this exception just propogates out past the retry code. --> 376 raise caught_exception ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/endpoint.py in _do_get_response(self, request, operation_model, context) 247 http_response = first_non_none_response(responses) 248 if http_response is None: --> 249 http_response = self._send(request) 250 except HTTPClientError as e: 251 return (None, e) ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/endpoint.py in _send(self, request) 319 320 def _send(self, request): --> 321 return self.http_session.send(request) 322 323 ~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/httpsession.py in send(self, request) 449 raise ConnectTimeoutError(endpoint_url=request.url, error=e) 450 except URLLib3ReadTimeoutError as e: --> 451 raise ReadTimeoutError(endpoint_url=request.url, error=e) 452 except ProtocolError as e: 453 raise ConnectionClosedError( ReadTimeoutError: Read timeout on endpoint URL: "https://runtime.sagemaker.ap-south-1.amazonaws.com/endpoints/pytorch-inference-2022-06-14-11-58-04-086/invocations" ```

I'm trying to create my own Docker image for use with SageMaker Batch Transform by extending an existing one. Following the documentation at https://docs.aws.amazon.com/sagemaker/latest/dg/prebuilt-containers-extend.html, I have created the following to run Detectron 2: ``` FROM 763104351884.dkr.ecr.eu-west-2.amazonaws.com/pytorch-inference:1.10.2-gpu-py38-cu113-ubuntu20.04-sagemaker ############# Installing latest builds ############ RUN pip install --upgrade torch==1.10.2+cu113 torchvision==0.11.3+cu113 -f https://download.pytorch.org/whl/torch_stable.html ENV FORCE_CUDA="1" # Build D2 only for Turing (G4) and Volta (P3) architectures. Use P3 for batch transforms and G4 for inference on endpoints ENV TORCH_CUDA_ARCH_LIST="Turing;Volta" # Install Detectron2 RUN pip install \ --no-cache-dir pycocotools~=2.0.0 \ --no-cache-dir https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/detectron2-0.6%2Bcu113-cp38-cp38-linux_x86_64.whl # Set a fixed model cache directory. Detectron2 requirement ENV FVCORE_CACHE="/tmp" ############# SageMaker section ############## ENV PATH="/opt/ml/code:${PATH}" COPY inference.py /opt/ml/code/inference.py ENV SAGEMAKER_SUBMIT_DIRECTORY /opt/ml/code ENV SAGEMAKER_PROGRAM inference.py ``` I then create a model (`create-model`) with this image using the following configuration: ``` { "ExecutionRoleArn": "arn:aws:iam::[redacted]:role/model-role", "ModelName": "model-test", "PrimaryContainer": { "Environment": { "SAGEMAKER_PROGRAM": "inference.py", "SAGEMAKER_SUBMIT_DIRECTORY": "/opt/ml/code", "SAGEMAKER_CONTAINER_LOG_LEVEL": "20", "SAGEMAKER_REGION": "eu-west-2", "MMS_DEFAULT_RESPONSE_TIMEOUT": "500" }, "Image": "[redacted].dkr.ecr.eu-west-2.amazonaws.com/my-image:latest", "ModelDataUrl": "s3://[redacted]/training/output/model.tar.gz" } } ``` And submit a batch transform job (`create-transform-job`) using the following configuration: ``` { "MaxPayloadInMB": 16, "ModelName": "model-test", "TransformInput": { "ContentType": "application/x-image", "DataSource": { "S3DataSource": { "S3DataType": "ManifestFile", "S3Uri": "s3://[redacted]/manifests/input.manifest" } } }, "TransformJobName": "transform-test", "TransformOutput": { "S3OutputPath": "s3://[redacted]/predictions/" }, "TransformResources": { "InstanceCount": 1, "InstanceType": "ml.m5.large" } } ``` Both of the above commands submit fine, but the transform job doesn't complete. When I look in the logs, the errors I'm getting seem to indicate that it's not using my inference script (`inference.py`, specified above) but is instead using the default script (`default_pytorch_inference_handler.py`) and therefore can't find the model. What am I missing so that it uses my inference script instead, and hence my model?

For our Sagemaker Notebook/studio deployments we have used lifecycle configs to turn off the app when inactive. Is this possible to apply to Canvas? Is this a supported function? is there an alternative?

Hello, I am trying to start Amazon SageMaker Studio today but without success. The default server is failing to start. On CloudWatch I only see SIGKILL messages as errors: ``` 2022-06-07T10:55:01.523+02:00 2022-06-07 08:55:00,466 WARN killing 'jupyterlabserver' (11) with SIGKILL 2022-06-07T10:55:04.524+02:00 2022-06-07 08:55:01,470 INFO waiting for jupyterlabserver-listener, jupyterlabserver to die ``` On the Apps section I see the default server in status "Failed", and I cannot do anything, not even delete it. I cannot use the service right now. Is it maybe related to the new release of Jupyter Lab v3.0? Thanks

I have a modest size dataset, and I am running Jupyter Notebook in Sagemaker (instance type ml.c5.xlarge with 200G instance size). I receive the error message " GC overhead limit exceeded" Everything ran fine with small data size. BTW, I need to go through the dataframe one row at a time using df.collect(), which seems t be an expensive operation... Would you suggest another way of accomplishing this? I would appreciate your kind help.

I am trying to run TensorBoard 2.9.0 in SageMaker Studio, as described [here](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-tensorboard.html#studio-tensorboard-launch). However, after launching TensorBoard, when I navigate to the proxy URL `https://<DOMAIN>.studio.<REGION>.sagemaker.aws/jupyter/default/proxy/6006/`, I get error 500. I am using the "PyTorch 1.10 Python 3.8 CPU Optimized" image. Any suggestions? I have also tried with the new JupyterLab 3 server version, but same result.

Hi all! So, we're trying to implement a very simple Sagemaker Pipeline with 3 steps: * **ETL:** for now it only runs a simple query * **Batch transform:** uses the ETL's result and generates predictions with a batch transform job * **Report:** generates an HTML report The thing is, when running the batch transform job alone in the Pipeline, everything runs OK. But when trying to run all the steps in a Pipeline, the batch transform job fails, and what we have seen in the logs is that the job takes the dataset which was generated in the ETL step, generates the predictions and saves them correctly in S3 (this is where we would expect the job to stop) but then it resends those predictions to the endpoint, as if they were a new input, and so the step fails as the model receives an array of 1 column thus mismatching the number of features which it was trained with. There's not much info out there on this, and Sagemaker is painfully hard to debug. Has anyone experienced anything like this? Our model and transformer code: ```python model = XGBoostModel( model_data=f"s3://{BUCKET}/{MODEL_ARTIFACTS_PATH}/artifacts.gzip", role=get_execution_role(), entry_point="predict.py", framework_version="1.3-1", ) transformer = model.transformer( instance_count=1, instance_type="ml.m5.large", output_path=f"s3://{BUCKET}/{PREDICTIONS_PATH}/", accept="text/csv", ) step = TransformStep( name="Batch", transformer=transformer, inputs=TransformInput( data=etl_step.properties.ProcessingOutputConfig.Outputs[ "dataset" ].S3Output.S3Uri, content_type="text/csv", split_type="Line", ), depends_on=[etl_step], ) ``` And our inference script: ```python def input_fn(request_body, content_type): return pd.read_csv(StringIO(request_body), header=None).values def predict_fn(input_obj, model): """ Function which takes the result of input_fn and generates predictions. """ return model.predict_proba(input_obj)[:, 1] def output_fn(predictions, content_type): return ",".join(str(pred) for pred in predictions) ```

i'm using a hugging face model and a container to create a batch transform job in sagemaker. i have a custom inference code and in the output_fn function i'm returning json_dumps(prediction). I'm using print(prediction) to see, if i can see it in the cloudwatch logs to find out type and what prediction is. how can log these messages . Also, the inference output i get is in the form below., i'm not sure why is it not a json object in each line instead it has square brackets. I want to use the filter to match the input and output in the batch job. I'm not sure how the output should look like , because i'm trying to associate input with output by using dataprocessing config as below. but i get an error. the documenation has example of csv not json. what should the output look like so that i can associate the input with output when they both are in json format. ``` "DataProcessing": { "JoinSource": "Input" }, ``` ``` [ output text 1 ] [output text 2 ] ``` ``` # Serialize the prediction result into the desired response content type def output_fn(prediction, accept=JSON_CONTENT_TYPE): logger.info("Serializing the generated output.") if accept == JSON_CONTENT_TYPE: output = json.dumps(prediction) return output, accept raise Exception("Requested unsupported ContentType in Accept: {}".format(accept)) ```

I am working on a use-case where I am using SageMaker multi-model endpoint for model inference and the models are trained using Databricks MLFlow platform. When I tried deploying a model trained from Databricks MLFlow platform on a single endpoint on SageMaker then it worked fine but the creation of multi-model endpoint for 'sagemaker-scikit-learn:0.23-1-cpu-py3' container is failed with the following error: Code Snippet::>> name = "sample-mme" sagemaker_client = boto3.client('sagemaker') model_path = "s3://test-bucket/multi-models" execution_role_arn = "IAM://sample-role" BASE_IMAGE = image_uris.retrieve( region=region, framework="sklearn",version='0.23-1',image_scope='inference' ) container = { 'Image': BASE_IMAGE, 'ModelDataUrl': model_path, 'Mode': 'MultiModel', 'MultiModelConfig': { 'ModelCacheSetting': 'Enabled' } } model_response = sagemaker_client.create_model( ModelName=name, ExecutionRoleArn=execution_role_arn, Containers=[container] ) config_response = sagemaker_client.create_endpoint_config( EndpointConfigName=f'{name}-config', ProductionVariants=[ { 'InstanceType': instance_type, 'InitialInstanceCount': instance_count, 'InitialVariantWeight': 1, 'ModelName': name, 'VariantName': 'AllTraffic' } ] ) response = sagemaker_client.create_endpoint( EndpointName=f'{name}-endpoint', EndpointConfigName=f'{name}-config' ) Endpoint creation is taking a lot if time and failing with the following error message : sagemaker_containers._errors.ImportModuleError: 'NoneType' object has no attribute 'startswith' Please provide me with some help to fix this. Also, my understanding is that I can train a model on the DataBricks MLFlow platform using sklearn libraries, and then I can store model artifacts "model.tar.gz" under the s3 directory for storing all multi-models. Now I can create a multi-model endpoint in SageMaker using the same s3 directory as the model path and using the above code. Once the endpoint is ready, I can do inference by providing the target model. Please let me know if my understanding is correct and share any relevant documents to follow for my use case.

I have tried Data Wrangler in Sagemaker last month and close the service. A few weeks later I have noticed the credit was charge $1 every hour and just realized that the Data Wranger auto-save the flow every minute. So, I deleted the unsaved flow and shut down all the services and instances according to advice on these two links : * https://docs.aws.amazon.com/sagemaker/latest/dg/studio-lab-use-shutdown.html * https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-cleanup.html Then, I left the Sagemaker untouched for the whole month of May, and just got back to the console yesterday. This is what I found out for May's bill: Amazon SageMaker RunInstance $531.74 == | Detail | Usage | Total | | --- | --- | --- | | $0.00 for Host:ml.m5.xlarge per hour under monthly free tier | 125.000 Hrs | $0.00 | | $0.00 for Notebk:ml.t2.medium per hour under monthly free tier | 107.056 Hrs | $0.00 | | $0.00 per Data Wrangler Interactive ml.m5.4xlarge hour under monthly free tier | 25.000 Hrs | $0.00 | | $0.23 per Hosting ml.m5.xlarge hour in US East (N. Virginia) | 88.997 Hrs | $20.47 | | $0.922 per Data Wrangler Interactive ml.m5.4xlarge hour in US East (N. Virginia) | 554.521 Hrs | $511.27 | So, with another attempt, I installed an extension to automatically shut down idle kernels and set the limit to 10 min from advice here: https://aws.amazon.com/blogs/machine-learning/save-costs-by-automatically-shutting-down-idle-resources-within-amazon-sagemaker-studio/ Checked the cost in usage report, it turns out that the service was shut down after installing the extension but then it revoked itself after 5 hours later (during my sleep time). There's still cost from Studio although with less charge than previous one. | Service | Operation | UsageType | StartTime | EndTime |UsageValue | | --- | --- | --- | --- | --- | --- | | AmazonSageMaker | RunInstance | USE1-Studio_DW:KernelGateway-ml.m5.4xlarge | 5/24/2022 23:00 | 5/25/2022 0:00 | 1 | | AmazonSageMaker | RunInstance | USE1-Studio_DW:KernelGateway-ml.m5.4xlarge | 5/25/2022 0:00 | 5/25/2022 1:00 | 1 | | AmazonSageMaker | RunInstance | USE1-Studio_DW:KernelGateway-ml.m5.4xlarge | 5/25/2022 1:00 | 5/25/2022 2:00 | 1 | | AmazonSageMaker | RunInstance | USE1-Studio_DW:KernelGateway-ml.m5.4xlarge | 5/25/2022 2:00 | 5/25/2022 3:00 | 0.76484417 | | AmazonSageMaker | RunInstance | USE1-Studio_DW:KernelGateway-ml.m5.4xlarge | 5/25/2022 8:00 | 5/25/2022 9:00 | 0.36636722 | | AmazonSageMaker | RunInstance | USE1-Studio_DW:KernelGateway-ml.m5.4xlarge | 5/25/2022 9:00 | 5/25/2022 10:00 | 0.38959556 | During this time, I'm sure that there're no running instances, running apps, kernel sessions or terminal sessions. I even deleted the user profile. Last thing I haven't tried is to set up scheduled shutdown coz I think the services should not cause difficulty to our life that much. Any advice for any effective action to completely shutdown the Sagemaker instance? Thanks.

I previously ran a hyperparameter tuning job for SageMaker DeepAR with the instance type ml.c5.18xlarge but it seems insufficient to complete the tuning job within the max_run time specified in my account. Now, having tried to use the accelerated GPU instance ml.g4dn.16xlarge, I am prompted with an error - "Instance type ml.g4dn.16xlarge is not supported by algorithm forecasting-deepar." I cannot find any documentation that indicates the list of instance types supported by deepar. What GPU/CPU instances have more compute capacity than ml.c5.18xlarge which I could leverage for my tuning job? If there isn't, I would appreciate any recommendations as to how I could hasten the run time of the job. I require the tuning job to complete within the max run time of 432000 seconds. Thank you in advance!

Which XGBoost hypertuning perameter is used to set the forecast horizon? I do not see a parameter listed here: https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost_hyperparameters.html

I am using the code below that I took from an example [https://aws.amazon.com/pt/blogs/machine-learning/automatically-extract-text-and-structured-data-from-documents-with-amazon-textract/](), in the example it is used only for a case of 2 columns, in the code where there is division by 2, if my file has 4 columns for example, I just change that it works. But how to detect the amount of columns automatically or some way that I don't need this manual input anymore? In summary I want to use this code for cases of pdf files that have more than 2 columns, how to do it? ``` import boto3 # Document s3BucketName = "amazon-textract-public-content" documentName = "blogs/two-column-image.jpg" # Amazon Textract client textract = boto3.client('textract') # Call Amazon Textract response = textract.detect_document_text( Document={ 'S3Object': { 'Bucket': s3BucketName, 'Name': documentName } }) #print(response) # Detect columns and print lines columns = [] lines = [] for item in response["Blocks"]: if item["BlockType"] == "LINE": column_found=False for index, column in enumerate(columns): bbox_left = item["Geometry"]["BoundingBox"]["Left"] bbox_right = item["Geometry"]["BoundingBox"]["Left"] + item["Geometry"]["BoundingBox"]["Width"] bbox_centre = item["Geometry"]["BoundingBox"]["Left"] + item["Geometry"]["BoundingBox"]["Width"]/2 column_centre = column['left'] + column['right']/2 if (bbox_centre > column['left'] and bbox_centre < column['right']) or (column_centre > bbox_left and column_centre < bbox_right): #Bbox appears inside the column lines.append([index, item["Text"]]) column_found=True break if not column_found: columns.append({'left':item["Geometry"]["BoundingBox"]["Left"], 'right':item["Geometry"]["BoundingBox"]["Left"] + item["Geometry"]["BoundingBox"]["Width"]}) lines.append([len(columns)-1, item["Text"]]) lines.sort(key=lambda x: x[0]) for line in lines: print (line[1]) ```

We deployed a LighGBM Regression model and endpoint using Sagemaker Jumpstart. We have attempted to configure this endpoint as 'asynchronous' via the console. Receiving Error: ValidationException-Network Isolation is not supported when specifying an AsyncInferenceConfig. Looking at the model's network details the model has Enable Network Isolation set as 'True'. This was default output setting set by JumpStart. How can we diasble Network Isolation to in order to make this endpoint asynchronous?

I encountered following erorr when I tried to create a new project in SageMaker Studio. I tried to add `AdminAccess` to `role/service-role/AmazonSageMakerServiceCatalogProductsLaunchRole`, but still doesn't help. What could be the problem? ``` Error getting the details of Service Catalog Provisioning Parameters. Error message: ValidationException: Access denied while assuming the role arn:aws:iam::XXXXXXXXX:role/service-role/AmazonSageMakerServiceCatalogProductsLaunchRole. Args: {"productId":"prod-xxxxxxxx","provisioningArtifactId":"pa-xxxxxxxxx","pathId":"lpv2-xxxxxxxxxx"} ``` [printscreen](https://raw.githubusercontent.com/qinjie/picgo-images/main/qerpoijsdfijsadfjaosdjfqwefasdfasdfasdfasdf.png)

Hi, based on https://github.com/aws/amazon-sagemaker-examples/blob/main/advanced_functionality/scikit_bring_your_own/scikit_bring_your_own.ipynb I've created a docker image which contains the model code implemented in Python. This model image is deployed as async inference endpoint. Everything works well, the endpoint can be invoked via InvokeEndpointAsync, custom attributes are provided to the model code via "X-Amzn-SageMaker-Custom-Attributes" header, the input located on S3 is provided as http body. After succesful processing the result is uploaded to S3 with an automatically generated output filename ("*.out"). The container behavior is logged to CloudWatch via Python logging module, initialized as: ``` import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger() ``` However, the request arriving inside the container neither contains an "X-Amzn-SageMaker-Inference-Id" nor "X-Amzn-SageMaker-InputLocation" field (mentioned here https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_runtime_InvokeEndpointAsync.html). Is there a way to access the inference_id/output filename from inside the container? Or any other opportunity to link input filename/container log/output filename? Otherwise it's not possible to fully track and examine requests. Thanks in advance for your advice!

I work in SM Studio, and I do not understand why CPU and memory usage do not appear in the notebook toolbar. These metrics should be there, at least given this description: https://docs.amazonaws.cn/en_us/sagemaker/latest/dg/notebooks-menu.html When I open a notebook in SM Studio, I see the same toolbar but without CPU and memory usage listed. Moreover, I see 'cluster' before the kernel's name in my toolbar. Has anyone experienced sth similar? I assume an alternative for me would be to use CloudWatch.

Good day! **My main purpose:** Easy way to collect information about different failure scenarios in SageMaker TrainingJob. **What do I use currently?** Sagemaker SKLearn Estimator(TrainingJobs are inside) **Where will my model train?** Different datasets. So, I need control and collect all information about all training processes and their final statuses on different datasets. **Which failure scenarios do I have?** There are plenty of them. I have create my own python Errors for them. For example: 1. There are labels only for one class. 2. Too small dataset(by my own criterions) 3. Missing data for crucial columns 4. e.t.c. **Where am I stuck?** After failed training I can't get own errors from training job response. All of them are "ExecuteUserScriptError" I can't pass my own info in FailureReason or ErrorMessage(always it's empty). I see which error was raised in CloudWatchLogs and TrainingJobTraceback(from SagemakerNotebook). So, bad solution is parse all CloudWatchLogs in case of failure. **Question: How to provide my own ErrorMessage or FailureReason? ** May be I am digging in the wrong direction. Anyway, I need your advice. Thank you so much for possibility to ask an advice here)

I am building a ml workflow using step function following [this](https://sagemaker-examples.readthedocs.io/en/latest/step-functions-data-science-sdk/machine_learning_workflow_abalone/machine_learning_workflow_abalone.html). However, when I start the state machine, I got error ``` AlgorithmError: framework error ... SM_CHANNEL_TRAIN ...exit code: 1 ``` Does anyone know how to fix it? or how to set SM_CHANNEL_TRAIN? Thank you

I am exploring the Sagemaker Built-in algorithms, and I am curious to learn more about the details of the algorithms. However, I am surprised that it is hard to find any references for the research background and implementation details in the numerous documents and tutorials for particular algorithms. If such information exists somewhere, I would highly appreciate a pointer. Thanks a lot in advance!

Hi Dears, I am building ML model using DeepAR Algorithm. I faced this error while i reached to this point : Error : ClientError: An error occurred (UnknownOperationException) when calling the CreateHyperParameterTuningJob operation: The requested operation is not supported in the called region. ------------------- Code: from sagemaker.tuner import ( IntegerParameter, CategoricalParameter, ContinuousParameter, HyperparameterTuner, ) from sagemaker import image_uris container = image_uris.retrieve(region= 'af-south-1', framework="forecasting-deepar") deepar = sagemaker.estimator.Estimator( container, role, instance_count=1, instance_type="ml.m5.2xlarge", use_spot_instances=True, # use spot instances max_run=1800, # max training time in seconds max_wait=1800, # seconds to wait for spot instance output_path="s3://{}/{}".format(bucket, output_path), sagemaker_session=sess, ) freq = "D" context_length = 300 deepar.set_hyperparameters( time_freq=freq, context_length=str(context_length), prediction_length=str(prediction_length) ) Can you please help in solving the error? I have to do that in af-south-1 region. Thanks Basem hyperparameter_ranges = { "mini_batch_size": IntegerParameter(100, 400), "epochs": IntegerParameter(200, 400), "num_cells": IntegerParameter(30, 100), "likelihood": CategoricalParameter(["negative-binomial", "student-T"]), "learning_rate": ContinuousParameter(0.0001, 0.1), } objective_metric_name = "test:RMSE" tuner = HyperparameterTuner( deepar, objective_metric_name, hyperparameter_ranges, max_jobs=10, strategy="Bayesian", objective_type="Minimize", max_parallel_jobs=10, early_stopping_type="Auto", ) s3_input_train = sagemaker.inputs.TrainingInput( s3_data="s3://{}/{}/train/".format(bucket, prefix), content_type="json" ) s3_input_test = sagemaker.inputs.TrainingInput( s3_data="s3://{}/{}/test/".format(bucket, prefix), content_type="json" ) tuner.fit({"train": s3_input_train, "test": s3_input_test}, include_cls_metadata=False) tuner.wait()

My IAM works, but when I login, it logs me into us-east-1. My project is on the S3 instance (sagemaker tool). How can I access buckets in S3 if I am logged into us-east-1 by default.

Hi all, I am asking if it's possible to use `framework processor` inside a `sagemaker pipeline`. I am asking because the to submit the source_dir for the framework processor, we have to do so when calling the .run() method, when wrapping the processor inside a `sagemaker.workflow.steps.ProcessingStep`, there isn't an available argument to specify the `source_dir`. Thank you! Best, Ruoy

Hi I received my aws sagemaker studio approval to create an account 1 hr ago When I go to the link to create an account it says my email has not been approved Even though I have an email to the contrary How do I contact amazon to sort this out?

Hi all, I've followed the guide provided in this medium blog by senior AWS ML SA: https://towardsdatascience.com/hosting-vs-code-in-sagemaker-studio-f211385e25f7 I've created the lifecycle configuration (jupyterlab app) for the following bash script: ``` #!/bin/bash mkdir -p vscode cd vscode sudo su curl -fsSL https://code-server.dev/install.sh | sh exit ``` However, when I attach the configuration to the domain or a specific user's jupyter server in the SM Studio, the JupyterServer App fails to be created with the following message displayed on the loading page of SM Studio's jupyterlab: > "The JupyterServer app default encountered a problem and was stopped.Details: ConfigurationError: LifecycleConfig execution failed with non zero exit code 1 for script arn:aws:sagemaker:eu-west-3:615740825886:studio-lifecycle-config/install-vscode-on-jupyterserver. Check https://docs.aws.amazon.com/sagemaker/latest/dg/studio-lcc-debug.html for debugging instructions." I checked the page for debugging instructions, specifically the part where I am instructed to check the cloudwatch logs for the log group: aws/sagemaker/studio The problem is this log group does not exist in CloudWatch (I have admin permission). I would like to have this set up for the entire team, Any help would be greatly appreciated! Best, Ruoy

Overview: I'm trying to run an XGboost model on a bunch of parquet files sitting in S3 using dask by setting up a fargate cluster and connecting it to a Dask cluster. Total dataframe size totals to about 140 GB of data. I scaled up a fargate cluster with properties: Workers: 40 Total threads: 160 Total memory: 1 TB So there should be enough data to hold the data tasks. Each worker has 9+ GB with 4 Threads. I do some very basic preprocessing and then I create a DaskDMatrix which does cause the task bytes per worker to get a little high, but never above the threshold where it would fail. Next I run xgb.dask.train which utilizes the xgboost package not the dask_ml.xgboost package. Very quickly, the workers die and I get the error `XGBoostError: rabit/internal/utils.h:90: Allreduce failed`. When I attempted this with a single file with only 17MB of data, I would still get this error but only a couple workers die. Does anyone know why this happens since I have double the memory of the dataframe? ``` X_train = X_train.to_dask_array() X_test = X_test.to_dask_array() y_train = y_train y_test = y_test ``` dtrain = xgb.dask.DaskDMatrix(client,X_train, y_train) output = xgb.dask.train( client, {"verbosity": 1, "tree_method": "hist", "objective": "reg:squarederror"}, dtrain, num_boost_round=100, evals=[(dtrain, "train")])`

Hi, I have created and endpoint in sagemaker using boto3 but never finishes creation, is stuck in Creating status for few days now. I have tried to delete it using the aws cli api but i get the message: *An error occurred (ValidationException) when calling the DeleteEndpoint operation: Cannot update in-progress endpoint* Usually endpoint fails after some time and can deleted but this time doesn't fail. Is there any way to force deletion?

Amazon SageMaker Data Wrangler reduces the time it takes to aggregate and prepare data for machine learning (ML) from weeks to minutes in Amazon SageMaker Studio, the first fully integrated development environment (IDE) for ML. With SageMaker Data Wrangler, you can simplify the process of data preparation and feature engineering, and complete each step of the data preparation workflow, including data selection, cleansing, exploration, and visualization, from a single visual interface. SageMaker Data Wrangler runs on ml.m5.4xlarge by default. SageMaker Data Wrangler includes built-in data transforms and analyses written in PySpark so you can process large data sets (up to hundreds of gigabytes (GB) of data) efficiently on the default instance. Starting today, you can use additional M5 or R5 instance types with more CPU or memory in SageMaker Data Wrangler to improve performance for your data preparation workloads. Amazon EC2 M5 instances offer a balance of compute, memory, and networking resources for a broad range of workloads. Amazon EC2 R5 instances are the memory optimized instances. Both M5 and R5 instance types are well suited for CPU and memory intensive applications such as running built-in transforms for very large data sets (up to terabytes (TB) of data) or applying custom transforms written in Panda on medium data sets (up to tens of GBs). To learn more about the newly supported instances with Amazon SageMaker Data Wrangler, visit the [blog ](https://aws.amazon.com/blogs/machine-learning/process-larger-and-wider-datasets-with-amazon-sagemaker-data-wrangler/) or the [AWS document](https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler-data-flow.html), and the[ pricing page](https://aws.amazon.com/sagemaker/pricing/). To get started with SageMaker Data Wrangler, visit the [AWS documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/data-wrangler.html).

I've got two CSV files that are loaded into Data Wrangler that are intended to augment each other. The tables have some columns that are the same (in name) and some that are not, many of the rows are missing entries for many of the columns. The two tables represent separate datasets. Consider the example below: Table 1: | Filename | LabelA | LabelB | | --- | --- | --- | | ./A/001.dat | 1 | 1 | | ./A/002.dat | 0 | 1 | Table 2: | Filename | LabelB | LabelC | | --- | --- | --- | | ./B/001.dat | | 0 | | ./B/002.dat | 0 | 1 | I am looking to merge / concatenate the two table. The problem is that neither Data Wrangler join nor concatenate seems to work (at least as expected). Desired result: | Filename | LabelA | LabelB | LabelC | | --- | --- | --- | --- | | ./A/001.dat | 1 | 1 | | | ./A/002.dat | 0 | 1 | | | ./B/001.dat | | | 0 | | ./B/002.dat | | 0 | 1 | When using a "Full Outer" join and ask to combine "Filename" and "LabelB" columns, it will take all the values from Table 1 OR Table 2 even if Table 1 does not have that entry (for example, some rows will have Filename = <nothing> rather than Filename = ./B/001.dat). When using concatenate, Data Wrangler errors on the fact that it cannot match EVERY column between the tables. Now in my example there are many columns and many rows which precludes a manual process of joining without merging columns and then going through a renaming and merging process one-by-one. How do get these tables to simply merge? I feel I must be missing something obvious. I am about to give up on Data Wrangler and do it all in a python script using pandas, but I thought I should give Data Wrangler a try while learning the MLops process.

I am using AWS Sagemaker services and delete any instances that belong to the services. However, i still get charged per day even though I dont use the service anymore. I sent the case to the customer support center but no clear explanation yet. Please any kind of help for my account to stop the cost.

I am using AWS Sagemaker services and delete any instances that belong to the services. However, i still get charged per day even though I dont use the service anymore. I sent the case to the customer support center but no clear explanation yet. Please any kind of help for my account to stop the cost.

How can you use the functions of SageMaker Canvas with python or outside the user interface?

I run neural network tensorflow train on studiolab. and I got: ``` Epoch 145/4000 1941/1941 - 10s - ... - 10s/epoch - 5ms/step ``` then I try to make a train job with script_mode with `ml.c5.xlarge` ``` estimator = TensorFlow(entry_point='untitled.py', source_dir='./training/', instance_type='ml.c5.xlarge', instance_count=1, output_path="s3://sagemaker-[skip]", role=sagemaker.get_execution_role(), framework_version='2.8.0', py_version='py39', hyperparameters={...}, metric_definitions=[...], script_mode=True) ``` and its got: ``` Epoch 19/4000 1941/1941 - 49s - ... - 49s/epoch - 25ms/step ``` Why is it 5 times slower than studiolab notebook? Is it because instance type?

In a sagemaker notebook with an associated git repository, when I try to create a boto3 session client using verify = False, I get the following : `SSLError: SSL validation failed for {service_name } [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)` The error resolves if I allow the default value of verify = None (meaning SSL certs are verified). My problem is that this session is created as part of a function call on the associated git repository and I don't want to change the behavior in the repo. I don't understand why this error occurs only when I specify *not* to validate SSL certificates. Any ideas of what explains this behavior?

Hello, I'd like to deploy SageMaker's built-in algorithm, BlazingText model on Fargate instead of Sagemaker endpoint. So, I tried to make an ECS task using BlazingText docker path. Here is my CDK code for it. const loadBalancedFargateService = new ecsPatterns.ApplicationLoadBalancedFargateService(this, 'Service', { memoryLimitMiB: 1024, desiredCount: 1, cpu: 512, taskImageOptions: { image: ecs.ContainerImage.fromRegistry("811284229777.dkr.ecr.us-east-1.amazonaws.com/blazingtext:1"), }, }); However, I got an error: CannotPullContainerError: inspect image has been retried 1 time(s): failed to resolve ref "811284229777.dkr.ecr.us-east-1.amazonaws.com/blazingtext:1": pulling from host 811284229777.dkr.ecr.us-east-1.amazonaws.com failed with status code [manifests 1]... Is it impossible to pull docker container of sagemaker built-in algorithm from ECS?

I have a private GitLab and would like to connect it to Sagemaker Studio and/or Notebooks. Is that supported? What is it is through a VPC? If so, how can it be done?

I have a sagemaker notebook instance having two jupyter notebook ipynb files. When I had one jupyter notebook, I was able to run it automatically with one lambda function trigger and lifecycle configuration. Now I have two jupyter notebooks and corresponding two lambda function triggers. How can I run them based on the trigger by changing the lifecycle configuration script. The trigger is file uploading into S3. Based on what location the file is added, the corresponding jupyter notebook should run

I have a sagemaker notebook instance having two jupyter notebook ipynb files. When I had one jupyter notebook, I was able to run it automatically with one lambda function trigger and lifecycle configuration. Now I have two jupyter notebooks and corresponding two lambda function triggers. How can I run them based on the trigger by changing the lifecycle configuration script.

We are trying to deploy a Lambda with some code which works in a Notebook. The code is rather simple and uses D-Wave — DW_2000Q_6. The problem is that when we execute the lambda (container lambda due to size problems), it give us the following error: ```json { "errorMessage": "[Errno 30] Read-only file system: '/home/sbx_user1051'", "errorType": "OSError", "stackTrace": [ " File \"/var/lang/lib/python3.8/imp.py\", line 234, in load_module\n return load_source(name, filename, file)\n", " File \"/var/lang/lib/python3.8/imp.py\", line 171, in load_source\n module = _load(spec)\n", " File \"<frozen importlib._bootstrap>\", line 702, in _load\n", " File \"<frozen importlib._bootstrap>\", line 671, in _load_unlocked\n", " File \"<frozen importlib._bootstrap_external>\", line 843, in exec_module\n", " File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n", " File \"/var/task/lambda_function.py\", line 6, in <module>\n from dwave.system.composites import EmbeddingComposite\n", " File \"/var/task/dwave/system/__init__.py\", line 15, in <module>\n import dwave.system.flux_bias_offsets\n", " File \"/var/task/dwave/system/flux_bias_offsets.py\", line 22, in <module>\n from dwave.system.samplers.dwave_sampler import DWaveSampler\n", " File \"/var/task/dwave/system/samplers/__init__.py\", line 15, in <module>\n from dwave.system.samplers.clique import *\n", " File \"/var/task/dwave/system/samplers/clique.py\", line 32, in <module>\n from dwave.system.samplers.dwave_sampler import DWaveSampler, _failover\n", " File \"/var/task/dwave/system/samplers/dwave_sampler.py\", line 31, in <module>\n from dwave.cloud import Client\n", " File \"/var/task/dwave/cloud/__init__.py\", line 21, in <module>\n from dwave.cloud.client import Client\n", " File \"/var/task/dwave/cloud/client/__init__.py\", line 17, in <module>\n from dwave.cloud.client.base import Client\n", " File \"/var/task/dwave/cloud/client/base.py\", line 89, in <module>\n class Client(object):\n", " File \"/var/task/dwave/cloud/client/base.py\", line 736, in Client\n @cached.ondisk(maxage=_REGIONS_CACHE_MAXAGE)\n", " File \"/var/task/dwave/cloud/utils.py\", line 477, in ondisk\n directory = kwargs.pop('directory', get_cache_dir())\n", " File \"/var/task/dwave/cloud/config.py\", line 455, in get_cache_dir\n return homebase.user_cache_dir(\n", " File \"/var/task/homebase/homebase.py\", line 150, in user_cache_dir\n return _get_folder(True, _FolderTypes.cache, app_name, app_author, version, False, use_virtualenv, create)[0]\n", " File \"/var/task/homebase/homebase.py\", line 430, in _get_folder\n os.makedirs(final_path)\n", " File \"/var/lang/lib/python3.8/os.py\", line 213, in makedirs\n makedirs(head, exist_ok=exist_ok)\n", " File \"/var/lang/lib/python3.8/os.py\", line 213, in makedirs\n makedirs(head, exist_ok=exist_ok)\n", " File \"/var/lang/lib/python3.8/os.py\", line 223, in makedirs\n mkdir(name, mode)\n" ] } ``` It seems that the library tries to write to some files which are not in /tmp folder. I'm wondering if is possible to do this, and if not, what are the alternatives. imports used: ```python import boto3 from braket.ocean_plugin import BraketDWaveSampler from dwave.system.composites import EmbeddingComposite from neal import SimulatedAnnealingSampler ```

Note: I am a senior developer, but am very new to the topic of machine learning. I have two frozen TensorFlow model weight files: `weights_face_v1.0.0.pb` and `weights_plate_v1.0.0.pb`. I also have some python code using Tensorflow 2, that loads the model and handles basic inference. The models detect respectively faces and license plates, and the surrounding code converts an input image to a numpy array, and applies blurring to the images in areas that had detections. I want to get a SageMaker endpoint so that I can run inference on the model. I initially tried using a regular Lambda function (container based), but that is too slow for our use case. A SageMaker endpoint should give us GPU inference, which should be much faster. I am struggling to find out how to do this. From what I can tell reading the documentation and watching some YouTube video's, I need to create my own docker container. As a start, I can use for example `763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-inference:2.8.0-gpu-py39-cu112-ubuntu20.04-sagemaker`. However, I can't find any solid documentation on how I would implement my other code. How do I send an image to SageMaker? Who tells it to convert the image to numpy array? How does it know the tensor names? How do I install additional requirements? How can I use the detections to apply blurring on the image, and how can I return the result image? Can someone here please point me in the right direction? I searched a lot but can't find any example code or blogs that explain this process. Thank you in advance! Your help is much appreciated.

I am having difficulty in getting started on SageMaker Studio using the Standard setup wizard of SageMaker Domain creation. The wizard prevents me from moving forward at the RStudio setup step even though RStudio is indicated to be optional and which I don't need or want to pay for. I have tried a lot of things (looked at the videos, searched the web) to no avail. Tried as a root user as well as a IAM user and have the same issue. Thanks for any help.

Hello We are using Greengrass v2 with the Sagemaker component in order to do inference on the edge. The following error is seen intermettently while running our application. Could you please advise of anything we can do to avoid it? File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 946, in __call__ return _end_unary_response_blocking(state, call, False, None) File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking raise _InactiveRpcError(state) grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "Socket closed" debug_error_string = "{"created":"@1650459124.460491853","description":"Error received from peer unix:/tmp/aws.greengrass.SageMakerEdgeManager.sock","file":"src/core/lib/surface/call.cc","file_line":906,"grpc_message":"Socket closed","grpc_status":14}"

I am trying to install a python package called rapidsai in aws notebook instance but I am getting this error errno 28 no space left on disk. I am using g4dn.xlarge instance for it. The package is almost 3 GB. While opening the notebook instance I have tried increasing volume size of notebook upto 50 GB, I am still getting this error. Let me know the solution to it. Thanks

what other data input types can be used pipe input mode in sagemaker? an example of implementation is here https://aws.amazon.com/blogs/aws/sagemaker-nysummit2018/, and can this be used for inference as well as, similar to training?

looking at few examples, for training in sagemaker . are there some guidelines based on the model size, data to be trained , what type of instance cpu/gpu to use? also, can one use spot instances ( may be with multiple gpu cores)?

We are developing a custom model monitoring container to be used to interact with Sagemaker's model monitoring API, as we require additional custom metrics. One of our requirements is for these metrics to populate and visualize appropriately in Studio visualization tabs, e.g. Data Quality, Model Quality, and Explainability. **None of the built-in containers are an option for us, and we are trying to interop with the Monitoring APIs as seamlessly as possible.** The docs specify the container contracts for output, specifically *statistics.json*, *constraints.json*, and *constraint_violations.json*. However, the docs are not clear on what JSON files to emit for specifically *custom Model Quality metrics*. There does not appear to be a provided schema. Things I have tried: - Emitting CloudWatch metrics to a different namespace, *sagemaker/Endpoint/model-quality-metrics* - Attempting to retrofit the statistics.json file to also include Model Quality metrics. Is there a separate JSON file I should write to */opt/ml/processing/resultdata* for our custom model quality metrics to populate the Studio visualization tab? Specifically, where do the base containers look for this JSON report: https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality-metrics.html

Hello, I have followed these docs https://docs.aws.amazon.com/comprehend/latest/dg/cer-annotation-pdf.html and have gotten to the point in which I have created the annotation task and I have uploaded several pdf's to a s3 bucket to be used for an annotation task so I can create a comprehend model. I put myself and a co-worker as annotators just so I can verify that I can set up the task properly and I only uploaded 37 pdf's. However when both of us log in and start the task, the webpage loads as the instructions tell us to however there is no pdf rendered (though I think I see it briefly flash on the screen before it goes blank) and there are also no entities to be selected as a part of the ui unlike how the documentation pictures the tool. I am trying to do named entity recognition and created this task with the full 25 entities I want to be able to label and Also another time with only 5 entities to label. However there seems to be something wrong with this native pdf annotation feature.

I have a customer already standardized Artifactory as the centralized image registration. They disabled ECR service at Org level. Now we want to understand the potential impact on customer's day-2-day use of SageMaker Studio as a platform to support their full ML lifecycle. (If customer only use built-in SageMaker algorithm or framework and use prebuilt SageMaker container images) Especially when customer trying to deploy the model to endpoint, does that need ECR service to be enabled in customer account?

Hi, I'm trying to run a sagemaker job on p3.2xlarge instance using the PyTorch estimator. My script has dependencies on several packages (possibly not pre-installed on the instance) for which I have a requirements.txt file. After loading the instance, it installs several packages but the job fails with following error: **sagemaker-training-toolkit ERROR InstallRequirementsError: Command "/opt/conda/bin/python3.6 -m pip install -r requirements.txt" ERROR: Could not find a version that satisfies the requirement scikit_image==0.18.3 ERROR: No matching distribution found for scikit_image==0.18.3** It fails for other packages too, like h5py, numpy etc. Any help is greatly appreciated. Thank you, Aditya

Does Data Capture feature used for model monitor and analytics work with the multi model endpoint (one container).. we ran into an error. See error " An error occurred (ValidationException) when calling the CreateEndPointConfig operation: Data Capture Feature is not supported with MultiModel mode" Theoretically, it should work because it is calling the DataCaptureConfig: from sagemaker.model_monitor import DataCaptureConfig endpoint_name = 'your-pred-model-monitor-' + strftime("%Y-%m-%d-%H-%M-%S", gmtime()) print("EndpointName={}".format(endpoint_name)) data_capture_config=DataCaptureConfig( enable_capture = True, sampling_percentage=100, destination_s3_uri=s3_capture_upload_path)

Can Sagemaker Git Repositories use ssh secrets (no name and password)?

I am testing inference in sagemaker , by using one of the container listed here -> https://github.com/aws/deep-learning-containers/blob/master/available_images.md. the model is zipped up as below and with in inference.py file , i am overwriting functions like model_fn method and predict_fn. I tested this with batch transform and it worked but for few small input files but for other larger files, i keep getting "Model server did not respond to /invocations request within 3600 seconds" . I'm trying to find out what is the cause of it? 3600 is the max we can set for "invocation timeout in seconds" parameter and the default input size for batch is 6mb , the input files i'm using are way smaller than that but i still get that error. Directory structure ``` model.tar.gz/ |- model.pth |- code/ |- inference.py |- requirements.txt ``` file : inference.py ``` import torch import os def model_fn(model_dir): model = Your_Model() with open(os.path.join(model_dir, 'model.pth'), 'rb') as f: model.load_state_dict(torch.load(f)) return model def predict_fn(): // ``` based on docs here, https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-batch-code.html#your-algorithms-batch-code-how-containers-should-respond-to-inferences, do we need to install flask and have an /invocations endpoint , that responds 200 ok , when we are using custom container?

I am trying to deploy Pytorch model on ml.inf1.xlarge instance. Image: 301217895009.dkr.ecr.us-west-2.amazonaws.com/sagemaker-neo-pytorch:1.5.1-inf-py3 My python code using some oepncv functions, and when I am trying to run the infernce I got the following error: ModuleNotFoundError: No module named 'cv2' I tried to add opencv-python-headless to requirements.txt, but then I got another error ImportError: libgthread-2.0.so.0: cannot open shared object file How I can use opencv with the ml.inf1 instances?

I want to bind processing pipeline to multiple training pipeline. I just want to compare algorithm accuracy. Same dataset will be trained by multiple algorithms and will be predicted by them. My goal for the future is consolidate predict results of different algorithms and generate combined/consolidated resulst. Is is possible to do in SageMaker. Example Schema: ``` - Train_Algo1 Process - Train_Algo2 - Predict Result - Train_AlgoN ```

I'm trying to troubleshoot why the sagemaker staging endpoint is not using the latest model version. I did the following steps: 1. I started a **new Sagemaker project** and used the build, train, deploy and monitor model **template**. 2. The pipeline automatically executes upon creating the project and I **approved that model**. The **endpoint was successfully created using this model version 1.** 3. I cloned the repo, made a inconsequential change (just changed some training hyper params) and **pushed the changes**. Pipeline executes again successfully **creating model version 2.** 4. I **approved the version 2** and deploy script also runs successfully it seems based on the deploy logs but the **endpoint keeps using version 1** of the model. (Also for some reason version 1 of the model appears as "Model-xxxxxxx" in the Models list while version 2 appears as "Pipeline-xxxxxx-xxxxxx" Would appreciate any help. Thank you!

My mother was automatically signed up for an AWS account or someone used her credentials to sign up. She did not know that she had been signed up, and it sat unused for 3 years. Last month, she got an email from AWS for "unusual activity" and she asked me to help her look into it. Someone racked up $800+ in charges in 10 days for AWS services she has never heard of, let alone used (SageMaker, LightSail were among the services). The card on the AWS account is a credit card that was stolen years ago and has since been cancelled. So when AWS tried to charge the card, it didn't go through. My experience with AWS customer service has been unhelpful so far. Mom changed her AWS password in time so we could get into the account and contact support. I deleted the instances so that the services incurring charges are now stopped. But now AWS is telling me to put in a "valid payment method" or else they will not review the fraudulent bill. They also said that I have to set up additional AWS services (Cost Management, Amazon Cloud Watch, Cloud Trail, WAF, security services) before they'll review the bill. I have clearly explained to them that this entire account is unauthorized and we want to close it ASAP, so adding further services and a payment method doesn't make sense. Why am I being told to use more AWS services when my goal is to use zero? Why do I have to set up "preventative services" when the issue I'm trying to resolve is a PAST issue of fraud? They also asked me to write back and confirm that we have read and understood the AWS Customer Agreement and shared responsibility model." Of course we haven't, because we didn't even know the account existed! Any advice or input into this situation? It's extremely frustrating to be told that AWS won't even look into the issue unless I set up these additional AWS services and give them a payment method. This is a clear case of identity fraud. We want this account shut down. Support Case # is xxxxxxxxxx. Edit- removed case ID -Ann D

We ran into the following error when trying to call SageMaker SKlearn Container using SklearnProcess run(). Also the current boto3 version is 1.20.23 ( there is a github post indicated an issue with one boto3 release 1.16 with timeout issue). Looks like a missing permission for STS service? Any other suggestions? - /opt/conda/lib/python3.7/site-packages/botocore/httpsession.py in send(self, request) 416 raise ProxyConnectionError(proxy_url=proxy_url, error=e) 417 except URLLib3ConnectTimeoutError as e: --> 418 raise ConnectTimeoutError(endpoint_url=request.url, error=e) 419 except URLLib3ReadTimeoutError as e: 420 raise ReadTimeoutError(endpoint_url=request.url, error=e) ConnectTimeoutError: Connect timeout on endpoint URL: "https://sts.us-xxxx-x.amazonaws.com/"

I'm trying to use dask in sagemaker because I have over 1B+ rows in a single dataset. Creating a dask.dataframe works fine, when I create a client through dask, it also works: ``` client = Client(n_workers=6, threads_per_worker=20) client ``` However when I try to use dask_ml.preprocessing.Categorizer, I get the error `NoCredentialsError: Unable to locate credentials`. I understand the issue might be dask distributed client doesn't have authorization to sagemaker client. Its confusing but how do I have them connect somehow? Code below: ``` import dask.dataframe as dd df3 = dd.read_parquet('s3://bucket/parquetfiles2/data_*.parquet', storage_options={'token': 'anon'}) if __name__ == "__main__": obj_df = df3.select_dtypes(include=['object','datetime64']) num_df = df3.select_dtypes(exclude=['object','datetime64']) ce = Categorizer() obj_df_1 = ce.fit_transform(obj_df) ```

Is it possible to deploy spot inf1 instances on sagemaker? We run an API 24/7, and it's costly to keep it up, considering we only have 2 hours of peak performance a day. We don't shut off those machines because we might have random bursts of traffic during the day that CPU instances can't hold. Alternatively, we could deploy spot EC2 inf machines; however, I'm unsure how I would invoke them from gateway and lambda. Does anybody have a tip or recommendation for our case? Thanks!

I hyper-tuned an XGBoost model, deployed the model and created an endpoint. Is there a way to export the model to my local computer? That way I can test the model locally.

When using scikit to create multi model, it throws an exception, but when in single model it works. Complains about model_fn implementation or ping issues, any tips on how to fix this? e.g container={\n", " 'Image' : image_uri,", " 'Mode': 'MultiModel',", " 'ModelDataUrl': 's3://somepatch/with/all/models/,", " 'Environment': {'SAGEMAKER_SUBMIT_DIRECTORY': mme_artifacts_path,", " 'SAGEMAKER_PROGRAM': 'inference.py'} " File "/miniconda3/bin/serve", line 8, in <module> sys.exit(serving_entrypoint()) File "/miniconda3/lib/python3.7/site-packages/sagemaker_sklearn_container/serving.py", line 144, in serving_entrypoint start_model_server() File "/miniconda3/lib/python3.7/site-packages/sagemaker_sklearn_container/serving_mms.py", line 124, in start_model_server modules.import_module(serving_env.module_dir, serving_env.module_name) File "/miniconda3/lib/python3.7/site-packages/sagemaker_containers/_modules.py", line 263, in import_module six.reraise(_errors.ImportModuleError, _errors.ImportModuleError(e), sys.exc_info()[2]) File "/miniconda3/lib/python3.7/site-packages/six.py", line 702, in reraise raise value.with_traceback(tb) File "/miniconda3/lib/python3.7/site-packages/sagemaker_containers/_modules.py", line 258, in import_module module = importlib.import_module(name) File "/miniconda3/lib/python3.7/importlib/__init__.py", line 118, in import_module if name.startswith('.'):

Currently, we are trying to SK-Learn model from a python script running in a local computer by uploading data to S3 bucket. ``` from sagemaker.amazon.amazon_estimator import get_image_uri # container = retrieve(framework='sklearn', region='us-east-1', version="0.23-1") container = sagemaker.image_uris.get_training_image_uri('us-east-1', 'sklearn', framework_version='0.23-1') sklearn_estimator = SKLearn( entry_point="script.py", # # role=get_execution_role(), role = role_aws, instance_count=1, instance_type="ml.m5.4xlarge", framework_version=FRAMEWORK_VERSION, base_job_name="rf-scikit", metric_definitions=[{"Name": "median-AE", "Regex": "AE-at-50th-percentile: ([0-9.]+).*$"}], hyperparameters={ "n-estimators": 100, "min-samples-leaf": 3, "features": "MedInc HouseAge AveRooms AveBedrms Population AveOccup Latitude Longitude", "target": "target", }, sagemaker_session=session, image_uri=container, image_uri_region='us-east-1', # output_path=model_output_path, ) # launch training job, with asynchronous call path_train_test = 's3://'+bucket_name+'/'+prefix sklearn_estimator.fit({"train": path_train_test, "test": path_train_test}, wait=False) ``` 'ClientError: An error occurred (IncompleteSignature) when calling the GetCallerIdentity operation: Credential must have exactly 5 slash-delimited elements, e.g. keyid/date/region/service/term, got 'https://elasticmapreduce.us-east-1b.amazonaws.com//20220406/us-east-1/sts/aws4_request' The access key and the secret key are passed through the session object via a client and passed to the SK-Learn estimator. ``` client_sagemaker = boto3.client('sagemaker', aws_access_key_id=accesskey , aws_secret_access_key=access_secret, ) session = sagemaker.Session(sagemaker_client =client_sagemaker ) ``` The same access key worked for Xgboost model (already available in sagemaker) Any ideas about the reason ?

example provided in the aws documentation , https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html, see sample input csv can be structured like a sample below. is there a link, to see the actual csv file that is used in an working example , instead of the sample posted in the docs. it is also mentioned that each record is one per line and no end of line character is allowed. what if the input type is other than csv, like json? do the same rules apply to json files as well, one record per line, or can one record be spread across multiple lines ,what about new line character in that case? ``` Record1-Attribute1, Record1-Attribute2, Record1-Attribute3, ..., Record1-AttributeM Record2-Attribute1, Record2-Attribute2, Record2-Attribute3, ..., Record2-AttributeM ... ... ```

based on the documentation , https://docs.aws.amazon.com/sagemaker/latest/dg/batch-transform.html, it states that " The ideal value for MaxConcurrentTransforms is equal to the number of compute workers in the batch transform job." how to figure out what the number of compute workers is , i assume this depends on the instance type. also what about the instance count parameter we can set , do we have to take that into account as well?

Hi, there When training model using sagemaker and calling panda.read_csv().. we are getting the above error. It appears that this is a known issue and the workaround is to upgrade aiobotocore to the latest 2.20 release. However, the missing link is where to upgrade the aiobotocore .. since we are running model against SageMaker and its fully managed M5 instance... instead of an EC2 instance. Any thoughts? Error message in full: /opt/conda/lib/python3.7/site-packages/aiobotocore/client.py in create_client(self, service_name, region_name, is_secure, endpoint_url, verify, credentials, scoped_config, api_version, client_config) 43 service_client, endpoint_url, client_config 44 ) ---> 45 self._register_lazy_block_unknown_fips_pseudo_regions(service_client) 46 return service_client 47 AttributeError: 'AioClientCreator' object has no attribute '_register_lazy_block_unknown_fips_pseudo_regions' See notes below under this link: https://github.com/boto/botocore/pull/2558/files#

Hi there, hope you are fine. Recently I came across Sagemaker Async inference API, there we can scale down even to 0 instances. What I want is that I deploy my solution to EC2 instances using FastAPI, uvicorn and Celery or Rabbit-MQ(as message broker, for queuing). Then I can scale up and scale down instances based on traffic. Also, if that's not the case, then I keep a minimal CPU instance on always and based on that I scale up and scale down GPU instances for handling requests. Thanks , for any help. Best Regards Muhammad Ali

does cloudformation support sagemaker batch transform? if yes, can the jobs be triggered/run automatically once the stack is created?

I recently tried the smddp v1.4.0 on SageMaker notebook instance (not sagemaker studio), using 8-GPU instances `ml.p3.16xlarge`, by directly using `smddp` as backend in the training scripts. I launched the estimator by setting `instance_type` to `local_gpu` and ended up with smddp error. Corresponding errors are attached below, saying an initialization error. ``` 42u1m0wni0-algo-1-36bbw | Traceback (most recent call last): 42u1m0wni0-algo-1-36bbw | File "true_main_notebook.py", line 636, in <module> 42u1m0wni0-algo-1-36bbw | main() 42u1m0wni0-algo-1-36bbw | File "true_main_notebook.py", line 178, in main 42u1m0wni0-algo-1-36bbw | dist.init_process_group(backend=args.dist_backend) 42u1m0wni0-algo-1-36bbw | File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 576, in init_process_group 42u1m0wni0-algo-1-36bbw | store, rank, world_size = next(rendezvous_iterator) 42u1m0wni0-algo-1-36bbw | File "/opt/conda/lib/python3.8/site-packages/torch/distributed/rendezvous.py", line 219, in _env_rendezvous_handler 42u1m0wni0-algo-1-36bbw | rank = int(_get_env_or_raise("RANK")) 42u1m0wni0-algo-1-36bbw | File "/opt/conda/lib/python3.8/site-packages/torch/distributed/rendezvous.py", line 203, in _get_env_or_raise 42u1m0wni0-algo-1-36bbw | raise _env_error(env_var) 42u1m0wni0-algo-1-36bbw | ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable RANK expected, but not set 42u1m0wni0-algo-1-36bbw | Environment variable SAGEMAKER_INSTANCE_TYPE is not set 42u1m0wni0-algo-1-36bbw | Running smdistributed.dataparallel v1.4.0 42u1m0wni0-algo-1-36bbw | Error in atexit._run_exitfuncs: 42u1m0wni0-algo-1-36bbw | Traceback (most recent call last): 42u1m0wni0-algo-1-36bbw | File "/opt/conda/lib/python3.8/site-packages/smdistributed/dataparallel/torch/torch_smddp/__init__.py", line 51, in at_exit_smddp 42u1m0wni0-algo-1-36bbw | hm.shutdown() 42u1m0wni0-algo-1-36bbw | RuntimeError: Was this script started with smddprun? For more info on using smddprun, run smddprun -h 42u1m0wni0-algo-1-36bbw | 2022-04-03 16:07:30,005 sagemaker-training-toolkit ERROR Reporting training FAILURE 42u1m0wni0-algo-1-36bbw | 2022-04-03 16:07:30,005 sagemaker-training-toolkit ERROR ExecuteUserScriptError: 42u1m0wni0-algo-1-36bbw | ExitCode 1 42u1m0wni0-algo-1-36bbw | ErrorMessage "ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable RANK expected, but not set 42u1m0wni0-algo-1-36bbw | Environment variable SAGEMAKER_INSTANCE_TYPE is not set Error in atexit._run_exitfuncs: Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/smdistributed/dataparallel/torch/torch_smddp/__init__.py", line 51, in at_exit_smddp hm.shutdown() RuntimeError: Was this script started with smddprun? For more info on using smddprun, run smddprun -h" ``` The original goal is to launch a single-node smddp for debugging. Does the smddp only support launched by AWS python SDK rather than the notebook? Or if something I've done is not correct?

Trying to open sagemaker studio and keep getting the following error message: The JupyterServer app default encountered a problem and was stopped. If you continue to experience issues, please contact Customer Service. Details: InternalFailure Have restarted the app for couple of times and always the error message is the same with no progress. Any help would be appreciated.

configuring a batch job for inference in sagemaker ( sample code below) . for my use case, there are multiple input files and i'm trying to configure such that it can process files one by one or if the instance type allows process them in parallel. if value of max concurrrent transforms is greater than 1, does it processes file in parallel. as things are predefined for the job , before it starts , for exampel the instance_count is already set to 1. how does it do horizontal scaling? can it add add more instances. does the value of instance type or instance count dictate what value we can configure for max concurrent transforms parameter? ``` from sagemaker.transformer import Transformer transformer = Transformer(model_name='my-previously-trained-model', max_concurrent_transforms=0 instance_count=1, instance_type='ml.m4.xlarge') ```