Processing Job automatically created when I start a training job

0

Hi,
I haven't used sagemaker for a while and today I started a training job (with the same old settings I always used before), but this time I noticed that a processing job has been automatically created and it's running while my training job runs (I don't even know what a processing job is). I also checked in the dashboard to be sure, this was not happening before, it's the second time (first time was in December) but I've been using sagemaker for the last two years..
Is this a wanted behaviour? I didn't find anything related in the documentation, but it's important to know because I don't want extra costs..
This is the image used by the processing job, with a instance type of ml.m5.2xlarge which I didn't set anywhere..

929884845733.dkr.ecr.eu-west-1.amazonaws.com/sagemaker-debugger-rules:latest  

And this is how I launch my training job (the entrypoint script is basically Keras code for a MobileNetV3)

import sagemaker
from sagemaker.tensorflow import TensorFlow
from sagemaker import get_execution_role

bucket = 'mybucket'

train_data = 's3://{}/{}'.format(bucket,'train')

validation_data = 's3://{}/{}'.format(bucket,'test')

s3_output_location = 's3://{}'.format(bucket)

hyperparameters = {'epochs': 130, 'batch-size' : 512, 'learning-rate' : 0.0002}

metrics = .. some regex here

tf_estimator = TensorFlow(entry_point='train.py',
role=get_execution_role(),
train_instance_count=1,
train_instance_type='ml.p2.xlarge',
train_max_run=172800,
output_path=s3_output_location,
framework_version='2.3.0',
py_version='py37',
metric_definitions = metrics,
hyperparameters = hyperparameters,
source_dir="data")

inputs = {'train': train_data, 'test': validation_data}
myJobName = 'myname'
tf_estimator.fit(inputs=inputs, job_name=myJobName)

Edited by: rokk07 on Jan 25, 2021 2:55 AM

rokk07
demandé il y a 3 ans490 vues
1 réponse
0

I can answer myself. It's described https://docs.aws.amazon.com/sagemaker/latest/dg/use-debugger-built-in-rules.html , must be a recent feature. The documentation explain also how to disable the debugger.

rokk07
répondu il y a 3 ans

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions