Processing Job automatically created when I start a training job

0

Hi,
I haven't used sagemaker for a while and today I started a training job (with the same old settings I always used before), but this time I noticed that a processing job has been automatically created and it's running while my training job runs (I don't even know what a processing job is). I also checked in the dashboard to be sure, this was not happening before, it's the second time (first time was in December) but I've been using sagemaker for the last two years..
Is this a wanted behaviour? I didn't find anything related in the documentation, but it's important to know because I don't want extra costs..
This is the image used by the processing job, with a instance type of ml.m5.2xlarge which I didn't set anywhere..

929884845733.dkr.ecr.eu-west-1.amazonaws.com/sagemaker-debugger-rules:latest  

And this is how I launch my training job (the entrypoint script is basically Keras code for a MobileNetV3)

import sagemaker
from sagemaker.tensorflow import TensorFlow
from sagemaker import get_execution_role

bucket = 'mybucket'

train_data = 's3://{}/{}'.format(bucket,'train')

validation_data = 's3://{}/{}'.format(bucket,'test')

s3_output_location = 's3://{}'.format(bucket)

hyperparameters = {'epochs': 130, 'batch-size' : 512, 'learning-rate' : 0.0002}

metrics = .. some regex here

tf_estimator = TensorFlow(entry_point='train.py',
role=get_execution_role(),
train_instance_count=1,
train_instance_type='ml.p2.xlarge',
train_max_run=172800,
output_path=s3_output_location,
framework_version='2.3.0',
py_version='py37',
metric_definitions = metrics,
hyperparameters = hyperparameters,
source_dir="data")

inputs = {'train': train_data, 'test': validation_data}
myJobName = 'myname'
tf_estimator.fit(inputs=inputs, job_name=myJobName)

Edited by: rokk07 on Jan 25, 2021 2:55 AM

rokk07
feita há 3 anos496 visualizações
1 Resposta
0

I can answer myself. It's described https://docs.aws.amazon.com/sagemaker/latest/dg/use-debugger-built-in-rules.html , must be a recent feature. The documentation explain also how to disable the debugger.

rokk07
respondido há 3 anos

Você não está conectado. Fazer login para postar uma resposta.

Uma boa resposta responde claramente à pergunta, dá feedback construtivo e incentiva o crescimento profissional de quem perguntou.

Diretrizes para responder a perguntas