Processing Job automatically created when I start a training job

0

Hi,
I haven't used sagemaker for a while and today I started a training job (with the same old settings I always used before), but this time I noticed that a processing job has been automatically created and it's running while my training job runs (I don't even know what a processing job is). I also checked in the dashboard to be sure, this was not happening before, it's the second time (first time was in December) but I've been using sagemaker for the last two years..
Is this a wanted behaviour? I didn't find anything related in the documentation, but it's important to know because I don't want extra costs..
This is the image used by the processing job, with a instance type of ml.m5.2xlarge which I didn't set anywhere..

929884845733.dkr.ecr.eu-west-1.amazonaws.com/sagemaker-debugger-rules:latest  

And this is how I launch my training job (the entrypoint script is basically Keras code for a MobileNetV3)

import sagemaker
from sagemaker.tensorflow import TensorFlow
from sagemaker import get_execution_role

bucket = 'mybucket'

train_data = 's3://{}/{}'.format(bucket,'train')

validation_data = 's3://{}/{}'.format(bucket,'test')

s3_output_location = 's3://{}'.format(bucket)

hyperparameters = {'epochs': 130, 'batch-size' : 512, 'learning-rate' : 0.0002}

metrics = .. some regex here

tf_estimator = TensorFlow(entry_point='train.py',
role=get_execution_role(),
train_instance_count=1,
train_instance_type='ml.p2.xlarge',
train_max_run=172800,
output_path=s3_output_location,
framework_version='2.3.0',
py_version='py37',
metric_definitions = metrics,
hyperparameters = hyperparameters,
source_dir="data")

inputs = {'train': train_data, 'test': validation_data}
myJobName = 'myname'
tf_estimator.fit(inputs=inputs, job_name=myJobName)

Edited by: rokk07 on Jan 25, 2021 2:55 AM

rokk07
已提问 3 年前479 查看次数
1 回答
0

I can answer myself. It's described https://docs.aws.amazon.com/sagemaker/latest/dg/use-debugger-built-in-rules.html , must be a recent feature. The documentation explain also how to disable the debugger.

rokk07
已回答 3 年前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则