Processing Job automatically created when I start a training job

0

Hi,
I haven't used sagemaker for a while and today I started a training job (with the same old settings I always used before), but this time I noticed that a processing job has been automatically created and it's running while my training job runs (I don't even know what a processing job is). I also checked in the dashboard to be sure, this was not happening before, it's the second time (first time was in December) but I've been using sagemaker for the last two years..
Is this a wanted behaviour? I didn't find anything related in the documentation, but it's important to know because I don't want extra costs..
This is the image used by the processing job, with a instance type of ml.m5.2xlarge which I didn't set anywhere..

929884845733.dkr.ecr.eu-west-1.amazonaws.com/sagemaker-debugger-rules:latest  

And this is how I launch my training job (the entrypoint script is basically Keras code for a MobileNetV3)

import sagemaker
from sagemaker.tensorflow import TensorFlow
from sagemaker import get_execution_role

bucket = 'mybucket'

train_data = 's3://{}/{}'.format(bucket,'train')

validation_data = 's3://{}/{}'.format(bucket,'test')

s3_output_location = 's3://{}'.format(bucket)

hyperparameters = {'epochs': 130, 'batch-size' : 512, 'learning-rate' : 0.0002}

metrics = .. some regex here

tf_estimator = TensorFlow(entry_point='train.py',
role=get_execution_role(),
train_instance_count=1,
train_instance_type='ml.p2.xlarge',
train_max_run=172800,
output_path=s3_output_location,
framework_version='2.3.0',
py_version='py37',
metric_definitions = metrics,
hyperparameters = hyperparameters,
source_dir="data")

inputs = {'train': train_data, 'test': validation_data}
myJobName = 'myname'
tf_estimator.fit(inputs=inputs, job_name=myJobName)

Edited by: rokk07 on Jan 25, 2021 2:55 AM

rokk07
質問済み 3年前490ビュー
1回答
0

I can answer myself. It's described https://docs.aws.amazon.com/sagemaker/latest/dg/use-debugger-built-in-rules.html , must be a recent feature. The documentation explain also how to disable the debugger.

rokk07
回答済み 3年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン