Processing Job automatically created when I start a training job

0

Hi,
I haven't used sagemaker for a while and today I started a training job (with the same old settings I always used before), but this time I noticed that a processing job has been automatically created and it's running while my training job runs (I don't even know what a processing job is). I also checked in the dashboard to be sure, this was not happening before, it's the second time (first time was in December) but I've been using sagemaker for the last two years..
Is this a wanted behaviour? I didn't find anything related in the documentation, but it's important to know because I don't want extra costs..
This is the image used by the processing job, with a instance type of ml.m5.2xlarge which I didn't set anywhere..

929884845733.dkr.ecr.eu-west-1.amazonaws.com/sagemaker-debugger-rules:latest  

And this is how I launch my training job (the entrypoint script is basically Keras code for a MobileNetV3)

import sagemaker
from sagemaker.tensorflow import TensorFlow
from sagemaker import get_execution_role

bucket = 'mybucket'

train_data = 's3://{}/{}'.format(bucket,'train')

validation_data = 's3://{}/{}'.format(bucket,'test')

s3_output_location = 's3://{}'.format(bucket)

hyperparameters = {'epochs': 130, 'batch-size' : 512, 'learning-rate' : 0.0002}

metrics = .. some regex here

tf_estimator = TensorFlow(entry_point='train.py',
role=get_execution_role(),
train_instance_count=1,
train_instance_type='ml.p2.xlarge',
train_max_run=172800,
output_path=s3_output_location,
framework_version='2.3.0',
py_version='py37',
metric_definitions = metrics,
hyperparameters = hyperparameters,
source_dir="data")

inputs = {'train': train_data, 'test': validation_data}
myJobName = 'myname'
tf_estimator.fit(inputs=inputs, job_name=myJobName)

Edited by: rokk07 on Jan 25, 2021 2:55 AM

rokk07
asked 3 years ago482 views
1 Answer
0

I can answer myself. It's described https://docs.aws.amazon.com/sagemaker/latest/dg/use-debugger-built-in-rules.html , must be a recent feature. The documentation explain also how to disable the debugger.

rokk07
answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions