Hello,
I've been trying to use batch transform to perform inference on a semantic segmentation problem but I've been struggling to get the correct output.
The code of my estimator is:
ss_estimator.set_hyperparameters(
backbone="resnet-101", # This is the encoder. Other option is resnet-101
algorithm="fcn", # This is the decoder. Other options are 'psp' and 'deeplab'
use_pretrained_model="True", # Use the pre-trained model.
crop_size=320, # Size of image random crop.
num_classes=3, # Pascal has 21 classes. This is a mandatory parameter.
epochs=30, # Number of epochs to run.
learning_rate=0.0001,
optimizer='adam', # Other options include 'adam', 'rmsprop', 'nag', 'adagrad'.
lr_scheduler='poly', # Other options include 'cosine' and 'step'.
mini_batch_size=4, # Setup some mini batch size.
validation_mini_batch_size=4,
early_stopping=True, # Turn on early stopping. If OFF, other early stopping parameters are ignored.
early_stopping_patience=5, # Tolerate these many epochs if the mIoU doens't increase.
early_stopping_min_epochs=2, # No matter what, run these many number of epochs.
num_training_samples=num_training_samples, # This is a mandatory parameter, 1464 in this case.
)
After training the model, I am configuring the batch transform as below:
timestamp = time.strftime("-%Y-%m-%d-%H-%M-%S", time.gmtime())
batch_job_name = "image-segmentation-model-batch" + timestamp
model_name = "DEMO-full-image-segmentation-model" + time.strftime(
"-%Y-%m-%d-%H-%M-%S", time.gmtime()
)
output_folder="s3://client/private_datasets/experiments/batch_transform_output/"
batch_input = "s3://client/private_datasets/experiments/test-images-small/"
request = {
"TransformJobName": batch_job_name,
"ModelName": model_name,
"MaxConcurrentTransforms": 16,
"MaxPayloadInMB": 6,
"BatchStrategy": "SingleRecord",
"TransformOutput": {"S3OutputPath": "{}/output".format(output_folder)},
"TransformInput": {
"DataSource": {"S3DataSource": {"S3DataType": "S3Prefix", "S3Uri": batch_input}},
"ContentType": 'image/jpeg',
"SplitType": "None",
"CompressionType": "None",
},
"TransformResources": {"InstanceType": "ml.c5.xlarge", "InstanceCount": 1},
}
print("Transform job name: {}".format(batch_job_name))
print("\nInput Data Location: {}".format(batch_input))
And I'm starting the batch transform job as:
sagemaker = boto3.client("sagemaker")
sagemaker.create_transform_job(**request)
print("Created Transform job with name: ", batch_job_name)
while True:
response = sagemaker.describe_transform_job(TransformJobName=batch_job_name)
status = response["TransformJobStatus"]
if status == "Completed":
print("Transform job ended with status: " + status)
break
if status == "Failed":
message = response["FailureReason"]
print("Transform failed with the following error: {}".format(message))
raise Exception("Transform job failed")
time.sleep(30)
My problem is that the batch transform outputs a "*.out" file, which I have no idea how to open.
Can you help me out? Or is there some tutorial to use batch transform for mask prediction/image segmentation?