How to get semantic segmentation output using Batchtransform

0

Hello,

I've been trying to use batch transform to perform inference on a semantic segmentation problem but I've been struggling to get the correct output.

The code of my estimator is:

ss_estimator.set_hyperparameters(
    backbone="resnet-101",  # This is the encoder. Other option is resnet-101
    algorithm="fcn",  # This is the decoder. Other options are 'psp' and 'deeplab'
    use_pretrained_model="True",  # Use the pre-trained model.
    crop_size=320,  # Size of image random crop.
    num_classes=3,  # Pascal has 21 classes. This is a mandatory parameter.
    epochs=30, # Number of epochs to run.
    learning_rate=0.0001,                             
    optimizer='adam', # Other options include 'adam', 'rmsprop', 'nag', 'adagrad'.
    lr_scheduler='poly', # Other options include 'cosine' and 'step'.   
    mini_batch_size=4,  # Setup some mini batch size.
    validation_mini_batch_size=4,
    early_stopping=True,  # Turn on early stopping. If OFF, other early stopping parameters are ignored.
    early_stopping_patience=5,  # Tolerate these many epochs if the mIoU doens't increase.
    early_stopping_min_epochs=2,  # No matter what, run these many number of epochs.
    num_training_samples=num_training_samples,  # This is a mandatory parameter, 1464 in this case.
)

After training the model, I am configuring the batch transform as below:

timestamp = time.strftime("-%Y-%m-%d-%H-%M-%S", time.gmtime())
batch_job_name = "image-segmentation-model-batch" + timestamp
model_name = "DEMO-full-image-segmentation-model" + time.strftime(
    "-%Y-%m-%d-%H-%M-%S", time.gmtime()
)
output_folder="s3://client/private_datasets/experiments/batch_transform_output/"
batch_input = "s3://client/private_datasets/experiments/test-images-small/"
request = {
    "TransformJobName": batch_job_name,
    "ModelName": model_name,
    "MaxConcurrentTransforms": 16,
    "MaxPayloadInMB": 6,
    "BatchStrategy": "SingleRecord",
    "TransformOutput": {"S3OutputPath": "{}/output".format(output_folder)},
    "TransformInput": {
        "DataSource": {"S3DataSource": {"S3DataType": "S3Prefix", "S3Uri": batch_input}},
        "ContentType": 'image/jpeg',
        "SplitType": "None",
        "CompressionType": "None",
    },
    "TransformResources": {"InstanceType": "ml.c5.xlarge", "InstanceCount": 1},
}

print("Transform job name: {}".format(batch_job_name))
print("\nInput Data Location: {}".format(batch_input))

And I'm starting the batch transform job as:

sagemaker = boto3.client("sagemaker")
sagemaker.create_transform_job(**request)

print("Created Transform job with name: ", batch_job_name)

while True:
    response = sagemaker.describe_transform_job(TransformJobName=batch_job_name)
    status = response["TransformJobStatus"]
    if status == "Completed":
        print("Transform job ended with status: " + status)
        break
    if status == "Failed":
        message = response["FailureReason"]
        print("Transform failed with the following error: {}".format(message))
        raise Exception("Transform job failed")
    time.sleep(30)

My problem is that the batch transform outputs a "*.out" file, which I have no idea how to open.

Can you help me out? Or is there some tutorial to use batch transform for mask prediction/image segmentation?

No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions