By using AWS re:Post, you agree to the Terms of Use

Sagemaker error when getting data from an endpoint

0

In Sagemaker, I have an endpoint trained using KMEANS. I'm trying to predict from a deployed endpoint using pyspark in a Sagemaker notebook server. I'm using the sample data to try to get predictions.

from  sagemaker_pyspark import SageMakerModel
from  sagemaker_pyspark.transformation.serializers import ProtobufRequestRowSerializer
from  sagemaker_pyspark.transformation.deserializers import KMeansProtobufResponseRowDeserializer
conf = (SparkConf()
        .set("spark.driver.extraClassPath", ":".join(sagemaker_pyspark.classpath_jars())))
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)
region = "us-east-1"
training_data = sqlContext.read.format("libsvm").option("numFeatures", "784")
training=training_data.load("s3a://sagemaker-sample-data-{}/spark/mnist/train/".format(region))



smModel = SageMakerModel.fromEndpoint(
    endpointName="endpoint-9ad5fcee9c52-2017-12-08T13-36-26-267",
    requestRowSerializer=rowSer,
    responseRowDeserializer=KMeansProtobufResponseRowDeserializer(
        closest_cluster_column_name="cluster",
        distance_to_cluster_column_name="closest",
       ),
    )


transformed=smModel.transform(training)
asked 5 years ago31 views
1 Answer
0
Accepted Answer

I figured it out. It turns out that the examples didn't spell out that you need to convert the Python model back from the java model, and you can't call transform() right on the dataframe. Complete code below.

from  sagemaker_pyspark import SageMakerModel
from  sagemaker_pyspark.transformation.serializers import ProtobufRequestRowSerializer
from  sagemaker_pyspark.transformation.deserializers import KMeansProtobufResponseRowDeserializer

rowSer=ProtobufRequestRowSerializer(featuresColumnName="features")

smModel = SageMakerModel.fromEndpoint(
    endpointName="endpoint-9ad5fcee9c52-2017-12-08T13-36-26-267",
    requestRowSerializer=rowSer,
    responseRowDeserializer=KMeansProtobufResponseRowDeserializer(
        closest_cluster_column_name="cluster",
        distance_to_cluster_column_name="closest")
)




ew_model = SageMakerModel._from_java(smModel)

data=SageMakerModel.transform(ew_model,pred)
answered 5 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions