Sagemaker error when getting data from an endpoint

Question

In Sagemaker, I have an endpoint trained using KMEANS.  I'm trying to predict from a deployed endpoint using pyspark in a Sagemaker notebook server.   I'm using the sample data to try to get predictions.

from  sagemaker_pyspark import SageMakerModel
    from  sagemaker_pyspark.transformation.serializers import ProtobufRequestRowSerializer
    from  sagemaker_pyspark.transformation.deserializers import KMeansProtobufResponseRowDeserializer
    conf = (SparkConf()
            .set("spark.driver.extraClassPath", ":".join(sagemaker_pyspark.classpath_jars())))
    sc = SparkContext(conf=conf)
    sqlContext = SQLContext(sc)
    region = "us-east-1"
    training_data = sqlContext.read.format("libsvm").option("numFeatures", "784")
    training=training_data.load("s3a://sagemaker-sample-data-{}/spark/mnist/train/".format(region))

smModel = SageMakerModel.fromEndpoint(
        endpointName="endpoint-9ad5fcee9c52-2017-12-08T13-36-26-267",
        requestRowSerializer=rowSer,
        responseRowDeserializer=KMeansProtobufResponseRowDeserializer(
            closest_cluster_column_name="cluster",
            distance_to_cluster_column_name="closest",
           ),
        )

transformed=smModel.transform(training)

Accepted Answer

I figured it out.  It turns out that the examples didn't spell out that you need to convert the Python model back from the java model, and you can't call transform() right on the dataframe.  Complete code below.

from  sagemaker_pyspark import SageMakerModel
    from  sagemaker_pyspark.transformation.serializers import ProtobufRequestRowSerializer
    from  sagemaker_pyspark.transformation.deserializers import KMeansProtobufResponseRowDeserializer
    
    rowSer=ProtobufRequestRowSerializer(featuresColumnName="features")
    
    smModel = SageMakerModel.fromEndpoint(
        endpointName="endpoint-9ad5fcee9c52-2017-12-08T13-36-26-267",
        requestRowSerializer=rowSer,
        responseRowDeserializer=KMeansProtobufResponseRowDeserializer(
            closest_cluster_column_name="cluster",
            distance_to_cluster_column_name="closest")
    )

ew_model = SageMakerModel._from_java(smModel)
    
    data=SageMakerModel.transform(ew_model,pred)

Sagemaker error when getting data from an endpoint

Relevant content