Sagemaker error when getting data from an endpoint

0

In Sagemaker, I have an endpoint trained using KMEANS. I'm trying to predict from a deployed endpoint using pyspark in a Sagemaker notebook server. I'm using the sample data to try to get predictions.

from  sagemaker_pyspark import SageMakerModel
from  sagemaker_pyspark.transformation.serializers import ProtobufRequestRowSerializer
from  sagemaker_pyspark.transformation.deserializers import KMeansProtobufResponseRowDeserializer
conf = (SparkConf()
        .set("spark.driver.extraClassPath", ":".join(sagemaker_pyspark.classpath_jars())))
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)
region = "us-east-1"
training_data = sqlContext.read.format("libsvm").option("numFeatures", "784")
training=training_data.load("s3a://sagemaker-sample-data-{}/spark/mnist/train/".format(region))



smModel = SageMakerModel.fromEndpoint(
    endpointName="endpoint-9ad5fcee9c52-2017-12-08T13-36-26-267",
    requestRowSerializer=rowSer,
    responseRowDeserializer=KMeansProtobufResponseRowDeserializer(
        closest_cluster_column_name="cluster",
        distance_to_cluster_column_name="closest",
       ),
    )


transformed=smModel.transform(training)
AWS
Ron_W
feita há 6 anos246 visualizações
1 Resposta
0
Resposta aceita

I figured it out. It turns out that the examples didn't spell out that you need to convert the Python model back from the java model, and you can't call transform() right on the dataframe. Complete code below.

from  sagemaker_pyspark import SageMakerModel
from  sagemaker_pyspark.transformation.serializers import ProtobufRequestRowSerializer
from  sagemaker_pyspark.transformation.deserializers import KMeansProtobufResponseRowDeserializer

rowSer=ProtobufRequestRowSerializer(featuresColumnName="features")

smModel = SageMakerModel.fromEndpoint(
    endpointName="endpoint-9ad5fcee9c52-2017-12-08T13-36-26-267",
    requestRowSerializer=rowSer,
    responseRowDeserializer=KMeansProtobufResponseRowDeserializer(
        closest_cluster_column_name="cluster",
        distance_to_cluster_column_name="closest")
)




ew_model = SageMakerModel._from_java(smModel)

data=SageMakerModel.transform(ew_model,pred)
AWS
Ron_W
respondido há 6 anos

Você não está conectado. Fazer login para postar uma resposta.

Uma boa resposta responde claramente à pergunta, dá feedback construtivo e incentiva o crescimento profissional de quem perguntou.

Diretrizes para responder a perguntas