How to use sagemaker-pyspark in batch inference


am trying to execute the code below

ENDPOINT_NAME = "my-endpoint"
from sagemaker_pyspark import SageMakerModel
from sagemaker_pyspark import EndpointCreationPolicy
from sagemaker_pyspark.transformation.serializers import ProtobufRequestRowSerializer
from sagemaker_pyspark.transformation.deserializers import ProtobufResponseRowDeserializer
from pyspark.sql.types import StructType, StructField, MapType, StringType, IntegerType, ArrayType, FloatType

attachedModel = SageMakerModel.fromEndpoint(
    endpointName = ENDPOINT_NAME,
        featuresColumnName = "col1"
        StructField('prediction', MapType(StringType(), FloatType()))

data=SageMakerModel.transform(attachedModel, df['col1'])

I keep getting the below error though

py4j.protocol.Py4JError: An error occurred while calling o58.__getstate__. Trace:
py4j.Py4JException: Method __getstate__([]) does not exist
        at py4j.reflection.ReflectionEngine.getMethod(
        at py4j.reflection.ReflectionEngine.getMethod(
        at py4j.Gateway.invoke(
        at py4j.commands.AbstractCommand.invokeMethod(
        at py4j.commands.CallCommand.execute(

any ideas ?

asked 2 years ago354 views
1 Answer

Hi there,

it would seem to me that there maybe an discrepancy in when one of your outputs is returned, they should only be returned after an if else statement and defined within the if else statement. Here is a resource that seems to address a problem similar to yours, perhaps applying a similar strategy will produce significant results.

Hopefully this will provide some insight into your problem.

Regards NN

answered 2 years ago
  • Hi Ntiyiso, where do you suggest to add such logic ?

  • Hi there exorcismus, I would suggest applying the logic to your predictions in the model itself if applicable.

  • Hi Ntiyiso, the model and the endpoint works fine and am able to use them for prediction, am not sure how/why implementing that logic affects being invoked from spark, what do you think ?

  • Hi exorcismus, it's possible that pyspark version and pyarrow settings aren't compatible with your os and/vm configurations similarly what sudopip was experiencing.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions