Error in endpoint access for model lama2

Question

```
from langchain import LLMChain
from langchain import SagemakerEndpoint
from langchain.prompts import PromptTemplate
from langchain.llms.sagemaker_endpoint import LLMContentHandler
import os
import json

endpoint_name = os.environ["LLAMA_2_ENDPOINT"]
region = os.environ["AWS_REGION"]

class ContentHandler(LLMContentHandler):
    content_type = "application/json"
    accepts = "application/json"

def transform_input(self, prompt: str, model_kwargs: dict) -> bytes:
        input_str = json.dumps({"inputs" : [[{"role" : "system",
        "content" : "You are a kind robot."},
        {"role" : "user", "content" : prompt}]],
        "parameters" : {**model_kwargs}})
        return input_str.encode('utf-8')
    
    def transform_output(self, output: bytes) -> str:
        response_json = json.loads(output.read().decode("utf-8"))
        return response_json[0]["generation"]["content"]

template = "{content}"

prompt = PromptTemplate.from_template(template)

content_handler = ContentHandler()

llm=SagemakerEndpoint(
     endpoint_name=endpoint_name, 
     region_name=region, 
     model_kwargs={"max_new_tokens": 700, "top_p": 0.9, "temperature": 0.6},
     endpoint_kwargs={"CustomAttributes": 'accept_eula=true'},
     content_handler=content_handler
 )

llm_chain = LLMChain(
     llm=llm,
     prompt=prompt
 )

llm_chain.run({"How can I travel from New York to Los Angeles?"})
```
For this code i am getting error 
ValueError: Error raised by inference endpoint: An error occurred (ValidationError) when calling the InvokeEndpoint operation: Inference Component Name header is required for endpoints to which you plan to deploy inference components. Please include Inference Component Name header or consider using SageMaker models.
Please help me to solve this error

Amir · Answer

The error you're encountering, ValueError: Error raised by inference endpoint: An error occurred (ValidationError) when calling the InvokeEndpoint operation: Inference Component Name header is required for endpoints to which you plan to deploy inference components, suggests that the request made to the SageMaker endpoint is missing a required header. This header is the Inference Component Name which is necessary for invoking the endpoint, especially when dealing with inference components.

The issue seems to be with how the request is being made to the SageMaker endpoint. To resolve this error, you need to include the Inference Component Name header in your endpoint invocation request.
You need to add the Inference Component Name header to the endpoint_kwargs. Since you are already using CustomAttributes, you can append this additional header to it.
llm = SagemakerEndpoint(
    endpoint_name=endpoint_name,
    region_name=region,
    model_kwargs={"max_new_tokens": 700, "top_p": 0.9, "temperature": 0.6},
    endpoint_kwargs={
        "CustomAttributes": 'accept_eula=true,InferenceComponentName=MyInferenceComponent'
    },
    content_handler=content_handler
)

Error in endpoint access for model lama2

Add your answer

Relevant content