Deploy Inference Endpoint for HG Model Pygmalion

Can someone help me load my model to create an endpoint?

Provided explanation of steps followed, error logs and code used to create everything...thank you in advance.

I'm trying very hard to infer the ''Pygmalion/pygmalion-6b'' model from HuggingFace. But it gives an error when trying to load the worker, it seems unable to. I'm pretty sure I selected an instance big enough, so it shouldn't be because of an OOM exception (I think...)

ERROR LOGS:

2023-06-27T00:20:24.851+02:00 2023-06-26T22:20:24,738 [WARN ] W-9000-model com.amazonaws.ml.mms.wlm.BatchAggregator - Load model failed: model, error: Worker died. 776125764525:/aws/sagemaker/Endpoints/huggingface-pytorch-inference-2023-06-26-22-10-51-934 Link 2023-06-26T22:20:24,738 [WARN ] W-9000-model com.amazonaws.ml.mms.wlm.BatchAggregator - Load model failed: model, error: Worker died. 2023-06-27T00:21:19.999+02:00 2023-06-26T22:21:16,306 [WARN ] W-9000-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s] 776125764525:/aws/sagemaker/Endpoints/huggingface-pytorch-inference-2023-06-26-22-10-51-934 Link 2023-06-26T22:21:16,306 [WARN ] W-9000-model-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]

STEPS FOLLOWED:

I creating the inference endpoint through a notebook and the sagemaker python sdk. I've taken the model from the following huggingFace url: https://huggingface.co/PygmalionAI/pygmalion-6b, downloaded it, removed the ''runs'' folder (which to my knowledge contain training logs), and compressed the model, uploaded to s3 and followed guides on how to use python sdk. If i try to load the model using the HF_ID directly (PygmalionAI/pygmalion-6b) it won't work. (not providing python code because of input msg length restriction.

Here's my python code: pip install sagemaker --upgrade pip install transformers --upgrade

import sagemaker import boto3 sess = sagemaker.Session() sagemaker_session_bucket=None if sagemaker_session_bucket is None and sess is not None: sagemaker_session_bucket = sess.default_bucket()

try: role = sagemaker.get_execution_role() except ValueError: iam = boto3.client('iam') role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']

sess = sagemaker.Session(default_bucket=sagemaker_session_bucket)

print(f"sagemaker role arn: {role}") print(f"sagemaker session region: {sess.boto_region_name}")

from sagemaker.huggingface import get_huggingface_llm_image_uri

llm_image = get_huggingface_llm_image_uri( "huggingface", version="0.8.2" )

print(f"llm image uri: {llm_image}")

import json from sagemaker.huggingface import HuggingFaceModel

instance_type = "ml.g4dn.2xlarge" number_of_gpu = 1 health_check_timeout = 750

config = { 'SM_NUM_GPUS': json.dumps(number_of_gpu), 'MAX_INPUT_LENGTH': json.dumps(1024),
'MAX_TOTAL_TOKENS': json.dumps(2048), }

llm_model = HuggingFaceModel( model_data='s3://pygmalion-6b-s3/model.tar.gz', role=role, transformers_version="4.26", pytorch_version="1.13", py_version='py39', env=config, )

llm = llm_model.deploy( initial_instance_count=1, instance_type=instance_type, container_startup_health_check_timeout=health_check_timeout, # 10 minutes to be able to load the model )

data = { "inputs": "hello, how are you?" }

python script based on: https://huggingface.co/blog/sagemaker-huggingface-llm but also tried with this version: https://huggingface.co/docs/sagemaker/inference#create-a-model-artifact-for-deployment and was also unable to load the worker.. TT