Async Inference for Meta JumpStart SageMaker Models not working

Thank you for bringing this issue to our attention. It appears that you're experiencing a problem with Asynchronous Inference for Meta's JumpStart SageMaker models, specifically with the Llama 3.1 8B instruct and Llama 3.2 3B instruct models.

The error you're seeing in CloudWatch logs, "The response from container primary did not specify the required Content-Length header," suggests that there might indeed be an issue with the prebuilt inference.py files for these specific models.

Asynchronous Inference in SageMaker requires certain configurations and responses to work correctly. The Content-Length header is a crucial part of the HTTP response that SageMaker expects when processing asynchronous inference requests. Without this header, SageMaker cannot properly handle the response and store it in the specified S3 output location.

This issue seems to be specific to the Meta JumpStart models you mentioned, as you've noted that asynchronous inference works correctly with other models like Hugging Face's FLAN-T5 XL.

To troubleshoot this issue:

1. Double-check that you're using the latest version of the Meta JumpStart models, as there might have been updates to address this issue.

2. Consider reaching out to AWS support or posting on the AWS forums to report this potential bug. They may be able to confirm if this is a known issue and provide a workaround or timeline for a fix.

3. In the meantime, you might need to stick with synchronous (real-time) inference for these specific Meta models until the issue is resolved.

4. If possible, you could try to modify the inference code to include the Content-Length header manually, but this would require deeper knowledge of the model's inference script and might not be straightforward.

5. Monitor the AWS Machine Learning Blog and SageMaker documentation for any updates or announcements regarding fixes for JumpStart models.

We apologize for the inconvenience this may be causing in your workflow. It's always helpful when users like you report such issues, as it contributes to improving the service for everyone.
Sources
Asynchronous inference - Amazon SageMaker
Troubleshooting - Amazon SageMaker
Llama 3.2 models from Meta are now available in Amazon SageMaker JumpStart | AWS Machine Learning Blog