SageMaker JumpStart Requires Language Parameter for Whisper Large-v3

Based on the information available, when using the Whisper model deployed on SageMaker, you need to specify both the "language" and "task" parameters in your payload, even though the original Whisper model has language detection capabilities.

Looking at the example implementation of transcription with Whisper on SageMaker, the payload structure requires these parameters:

payload = {
"audio_input": hex_audio,
"language": "english",
"task": "transcribe",
"top_p": 0.9
}

The SageMaker implementation of Whisper appears to have this requirement built into its interface. To use the language detection feature while still satisfying the SageMaker endpoint requirements, you might try:

1. Setting the "language" parameter to "auto" or an empty string to indicate you want to use language detection
2. Checking the SageMaker container documentation for the specific Whisper implementation to see if there's a designated value for enabling automatic language detection

If you need to debug further, you could add logging statements to your code and check the CloudWatch logs of the endpoint to understand how the model is processing your requests and what specific requirements the SageMaker implementation has.
Sources
Build a serverless audio summarization solution with Amazon Bedrock and Whisper | AWS Machine Learning Blog
Troubleshoot invoke issues with Sagemaker AI asynchronous endpoints | AWS re:Post

Why SageMaker JumpStart requires the language parameter:

The SageMaker JumpStart deployment of Whisper Large-v3 uses a pre-built inference script that explicitly validates and requires both the language and task parameters. This is a design decision in the JumpStart implementation, not a limitation of the Whisper model itself. While the native Whisper model supports automatic language detection when language=None, the JumpStart wrapper enforces this parameter validation.

How to enable language detection with Whisper on SageMaker:

Since the JumpStart implementation doesn't currently support automatic language detection, here are alternative approaches:

Option 1: Deploy with a Custom Inference Script

You can bring your own inference script that removes the language validation requirement. Create a custom inference.py:

python from transformers import WhisperProcessor, WhisperForConditionalGeneration import torch import json

def model_fn(model_dir): processor = WhisperProcessor.from_pretrained(model_dir) model = WhisperForConditionalGeneration.from_pretrained(model_dir) device = "cuda" if torch.cuda.is_available() else "cpu" model = model.to(device) return {"model": model, "processor": processor, "device": device}

def predict_fn(input_data, model_dict): model = model_dict["model"] processor = model_dict["processor"] device = model_dict["device"]

audio = input_data.get("audio_input")

input_features = processor(
    audio, 
    sampling_rate=16000, 
    return_tensors="pt"
).input_features.to(device)

# Generate without forced_decoder_ids enables auto language detection
predicted_ids = model.generate(input_features)

transcription = processor.batch_decode(
    predicted_ids, 
    skip_special_tokens=True
)[0]

return {"transcription": transcription}

Then deploy using a HuggingFace Model with your custom script:

python from sagemaker.huggingface import HuggingFaceModel

huggingface_model = HuggingFaceModel( model_data="s3://your-bucket/model.tar.gz", # Your model artifacts role=role, transformers_version="4.37.0", pytorch_version="2.1.0", py_version="py310", entry_point="inference.py", source_dir="src" # Directory containing your inference.py )

predictor = huggingface_model.deploy( initial_instance_count=1, instance_type="ml.g5.xlarge" )

Option 2: Deploy Directly from HuggingFace Hub

Deploy the model directly using the HuggingFace Inference Toolkit:

python from sagemaker.huggingface import HuggingFaceModel

hub = { 'HF_MODEL_ID': 'openai/whisper-large-v3', 'HF_TASK': 'automatic-speech-recognition' }

huggingface_model = HuggingFaceModel( transformers_version='4.37.0', pytorch_version='2.1.0', py_version='py310', env=hub, role=role, )

predictor = huggingface_model.deploy( initial_instance_count=1, instance_type='ml.g5.xlarge' )

This approach uses the default HuggingFace inference handler which may provide more flexibility with the language parameter.

Option 3: Pre-detect Language Separately

If you prefer to continue using JumpStart, you could implement a two-step process:

1. Use a separate language detection model/service to identify the language first
2. Pass the detected language to the Whisper endpoint

Additional Resources:

• Deploy models with SageMaker AI - https://huggingface.co/docs/sagemaker/inference
• HuggingFace Inference Toolkit - https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html

If you need account-specific assistance with implementation details, please open a support case through the AWS Console.