Model Monitor Schedule Ground Truth Merge Job Fails: Cannot Load Main Class from JAR File
I'm encountering an error when creating a Model Monitor schedule through SageMaker Studio. The job fails during the ground truth merge step. In the CloudWatch logs, I see the following error:
Error: Cannot load main class from JAR file:/opt/amazon/sagemaker-ground-truth-merger-1.0-jar-with-dependencies.jar.
and then
main - ERROR - Exception performing analysis: Command 'bin/spark-submit --master yarn --deploy-mode client --conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider --conf spark.serializer=org.apache.spark.serializer.KryoSerializer /opt/amazon/sagemaker-ground-truth-merger-1.0-jar-with-dependencies.jar --analytics_input /tmp/spark_job_config.json' returned non-zero exit status 1.
From the processing job page, it looks like this:
I'm using the built-in image 159807026194.dkr.ecr.us-west-2.amazonaws.com/sagemaker-model-monitor-groundtruth-merger for the ground truth merger. My configuration (including the /tmp/spark_job_config.json file and S3 settings) has not changed, and the same code was working fine before. However, now I keep getting this error.
Has anyone encountered this issue or have any ideas on what might be causing the JAR file to fail to load its main class? Could this be due to an update in the underlying Spark environment or a change in the built-in merger image?
Any help or troubleshooting tips would be greatly appreciated!
- Newest
- Most votes
- Most comments
I solved this question by recreating the project
This was an intermittent issue that has been resolved by the internal AWS team. The error you encountered with the ground truth merger JAR file was temporary and the service is now functioning as expected.
The error you're encountering suggests there's an issue with loading the main class from the JAR file used for the ground truth merge job in Amazon SageMaker Model Monitor. This problem could be caused by several factors, and here are some troubleshooting steps you can try:
-
Check your IAM permissions: Ensure that the IAM role associated with your SageMaker execution has the necessary permissions to access the required resources, including the S3 bucket where your data and Ground Truth labels are stored.
-
Verify the Ground Truth data format: Make sure your Ground Truth labels are in the correct format (jsonlines) and stored in the appropriate S3 path structure as required by SageMaker.
-
Review your monitoring job configuration: Double-check the parameters you're using to create the monitoring schedule, particularly the
start_time_offsetandend_time_offsetfor the EndpointInput. These should be set to ensure that the monitoring job only uses data for which Ground Truth is available. -
Examine the S3 paths: Verify that the S3 paths for your input data, Ground Truth labels, and output are correct and accessible.
-
Check for recent updates: Since you mentioned this was working before, check if there have been any recent updates to the SageMaker Model Monitor or the built-in merger image that might have introduced compatibility issues.
-
Consider using an on-demand monitoring job: As an alternative, you could try running an on-demand monitoring job using a SageMaker Processing job to see if the issue persists.
-
Review CloudWatch Logs: Examine the full CloudWatch logs for any additional error messages or stack traces that might provide more context about the failure.
If the issue persists after trying these steps, you may need to contact AWS Support for further assistance. They can help investigate if there's an issue with the built-in merger image or provide more detailed troubleshooting based on your specific setup.
Sources
Model Monitor FAQs - Amazon SageMaker AI
Ingest Ground Truth labels and merge them with predictions - Amazon SageMaker
Schedule model quality monitoring jobs - Amazon SageMaker
Relevant content
- asked 8 months ago
