Skip to content

Nova Pro finetune job keeps stuck

0

I tried to finetune Nova Pro in bedrock on my account, but after creating the job, the training status has been in Not Started for almost 14 hours. Is there any way to check what happened? Enter image description here

1 Answer
0

Based on your description, your Nova Pro fine-tuning job has been stuck in "Not Started" status for an extended period (14 hours), which is unusual.

There are a few ways to troubleshoot this issue:

  1. Check if you've hit any service quotas or limits for Bedrock model customization jobs. Even if you're not running multiple jobs concurrently, there might be account-level quotas affecting your job.

  2. Verify that your input data in S3 is correctly formatted and accessible. Make sure the S3 bucket permissions are properly configured to allow Bedrock to access your training data.

  3. Review the IAM role you provided for the fine-tuning job. The role needs appropriate permissions to access your S3 buckets (both input and output) and to perform model customization operations.

  4. Look for any error messages or logs in CloudWatch that might provide more insight into why the job isn't starting.

  5. If you created the job using the API, double-check that all required parameters were correctly specified, including hyperparameters like stepCount, batchSize, and learningRate.

If none of these steps help identify the issue, I'd recommend contacting AWS Support for further assistance. They can look into the specifics of your account and job configuration to identify any underlying issues that might be preventing the job from starting.
Sources
AWS Bedrock batch inference stuck on SCHEDULED | AWS re:Post
Build character consistent storyboards using Amazon Nova in Amazon Bedrock – Part 2 | Artificial Intelligence

answered a year ago

AWS
SUPPORT ENGINEER

reviewed a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.