Bedrock model customization job stuck at trainingDetails.status=NotStarted for 24+ hours (validation completed in minutes)

Question

I have an Amazon Bedrock model customization (fine-tuning) job that completed data validation almost immediately, but training has not started even after 24+ hours.

Details:
- Job name: marco-demo01-pharma-ft
- Region: us-east-1
- Job creation time: 2026-08-03 14:25:29 UTC
- Current status (as of this post): InProgress
- statusDetails.validationDetails.status: Completed (finished at 2026-08-03 14:28:36 UTC — about 3 minutes after job creation)
- statusDetails.trainingDetails.status: NotStarted — has not changed since job creation, now over 24 hours ago
- No failureMessage returned
- No outputModelArn (job hasn't produced any output)

I'm querying GetModelCustomizationJob via boto3 directly to get this level of detail, since the console only shows the same top-level "InProgress" status. There's no CloudWatch log activity for this job (checked the associated log group — 0 bytes stored), and I'm aware Bedrock doesn't expose real-time training progress, but a 24+ hour stall at NotStarted for the training phase (with validation long done) seems outside normal behavior. AWS's own docs mention jobs typically take 3-4 hours and to contact support if training exceeds 24 hours — but I'm on the Basic support plan, so I don't have access to a technical support case for this.

Has anyone else seen a job stuck like this? Is there a known issue with job scheduling in us-east-1, or is the right move to stop and relaunch the job? Any insight into why trainingDetails.status would stay NotStarted for this long would be appreciated.

re:Post Agent · Accepted Answer

A model customization job stuck at trainingDetails.status=NotStarted for over 24 hours after validation has completed is not normal behavior. Based on the job lifecycle, customization jobs progress through three stages: validation, data processing, and training. In your case, validation completed successfully in about 3 minutes, but the training phase has remained at NotStarted status for more than 24 hours.

This extended delay suggests there may be an internal service issue preventing your job from progressing to the training phase. Since the job shows no failure message and remains in InProgress status at the top level, it appears the job hasn't been explicitly failed by the system, but it's also not advancing as expected.

Given that this has persisted well beyond typical job completion times, you have a few options:

1. Stop the current job and create a new one. Sometimes resubmitting the job can resolve transient service issues.

2. Since you're experiencing what appears to be a service-level issue that has persisted beyond normal timeframes, you should reach out through the AWS Support Center at https://console.aws.amazon.com/support/ to add correspondence about this issue. Even on a Basic support plan, you can report service issues and get guidance.

3. If you need faster response times for technical issues like this in the future, you might consider reviewing your AWS Support plan options for access to technical support cases.

Before stopping and relaunching, verify that your IAM role has all necessary permissions for the training phase, including access to your S3 buckets for both input data and output model artifacts. However, given that validation completed successfully, permissions are likely configured correctly, and this appears to be a service-side scheduling or processing issue.  
**Sources**  
[Troubleshooting model customization issues - Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/fine-tuning-troubleshooting.html)  
[Monitor your model customization job - Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-monitor.html)  
[StatusDetails - Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_StatusDetails.html)

Florian Turnwald · Answer

The job is likely queued waiting for backend compute resources. High-demand regions occasionally experience capacity constraints for model training, causing jobs to stall indefinitely at the NotStarted phase.

**I would try:**
 * Stop the job: Terminate the current job. You are only billed for successful training hours, so you will not be charged for this run.
 * Relaunch: Submit a new customization job.
 * Change Region: If the second job also stalls, try running the customization in another supported region (e.g., us-west-2), provided your data residency requirements allow it.

**Note regarding AWS Support:** The Basic support plan strictly limits cases to account, billing, and quota increases. You cannot open a technical support case for this issue via the Support Center. Posting here on re:Post is the correct escalation path for Basic Support users to get this reviewed by AWS staff.

**Reference:** https://docs.aws.amazon.com/bedrock/latest/userguide/custom-models.html

Bedrock model customization job stuck at trainingDetails.status=NotStarted for 24+ hours (validation completed in minutes)

Add your answer

Relevant content