Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Bedrock model customization job stuck at trainingDetails.status=NotStarted for 24+ hours (validation completed in minutes)

0

I have an Amazon Bedrock model customization (fine-tuning) job that completed data validation almost immediately, but training has not started even after 24+ hours.

Details:

  • Job name: marco-demo01-pharma-ft
  • Region: us-east-1
  • Job creation time: 2026-08-03 14:25:29 UTC
  • Current status (as of this post): InProgress
  • statusDetails.validationDetails.status: Completed (finished at 2026-08-03 14:28:36 UTC — about 3 minutes after job creation)
  • statusDetails.trainingDetails.status: NotStarted — has not changed since job creation, now over 24 hours ago
  • No failureMessage returned
  • No outputModelArn (job hasn't produced any output)

I'm querying GetModelCustomizationJob via boto3 directly to get this level of detail, since the console only shows the same top-level "InProgress" status. There's no CloudWatch log activity for this job (checked the associated log group — 0 bytes stored), and I'm aware Bedrock doesn't expose real-time training progress, but a 24+ hour stall at NotStarted for the training phase (with validation long done) seems outside normal behavior. AWS's own docs mention jobs typically take 3-4 hours and to contact support if training exceeds 24 hours — but I'm on the Basic support plan, so I don't have access to a technical support case for this.

Has anyone else seen a job stuck like this? Is there a known issue with job scheduling in us-east-1, or is the right move to stop and relaunch the job? Any insight into why trainingDetails.status would stay NotStarted for this long would be appreciated.

Topics
Machine Learning & AIGenerative AI on AWS
Tags
Machine Learning & AIAmazon Bedrock Custom Model ImportAmazon Bedrock
Language
English

asked 11 days ago65 views

2 Answers
0
Accepted Answer

A model customization job stuck at trainingDetails.status=NotStarted for over 24 hours after validation has completed is not normal behavior. Based on the job lifecycle, customization jobs progress through three stages: validation, data processing, and training. In your case, validation completed successfully in about 3 minutes, but the training phase has remained at NotStarted status for more than 24 hours.

This extended delay suggests there may be an internal service issue preventing your job from progressing to the training phase. Since the job shows no failure message and remains in InProgress status at the top level, it appears the job hasn't been explicitly failed by the system, but it's also not advancing as expected.

Given that this has persisted well beyond typical job completion times, you have a few options:

  1. Stop the current job and create a new one. Sometimes resubmitting the job can resolve transient service issues.

  2. Since you're experiencing what appears to be a service-level issue that has persisted beyond normal timeframes, you should reach out through the AWS Support Center at https://console.aws.amazon.com/support/ to add correspondence about this issue. Even on a Basic support plan, you can report service issues and get guidance.

  3. If you need faster response times for technical issues like this in the future, you might consider reviewing your AWS Support plan options for access to technical support cases.

Before stopping and relaunching, verify that your IAM role has all necessary permissions for the training phase, including access to your S3 buckets for both input data and output model artifacts. However, given that validation completed successfully, permissions are likely configured correctly, and this appears to be a service-side scheduling or processing issue.
Sources
Troubleshooting model customization issues - Amazon Bedrock
Monitor your model customization job - Amazon Bedrock
StatusDetails - Amazon Bedrock

answered 11 days ago

EXPERT

reviewed 11 days ago

  • Marcos
    7 days ago

    Thanks for the suggestion — following up with results.

    Nova 2 Lite fine-tuning is us-east-1 only per AWS docs, so us-west-2 isn't available for this model. Tried stop+resubmit instead — now 4 attempts in, all in us-east-1, same IAM role/S3/params (only job name differs, since Bedrock never releases a job name):

    1. marco-demo01-pharma-ft .../amazon.nova-2-lite-v1:0:256k/jr2dm97inxxm Created 08-03 14:25 UTC, validation done 14:28, stopped after ~25.5h stuck at NotStarted

    2. marco-demo01-pharma-ft-2 .../amazon.nova-2-lite-v1:0:256k/iccv5z0tajyc Created 08-04 15:59 UTC, validation done 16:03, stopped after ~25.25h stuck at NotStarted

    3. marco-demo01-pharma-ft-3 .../amazon.nova-2-lite-v1:0:256k/n45xnijacww6 Created 08-05 17:15 UTC, validation done 17:19, stopped after ~72h stuck at NotStarted

    4. marco-demo01-pharma-ft-4 .../amazon.nova-2-lite-v1:0:256k/lceg313i66qh Created 08-08 22:43 UTC — this one FAILED instead of stalling: validation Failed at 22:45 UTC, failureMessage "Encountered an internal error when processing the request."

    So: 3 silent stalls at NotStarted (never failed, no error) plus one generic internal error, all on the same account/model/region. Trying a 5th attempt now. On Basic support so no Health Dashboard/case access — is this a known issue, or anything else worth trying?

  • Marcos
    7 days ago

    Update: tried a 6th attempt after auditing everything on my end — still failing identically.

    Before retrying, I fully audited whether anything changed on my side: CloudTrail shows zero changes to the IAM role or S3 bucket since initial setup on 08-03, before the very first job ever ran. Data files (210 records, valid JSON, correct schema) are also untouched since 08-03 and already passed validation cleanly on the first 3 attempts.

    I also traced the exact failure moment via CloudTrail: Bedrock's preflight writes a temp file to the output/ prefix (KMS GenerateDataKey succeeds, no errors logged anywhere in that window). My IAM policy only granted PutObject on output/ (not Get/Delete), so as a precaution I hardened it to also allow GetObject/DeleteObject there — zero-cost Terraform change, applied cleanly.

    Retried as attempt #6 with the hardened policy:

    marco-demo01-pharma-ft-6 .../amazon.nova-2-lite-v1:0:256k/vsa9v6r1zahj Created 08-09 01:33:32 UTC, validation Failed at 01:35:28 UTC — same failureMessage: "Encountered an internal error when processing the request."

    So that's now 3 consecutive identical failures (attempts 4, 5, 6), with data and permissions both ruled out as causes. Combined with the first 3 attempts silently stalling at NotStarted, this looks like a persistent issue on this account/model/region combination rather than anything client-side. Any insight into whether this is a known/active issue would be really helpful — running out of things to try on my end.

0

The job is likely queued waiting for backend compute resources. High-demand regions occasionally experience capacity constraints for model training, causing jobs to stall indefinitely at the NotStarted phase.

I would try:

  • Stop the job: Terminate the current job. You are only billed for successful training hours, so you will not be charged for this run.
  • Relaunch: Submit a new customization job.
  • Change Region: If the second job also stalls, try running the customization in another supported region (e.g., us-west-2), provided your data residency requirements allow it.

Note regarding AWS Support: The Basic support plan strictly limits cases to account, billing, and quota increases. You cannot open a technical support case for this issue via the Support Center. Posting here on re:Post is the correct escalation path for Basic Support users to get this reviewed by AWS staff.

Reference: https://docs.aws.amazon.com/bedrock/latest/userguide/custom-models.html

EXPERT

answered 11 days ago

EXPERT

reviewed 11 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content