Knowledgebase sync not updating the AI

Question

Problem:
When I add new documents to my S3 bucket (which is a Bedrock KB data source) and sync, all metrics indicate success, but the AI can't retrieve the new data.

Specifics:
Initial KB creation/sync works fine; AI references initial data.
After uploading new documents to S3 and syncing:
Bedrock KB sync history shows "Complete", "Added +1", and total "Source files" increases.
OpenSearch "Total document count" for the KB's index also increases.

However, when querying via the Retrieve API (with unique phrases from the new docs), the new documents are NEVER returned. The AI in TEST Knowledgebase is also completely unaware of them.

This happens with both large KBs (tens of thousands of docs) and small test KBs (single docs).

I get zero errors in Bedrock or during sync. All statuses are green.

I've verified S3 access, IAM permissions, PDF searchability, and consistency of S3 paths.

It seems the data is ingested and indexed, but then it's not searchable. Any ideas why retrieval might fail despite all success indicators?

Thanks!

re:Post Agent · Answer

Based on your description, there are several potential reasons why your newly added documents aren't being retrieved despite successful sync indicators:

1. Sync Process Understanding: When you add new files to your S3 bucket, the Knowledge Base doesn't automatically update. You need to manually trigger a sync operation. It appears you're already doing this, but it's worth noting that syncing is incremental - Bedrock only processes added, modified, or deleted documents since the last sync.

2. Model Access Issues: One common cause of sync problems is not having access to the embedding models being used. Verify that you have requested and been granted access to the specific models your Knowledge Base is using for vector embeddings under "model access" in Bedrock.

3. File Overwriting Problems: If you're overwriting existing files in S3 rather than adding new ones with different names, this can sometimes cause issues with the sync process. The logs might show "PARTIALLY_INDEXED" status with "failed_to_update_metadata" errors. In some cases, deleting the problematic files and uploading them again (rather than overwriting) resolves the issue.

4. Sync Timing: After a successful sync, it may take a few minutes for the vector embeddings to be fully reflected in the knowledge base for querying, especially if you're using a vector store other than Amazon Aurora.

5. Console Refresh Issues: Sometimes the console may not reflect the actual state of operations. Try refreshing the page or waiting a few minutes to see if the retrieval starts working.

If you've verified all these points and the issue persists, you might want to check CloudWatch logs for any "StartIngestionJob" API execution details or CloudTrail event history, which could provide more insights into what might be happening behind the scenes during the sync process.  
**Sources**  
[Bedrock knowledge base data source sync is not working | AWS re:Post](https://repost.aws/questions/QUjagkPYWnTu299Hr7JMnHIQ/bedrock-knowledge-base-data-source-sync-is-not-working)  
[AWS Bedrock Knowledge Base not syncing when files are being over-written in S3 Bucket | AWS re:Post](https://repost.aws/questions/QUKrOVhbmHSTmqAczsbgvjfA/aws-bedrock-knowledge-base-not-syncing-when-files-are-being-over-written-in-s3-bucket)  
[Data sync in bedrock knowledge base | AWS re:Post](https://repost.aws/questions/QU-fiWWnNoQ3ilQEr5sYiCRQ/data-sync-in-bedrock-knowledge-base)  
[Bedrock Data source sync does not work | AWS re:Post](https://repost.aws/questions/QUkePzI8W_TQqBGBlSKBeRnA/bedrock-data-source-sync-does-not-work)  
[Does Knowledge Base auto update data from s3 | AWS re:Post](https://repost.aws/questions/QUZOmMEo5hTLOZ4cZn9l7jrg/does-knowledge-base-auto-update-data-from-s3)

Knowledgebase sync not updating the AI

Add your answer

Relevant content