When Amazon Bedrock performs incremental syncing with the start_ingestion_job API for a knowledge base connected to an S3 bucket, it is designed to detect and process changes efficiently. Incremental syncing means that only the files that have been added, modified, or deleted since the last sync are processed, rather than re-ingesting the entire dataset. This applies to both content files and their associated metadata files (e.g., file_name.metadata.json).

In your scenario, where you have content files and corresponding metadata files (e.g., file_name.txt and file_name.txt.metadata.json) in your S3 bucket, and you modify only the metadata file (e.g., file_name.txt.metadata.json) without changing the content file (file_name.txt), here’s what happens:

If you change only the file_name.txt.metadata.json file and not the file_name.txt content file, Bedrock will sync and process just the updated metadata file. The content file’s embeddings in the vector store remain unchanged, and only the metadata associated with it is updated in the knowledge base.

Bedrock’s incremental syncing process tracks changes based on the S3 object’s metadata, such as the last modified timestamp. When you update the metadata file, its timestamp changes, signaling to Bedrock that it has been modified.

If you modify both the metadata file and the content file, Bedrock will detect changes to both and process them accordingly. The content file will be re-ingested (re-chunked and re-embedded if necessary), and the updated metadata will be applied.

Bedrock treats the content file and its metadata file as separate entities in S3. The metadata file (file_name.metadata.json) provides additional attributes that are stored alongside the embeddings of the content file in the vector store, but it does not inherently trigger a reprocessing of the content file unless the content file itself is modified. This design optimizes performance by avoiding unnecessary reprocessing of unchanged content.

If your goal is to update only the metadata (e.g., to change filtering attributes or add new metadata fields) without altering the content embeddings, you can safely modify just the file_name.metadata.json file and run start_ingestion_job. Bedrock will handle this efficiently by syncing only the changed metadata file and updating the knowledge base accordingly.

Reference:

Sync your data with your Amazon Bedrock knowledge base - Amazon Bedrock Modify a data source for your Amazon Bedrock knowledge base - Amazon Bedrock