Injecting custom metadata to an index created by Bedrock ingesting a file/crawled website via Lambda

Question

Hello, I am trying to insert custom metadata (for example 'year':2025) to the index that is created automatically by Bedrock when you ingest a file/crawled website. The index created automatically by bedrock has the following fields on Open Search Serverless: AMAZON_BEDROCK_METADATA, AMAZON_BEDROCK_TEXT, id, x-amz-bedrock-kb-data-source-id, x-amz-bedrock-kb-document-page-number, x-amz-bedrock-kb-source-uri. Thus, the lambda function I created on AWS Lambda targets AMAZON_BEDROCK_METADATA :

import json

def lambda_handler(event, context):
    """
    Lambda function to modify chunk-level metadata in ingestion.
    """
    try:
        print(json.dumps(event, indent=2))

output_files = []

for file in event.get('inputFiles', []):
            # Prepare custom metadata payload nested under the key AMAZON_BEDROCK_METADATA
            custom_metadata = {
                "AMAZON_BEDROCK_METADATA": {
                    "source": file.get('originalFileLocation', {})
                                 .get('s3_location', {})
                                 .get('uri', "Unknown"),
                    "year": "2025",
                    "author": "Holly Wood"
                }
            }

# Process content batches and inject custom metadata only at the chunk level.
            content_batches = []
            for batch in file.get('contentBatches', []):
                content_batches.append({
                    "key": batch.get('key'),
                    "contentMetadata": custom_metadata  # Custom metadata is nested correctly
                })

# Also set file-level metadata to the custom metadata (optional but may help ensure consistency)
            output_files.append({
                "originalFileLocation": file.get('originalFileLocation'),
                "fileMetadata": custom_metadata,  # Now file-level metadata has the custom metadata nested under AMAZON_BEDROCK_METADATA
                "contentBatches": content_batches
            })

print(json.dumps({"outputFiles": output_files}, indent=2))
        return {"outputFiles": output_files}

except Exception as e:
        raise e

It targets chunk-level metadata as the documentation (https://docs.aws.amazon.com/bedrock/latest/userguide/kb-custom-transformation.html) specifies "Please note that chunk-level metadata take precedence and overwrite file-level metadata, in case of any collisions."

However, when I run my ingestion code which utilises this Lambda function (which does work under testing conditions) , the printed metadata only includes the "source" key, which is automatically created by Bedrock during ingestion.

Is Bedrock overwriting the metadata my Lambda function attempts to inject into AMAZON_BEDROCK_METADATA? is there a work around for this? I just want to be able to add my own custom metadata to files ingested by Bedrock.

Thank you very much.

re:Post Agent · Answer

Based on your description, it appears that you're encountering an issue where the custom metadata you're trying to inject via your Lambda function is not being preserved in the final index created by Amazon Bedrock. This is a common challenge when working with custom transformations during the ingestion process.

There are a few potential reasons and solutions to consider:

1. Metadata Overwriting: As you've noted, Bedrock may be overwriting the metadata. The documentation does state that chunk-level metadata takes precedence, but it's possible that Bedrock is still applying its own processing after your Lambda function.

2. Metadata Structure: Ensure that your custom metadata is structured correctly. The AMAZON_BEDROCK_METADATA field is typically reserved for Bedrock's use. Instead of nesting your custom metadata under this key, try adding your custom fields directly to the root level of the metadata object.

3. Timing of Transformation: Make sure your Lambda function is being applied at the correct step in the ingestion process. The 'stepToApply' parameter in your ingestion configuration should be set to "POST_CHUNKING" to ensure your metadata is added after Bedrock's initial processing.

4. Permissions: Verify that your Lambda function has the necessary permissions to modify the metadata and that Bedrock has permissions to read the modified data.

5. Alternative Approach: If the above doesn't work, you might consider using a custom field name for your metadata instead of trying to modify AMAZON_BEDROCK_METADATA. For example, you could use "CUSTOM_METADATA" as your top-level key.

Here's a modified version of your Lambda function that might work better:

```python
def lambda_handler(event, context):
    try:
        output_files = []
        for file in event.get('inputFiles', []):
            custom_metadata = {
                "CUSTOM_METADATA": {
                    "year": "2025",
                    "author": "Holly Wood"
                }
            }
            content_batches = []
            for batch in file.get('contentBatches', []):
                content_batches.append({
                    "key": batch.get('key'),
                    "contentMetadata": custom_metadata
                })
            output_files.append({
                "originalFileLocation": file.get('originalFileLocation'),
                "fileMetadata": custom_metadata,
                "contentBatches": content_batches
            })
        return {"outputFiles": output_files}
    except Exception as e:
        raise e
```

If these approaches don't resolve the issue, you may need to contact AWS support for further assistance, as there might be limitations or specific requirements in Bedrock's ingestion process that are not fully documented.

Remember to test your changes incrementally and monitor the ingestion process closely to see how your modifications affect the final index structure in OpenSearch Serverless.
      
**Sources**  
[Use a custom transformation Lambda function to define how your data is ingested - Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/kb-custom-transformation.html)  
[Customize ingestion for a data source - Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/kb-data-source-customize-ingestion.html)

Injecting custom metadata to an index created by Bedrock ingesting a file/crawled website via Lambda

Add your answer

Relevant content