How can I stop Bedrock Knowledge Bases from parsing html source files for custom chunking lambda

Question

Background Info: 
- My source files are html files.
- I am using OpenSearch Serverless Vector Index
- I have created a custom chunking lambda, which uses a chunking logic that leverages html structure, in addition to llama-index's SentenceSplitter.

Problem:  
I cannot not figure out how to keep the datasource syncing process from parsing the source html files into text which is dumped into the intermediate lambda bucket.

I have tried:
- renaming files into .txt, 
- changing their content type metadata in S3 to txt/plain
- removing all tags up to the  tag and also removing all tags after the  tag
- added a custom parser with a prompt to "leave the content alone and not to change it in any way".

Nothing has helped.  The content that is dumped into the intermediate bucket are all parsed, and the html tags are removed.  I was finally able to keep it from being parsed when I put a fragmented and broken html source in the files.

Is there really no way to keep the ingestion from applying any parsing?  If I can't turn parsing off, is there a trick to dupe the parser into thinking it's a plain text file?

Thanks!

Didier Durand · Answer

Hi,

Source:  https://docs.aws.amazon.com/bedrock/latest/userguide/kb-chunking-parsing.html#kb-custom-transformation

```
You may have specific chunking logic, not natively supported by Amazon Bedrock knowledge bases. 
Use the no chunking strategy option, while specifying a Lambda function that contains your chunking logic. 
Additionally, you'll need to specify an Amazon S3 bucket for the knowledge base to write files to be chunked 
by your Lambda function.
```

So, if you apply the above, Bedrock will not do anything re. chunking, it will defer that to your Lambda. In that case, you can do exactly what you want in the Lambda to match your chunking strategy: in particular, if you don't want to chunk some of your files at all, you can just returned them unchanged (and also skip their chunking and vectorization if you need).

Your Lambda has to contain the logic to choose between your various (non-) chunking options

Best,

Didier

How can I stop Bedrock Knowledge Bases from parsing html source files for custom chunking lambda

Add your answer

Relevant content