Amazon Bedrock Web Crawler does not support Pinecone

Question

Although there is no mention in the documentation, when trying to configure a knowledge base using Amazon Bedrock's Web Crawler and Pinecone as the vector base, I receive the error message that the WEB option is only supported with AWS OpenSearch. What should I do if I want to use Pinecone and Bedrock's Web Crawler?

re:Post Agent · Answer

You're correct that there's a limitation when using Amazon Bedrock's Web Crawler with Pinecone. Currently, the Web Crawler data source connector for Amazon Bedrock knowledge bases is only compatible with Amazon OpenSearch Serverless as the vector store. This is a specific limitation of the Web Crawler implementation.

If you want to use Pinecone as your vector store while still leveraging web content, you have a few options:

1. Use OpenSearch Serverless specifically for your web-crawled content: You could create a separate knowledge base using OpenSearch Serverless for web content, while using Pinecone for other data sources.

2. Create a custom ingestion pipeline: Instead of using Bedrock's Web Crawler, you could develop your own web scraping solution, process the content, and then load it into Pinecone directly. You would then connect this Pinecone vector store to your Bedrock knowledge base.

3. Use a different supported data source type with Pinecone: While Web Crawler specifically requires OpenSearch Serverless, other data source types (like S3) should work with Pinecone as your vector store.

Keep in mind that vector store options for Amazon Bedrock Knowledge Bases include Amazon OpenSearch Serverless, Amazon Aurora PostgreSQL, Amazon Neptune Analytics, and third-party options like Pinecone, Redis Enterprise Cloud, and MongoDB Atlas - but the Web Crawler specifically has the OpenSearch Serverless requirement.
      
**Sources**  
[Bedrock Knowledge base with Normal PostgreSQL Vector DB (without Aurora) | AWS re:Post](https://repost.aws/questions/QU1QTDHdF9QIu-L-YlgSUQTg/bedrock-knowledge-base-with-normal-postgresql-vector-db-without-aurora)  
[Vector store options for Bedrock Knowledgebase | AWS re:Post](https://repost.aws/questions/QUbsDhN6CRQvGdP43k_xL3Fw/vector-store-options-for-bedrock-knowledgebase)  
[Crawl web pages for your knowledge base - Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/webcrawl-data-source-connector.html)

Riku_Kobayashi · Answer

Hello.

I tried it with my AWS account, but I was unable to set up the Web Crawler with Pinecone.  
As of May 2025, it appears that if you want to use the Web Crawler, you will need to use OpenSearch Serverless.  
Therefore, there is no option to use it with Pinecone.  
https://docs.aws.amazon.com/bedrock/latest/userguide/webcrawl-data-source-connector.html

Amazon Bedrock Web Crawler does not support Pinecone

Add your answer

Relevant content