Bug: Bedrock KB RetrieveAndGenerate fails with filters applied when query decomposition is enabled

Question

When calling Bedrock python SDK's retrieve_and_generate with the following configuration, I am consistently getting the "Sorry, I am unable to assist you with this request." response, when I know there is data that should be retrieved as part of the vector search:

```{
    "type": "KNOWLEDGE_BASE",
    "knowledgeBaseConfiguration": {
        "knowledgeBaseId": "...",
        "modelArn": "us.anthropic.claude-sonnet-4-20250514-v1: 0",
        "retrievalConfiguration": {
            "vectorSearchConfiguration": {
                "numberOfResults": 10,
                "overrideSearchType": "SEMANTIC",
                "rerankingConfiguration": {
                    "type": "BEDROCK_RERANKING_MODEL",
                    "bedrockRerankingConfiguration": {
                        "modelConfiguration": {
                            "modelArn": "arn:aws:bedrock:us-east-1: :foundation-model/cohere.rerank-v3-5: 0"
                        }
                    }
                },
                "filter": {
                    "equals": {
                        "key": "...",
                        "value": "..."
                    }
                }
            }
        },
        "orchestrationConfiguration": {
            "queryTransformationConfiguration": {
                "type": "QUERY_DECOMPOSITION"
            }
        },
        "generationConfiguration": {
            "inferenceConfig": {
                "textInferenceConfig": {
                    "temperature": 0
                }
            }
        }
    }
}```

Removing the query decomposition from the request resolves the issue:

```
        "orchestrationConfiguration": {
            "queryTransformationConfiguration": {
                "type": "QUERY_DECOMPOSITION"
            }
        },```

Wanted to make this post for others that may run into this as well.

re:Post Agent · Accepted Answer

Thank you for reporting this issue. Based on the configuration you've shared, it appears you're experiencing a conflict between query decomposition and metadata filtering in your Knowledge Base retrieval setup.

Query decomposition is a query expansion technique that breaks complex queries into targeted sub-questions to improve retrieval effectiveness. However, when combined with metadata filters, there may be challenges in how the decomposed sub-queries interact with your filter conditions.

When you enable query decomposition, the system transforms your original query into multiple sub-questions. If your metadata filter is restrictive (filtering on a specific key-value pair), it's possible that:

1. The decomposed sub-queries may not align well with your filtered dataset
2. Each sub-query is being evaluated against the same filter, potentially resulting in no matches for some sub-questions
3. The orchestration layer may be failing to aggregate results properly when filters are applied to decomposed queries

Your workaround of removing query decomposition is a valid approach, especially if your queries are relatively straightforward and don't require decomposition. Alternatively, you could consider:

- Adjusting your metadata filter to be less restrictive (if appropriate for your use case)
- Using query decomposition without filters, or filters without query decomposition
- Increasing the numberOfResults to ensure sufficient chunks are retrieved for each sub-query
- Testing with different query formulations that may work better with your filter constraints

This appears to be a configuration interaction issue rather than a bug per se, but it's valuable feedback for others implementing similar retrieval patterns with Knowledge Bases.  
**Sources**  
[Evaluate and improve performance of Amazon Bedrock Knowledge Bases | AWS Machine Learning Blog](https://aws.amazon.com/blogs/machine-learning/evaluate-and-improve-performance-of-amazon-bedrock-knowledge-bases/)  
[RetrieveAndGenerate - Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent-runtime_RetrieveAndGenerate.html)

Bug: Bedrock KB RetrieveAndGenerate fails with filters applied when query decomposition is enabled

Add your answer

Relevant content