Knowledge Base for Bedrock Indexing

Question

I had a doubt regarding knowledge base for bedrock, let's say I have 3 documents in s3 data source of knowledge base and I add two more, then I run start_ingestion_job using lambda, will it re-index the entire s3 data source to knowledge base or only embed the newly added documents.

It would be great if you could help me with this.

Thanks and Regards.

It would be great if you could help me with this.

Thanks and Regards.

Didier_Durand · Accepted Answer

Hi,

You get this information when running `aws bedrock-agent list-ingestion-jobs --data-source-id  .` 
See https://docs.aws.amazon.com/cli/latest/reference/bedrock-agent/get-ingestion-job.html

```
"ingestionJobSummaries": [
        {
            "dataSourceId": "",
            "description": "ds sync after S3 refresh: 2024-05-30-10-45-17",
            "ingestionJobId": "7JLW4JTNVS",
            "knowledgeBaseId": """,",
            "startedAt": "2024-05-30 08:45:18.918511+00:00",
            "statistics": {
                "numberOfDocumentsDeleted": 0,
                "numberOfDocumentsFailed": 1,
                "numberOfDocumentsScanned": 641,
                "numberOfMetadataDocumentsModified": 0,
                "numberOfMetadataDocumentsScanned": 0,
                "numberOfModifiedDocumentsIndexed": 0,
                "numberOfNewDocumentsIndexed": 640
            },
            "status": "COMPLETE",
            "updatedAt": "2024-05-30 08:51:30.050024+00:00"
        }
    ]
```

Explanations:

```
numberOfDocumentsDeleted -> (long) : The number of source documents that was deleted.
numberOfDocumentsFailed -> (long): The number of source documents that failed to be ingested.
numberOfDocumentsScanned -> (long) : The total number of source documents that were scanned. Includes new, updated, and unchanged documents.
numberOfMetadataDocumentsModified -> (long) : The number of metadata files that were updated or deleted.
numberOfMetadataDocumentsScanned -> (long) : The total number of metadata files that were scanned. Includes new, updated, and unchanged files.
numberOfModifiedDocumentsIndexed -> (long) : The number of modified source documents in the data source that were successfully indexed.
numberOfNewDocumentsIndexed -> (long): The number of new source documents in the data source that were successfully indexed.
```

So, the answer to your question is in numberOfModifiedDocumentsIndexed: Bedrock detects the modified documents via some form of checksum and re-index those only

Best,

Didier

Nithin Nair · Answer

Thankyou Didier for your answer.

I was checking the response object but everytime I was getting 0 for all fields like documents scanned and newly indexed. But when searching the knowledge base it was reflecting the proper data source, so I was confused as to whether it is re-indexing all documents even if unchanged.
But thanks for your explanation.

Thanks again and Regards.
Nithin

Knowledge Base for Bedrock Indexing

Relevant content