Skip to content

Hierarchical structure of S3 KB data source

0

All, does anybody know if hierarchical structure of S3 KB data source is supported? Or do i need include each folder as a separate data source? In my simple test (2 "parallel" folders) only one folder is being indexed (but with full path). I am using no chunk strategy. One folder (with json files) does not have metadata, the other one (with csv files) has metadata. Any information would be greatly appreciated Thank you YK

asked 2 years ago372 views
1 Answer
1

When defining your data source you can choose the S3 bucket or a prefix within the bucket. Everything under what you choose will be indexed.

Hope this helps!

AWS
EXPERT
answered 2 years ago
EXPERT
reviewed 2 years ago
EXPERT
reviewed 2 years ago
  • Can you please elaborate prefix part? Currently i have S3 URI as s3://yk-test-foldered, which is a root folder, under which i have the required hierarchy. How URI should look with prefix? Thank you YK PS Also when i tried again everything worked as it supposed to...i still need to check number of indexes...

  • A prefix would be the same as a directory within the bucket. In S3 they are not really directories but just prefixes in the name of key (file). In the below, folder-1 and folder-2 would be prefixes.

    s3://my-test-bucket s3://my-test-bucket/folder-1/file-1 s3://my-test-bucket/folder-1/file-2 s3://my-test-bucket/folder-2/file-1 s3://my-test-bucket/folder-2/file-2

  • so for s3://my-test-bucket everything in this folder and underlaying hierarchy should be indexed? that did not fork before but worked yesterday. i will do a clear source and try again. Thank you YK

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.