Hierarchical structure of S3 KB data source

0

All, does anybody know if hierarchical structure of S3 KB data source is supported? Or do i need include each folder as a separate data source? In my simple test (2 "parallel" folders) only one folder is being indexed (but with full path). I am using no chunk strategy. One folder (with json files) does not have metadata, the other one (with csv files) has metadata. Any information would be greatly appreciated Thank you YK

asked 8 months ago238 views
1 Answer
1

When defining your data source you can choose the S3 bucket or a prefix within the bucket. Everything under what you choose will be indexed.

Hope this helps!

profile pictureAWS
EXPERT
answered 8 months ago
profile picture
EXPERT
reviewed 8 months ago
profile pictureAWS
EXPERT
reviewed 8 months ago
  • Can you please elaborate prefix part? Currently i have S3 URI as s3://yk-test-foldered, which is a root folder, under which i have the required hierarchy. How URI should look with prefix? Thank you YK PS Also when i tried again everything worked as it supposed to...i still need to check number of indexes...

  • A prefix would be the same as a directory within the bucket. In S3 they are not really directories but just prefixes in the name of key (file). In the below, folder-1 and folder-2 would be prefixes.

    s3://my-test-bucket s3://my-test-bucket/folder-1/file-1 s3://my-test-bucket/folder-1/file-2 s3://my-test-bucket/folder-2/file-1 s3://my-test-bucket/folder-2/file-2

  • so for s3://my-test-bucket everything in this folder and underlaying hierarchy should be indexed? that did not fork before but worked yesterday. i will do a clear source and try again. Thank you YK

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions