Skip to content

Is AWS S3 provide a generic way to get S3 object with respect to Header attribute, any APIs available ?

0

What currently I am doing to achieve is below way. using library awssdk JAR file

ListObjectVersionsIterable versions = s3.listObjectVersionsPaginator(
              ListObjectVersionsRequest.builder().bucket(bucket).prefix(getPath()).build());
          for (ObjectVersion vs : versions.versions()) {
            HeadObjectResponse metadata = s3.headObject(
                HeadObjectRequest.builder().bucket(bucket).key(getPath()).versionId(vs.versionId()).build());
            if (metadata != null) {
              String hxVersion = metadata.hasMetadata() ? metadata.metadata().get("x-attr") : null;
              if (withVersion.equals(hxVersion) || withVersion.equals(metadata.versionId())) {
                GetObjectRequest objectRequest =
                    GetObjectRequest.builder().bucket(bucket).key(getPath()).versionId(metadata.versionId()).build();
                s3.getObject(objectRequest); // Here is actual object retrieved.
              }
            }
          }

The problem with this approach is to traverse through long number of items over pagination which is performance impacting mechanism.

I am looking whether AWS provide direct APIs to work with header attribute values to pull out S3 documents.

asked 2 years ago472 views
1 Answer
0

Hi,

You may want to try S3 Inventory to achieve what you need: https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory.html

The inventory structure is described here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory-location.html

If you don't want to deal with the inventory file directly to process it and filter what you need, you can even load the inventory in Athena to query it via SQL: https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory-athena-query.html

Best,

Didier

EXPERT
answered 2 years ago
EXPERT
reviewed 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.