Skip to content

How to delete objects using mask in S3?

0

I am using aws-sdk (JavaScript) and want to delete everything that starts with prefix 130/**/**/25/*. In other words, I want to delete every object with the key that starts with 130 and ends with 25, however the exact path between 130 and 25 is unknown. The number of path segments between 130 and 25 is fixed (exactly two in this case). The path that has 25 at this exact position (the 4th position) is unique. In other words, if for example the path 130/10/76/25 exists, then there are no other paths that have 25 at 4th position. It's also known that the desired path starts with 130.

We can't list all items and search just by 25, because there might be paths like 130/25/71/4. The desired unique path may be like 130/25/25/25.

How to do this efficiently. Ideally, I would love to be able to send just one request with the desired mask to S3 that will delete everything with desired prefix. If it's not possible then I would like to make as few requests as possible by transferring as few data between external app and S3 as possible. Retrieving the full list of objects by the app and searching on the app side would be undesirable.

asked a year ago240 views
1 Answer
0
Accepted Answer

To delete objects in S3 using a mask with the AWS SDK for JavaScript (v3), you'll need to use a combination of listing objects and then deleting them. Unfortunately, S3 doesn't provide a direct way to delete objects using wildcards or complex patterns in a single operation. Here's an efficient approach to accomplish your task:

  1. Use the ListObjectsV2Command to list objects with the prefix "130/". This will return all objects that start with "130/".

  2. Filter the results on the S3 side using the "Delimiter" parameter set to "/" and the "Prefix" parameter set to "130/". This will return only the immediate subdirectories under "130/".

  3. For each subdirectory, repeat step 2 with the updated prefix.

  4. When you reach the level with "25" as the last segment, you've found the objects to delete.

  5. Use the DeleteObjectsCommand to delete the matching objects in batches.

Here's a code outline to implement this approach:

import { S3Client, ListObjectsV2Command, DeleteObjectsCommand } from "@aws-sdk/client-s3";

const s3Client = new S3Client({});
const bucketName = "your-bucket-name";

async function deleteObjects(prefix) {
  const listParams = {
    Bucket: bucketName,
    Prefix: prefix,
    Delimiter: '/'
  };

  const listCommand = new ListObjectsV2Command(listParams);
  const listResponse = await s3Client.send(listCommand);

  if (prefix.split('/').length === 4 && prefix.endsWith('25/')) {
    // We've reached the target level, delete all objects
    const objectsToDelete = listResponse.Contents.map(obj => ({ Key: obj.Key }));
    if (objectsToDelete.length > 0) {
      const deleteParams = {
        Bucket: bucketName,
        Delete: { Objects: objectsToDelete }
      };
      const deleteCommand = new DeleteObjectsCommand(deleteParams);
      await s3Client.send(deleteCommand);
      console.log(`Deleted ${objectsToDelete.length} objects with prefix ${prefix}`);
    }
  } else {
    // Continue traversing subdirectories
    for (const commonPrefix of listResponse.CommonPrefixes || []) {
      await deleteObjects(commonPrefix.Prefix);
    }
  }
}

// Start the deletion process
deleteObjects('130/');

This approach minimizes data transfer by using S3's listing capabilities to narrow down the objects to delete. It makes multiple small requests rather than retrieving the full list of objects at once. The deletion is performed in batches for efficiency.

Keep in mind that this method still requires multiple API calls to S3, but it's more efficient than retrieving all object keys and filtering on the client side. If you have a very large number of objects, you may need to implement pagination for the listing operation.
Sources
Delete all objects in a given Amazon S3 bucket using an AWS SDK - Amazon Simple Storage Service
Delete all objects in a given Amazon S3 bucket using an AWS SDK - AWS SDK Code Examples

answered a year ago
EXPERT
reviewed a year ago
  • Provided code may do the thing, but to be honest I just settled to make an extra call to my DB to retrieve the full prefix, so I will not need to list objects in the S3 bucket.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.